QuestionQ14

VerifiedData Pipeline Orchestration

You work for an online retail company. The company collects customer purchase data in CSV files and uploads them to Cloud Storage every 10 minutes. The data must be transformed and loaded into BigQuery for analysis. The transformation includes cleaning the data, removing duplicates, and enriching it with product information from a separate table in BigQuery. You need a low-overhead solution that starts data processing as soon as files are uploaded to Cloud Storage. What should you do?

Explanation

Cloud Storage can publish an OBJECT_FINALIZE notification to Pub/Sub whenever a new object is successfully created. A Dataflow streaming pipeline can consume these events, read the referenced CSV files, perform cleaning, deduplication, and BigQuery-based enrichment, and write the transformed data to BigQuery with low latency and without operating Composer workflows or Dataproc clusters.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!