QuestionQ118

VerifiedMaintaining and automating data workloads

You need to schedule multiple sequential loading and transformation jobs. An upstream process will add data files to a Cloud Storage bucket, but there is no set schedule for when new data arrives. Next, a Dataproc job is triggered to perform transformations and write the data to BigQuery. You must then run additional transformation jobs in BigQuery. The transformation jobs differ for every table. These jobs can take hours to finish. You need to identify the most efficient and maintainable workflow for processing hundreds of tables while delivering the freshest data to end users. What should you do?

Explanation

A Cloud Storage object trigger provides event-driven processing as soon as new data arrives, avoiding the latency of an hourly schedule. A Cloud Function can invoke a Cloud Composer DAG through the Airflow REST API, and each DAG can sequence the Dataproc transformation followed by its table-specific BigQuery transformations. Separate DAGs isolate the distinct, long-running workflows for individual tables and allow independent execution and maintenance.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!