QuestionQ9

VerifiedData Pipeline Orchestration

Your team is developing several data pipelines that include a collection of complex tasks and dependencies that must run on a schedule in a particular order. The tasks and dependencies involve files in Cloud Storage, Apache Spark jobs, and data in BigQuery. You need to design a fully managed system to schedule and automate these data-processing tasks. What should you do?

Explanation

Cloud Composer is a fully managed Apache Airflow service designed for data-driven workflows. Its DAGs define task ordering and dependencies, support scheduled execution, and use Google Cloud operators to interact with Cloud Storage, Apache Spark, and BigQuery.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!