Associate Data Practitioner

By Google · Question Mode

QuestionQ36

Data Pipeline Orchestration

You are designing a pipeline to process data files that arrive in Cloud Storage by 3:00 am daily. Processing occurs in stages, with each stage’s output serving as input to the following stage. Every stage requires a long time to run. Occasionally, a stage fails and the issue must be resolved. You need to ensure the final output is produced as quickly as possible. What should you do?

Explanation

Cloud Composer orchestrates processing as an Apache Airflow DAG, preserving the status of individual tasks. After fixing the cause of a failed stage or its output data, clearing that failed task’s state lets Airflow rerun the failed task and the necessary downstream tasks while retaining successful upstream work. This avoids the delay of restarting the entire long-running pipeline.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!