QuestionQ9
VerifiedData Pipeline OrchestrationYour team is developing several data pipelines that include a collection of complex tasks and dependencies that must run on a schedule in a particular order. The tasks and dependencies involve files in Cloud Storage, Apache Spark jobs, and data in BigQuery. You need to design a fully managed system to schedule and automate these data-processing tasks. What should you do?
Community Discussion