QuestionQ129

VerifiedMaintaining and automating data workloads

Several Spark jobs run on a Cloud Dataproc cluster according to a schedule. Some jobs execute sequentially, while others execute concurrently. You need to automate this process. What should you do?

Explanation

Cloud Composer DAGs define scheduled workflows and task dependencies. Dependent Spark job tasks run in sequence, while tasks without dependencies can run concurrently; Composer can orchestrate Dataproc jobs as part of that workflow.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!