QuestionQ96

VerifiedData Pipeline Orchestration

You have a Dataproc cluster that runs batch processing on data in Cloud Storage. You must schedule a daily Spark job to create a report that will be emailed to stakeholders. You need a fully managed solution that is straightforward to implement and keeps complexity to a minimum. What should you do?

Explanation

Cloud Composer is a fully managed Apache Airflow service for creating, scheduling, monitoring, and managing workflows. An Airflow DAG can invoke a Dataproc Spark job and run a subsequent email task after the report is produced, providing both required steps in one managed orchestration workflow.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!