QuestionQ84

VerifiedMaintaining and automating data workloads

You are deploying an Apache Airflow directed acyclic graph (DAG) in a Cloud Composer 2 instance. Incoming files in a Cloud Storage bucket are processed by the DAG, one file at a time. The Cloud Composer instance is deployed in a subnetwork without Internet access. Rather than running the DAG on a schedule, you want it to run reactively whenever a new file arrives. What should you do?

Explanation

Cloud Storage object notifications can trigger a Cloud Function, which can call the Airflow REST API on the Airflow web server to create a DAG run for each file event. Serverless VPC Access provides the function with network connectivity to resources in the VPC when the Composer environment has no Internet access. Airflow 2 provides the stable REST API for programmatic DAG-run operations.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!