Professional Data Engineer

By Google · Question Mode

QuestionQ11

Maintaining and automating data workloads

You need to create a SQL pipeline. The pipeline performs an aggregate SQL transformation on a BigQuery table every two hours and appends the result to another existing BigQuery table. You must configure the pipeline to retry when errors occur. You want it to send an email notification after three consecutive failures. What should you do?

Explanation

BigQueryInsertJobOperator submits the BigQuery SQL job that performs the transformation and can write results to an existing destination table. Cloud Composer/Airflow task settings support retries and email_on_failure; the email is sent when the operator reaches its final failed state after retries are exhausted. This meets the requirement to retry the SQL job and notify only after the configured failed attempts.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!