A Data Engineer must monitor a continuous data pipeline that processes data. Data is ingested into Snowflake from an Amazon S3 bucket through a pipe, and then transformed through a series of tasks and streams. The pipeline sends notifications when data importation fails.
How should the Engineer add a notification to indicate a failure in a task used for data transformations?
A Create a queue-type error integration and assign it to this root task:CREATE INTEGRATION my_error_notifyTYPE = QUEUE -ENABLED = TRUE -DIRECTION = OUTBOUND -NOTIFICATION-PROVIDER = AWS_SNS -AWS_SNS_TOPIC_ARN = 'arn:aws:sns:us-east-2:111122223333:my_topic'AWS_SNS_ROLE_ARN = 'arn:aws:iam::111122223333:role/error_sns_role';ALTER TASK order_start SET ERROR_INTEGRATION = my_error_notify; B Create a Snowflake alert and assign it to this root task:CREATE ALERT pipeline_failure -WAREHOUSE = MY_WH -NOTIFY = my_user1, my_user2;ALTER TASK order_start -set ERROR_INTEGRATION = pipeline_failure; C Create an email notification and assign it to this root task:CREATE NOTIFICATION INTEGRATION my_email_notifyTYPE = EMAIL -ENABLED = TRUE -DEFAULT RECIPIENTS = ('[email protected]');ALTER TASK order_start -set NOTIFICATION_INTEGRATION = my_email_notify; D Create a queue notification integration error and assign it to this root task:CREATE NOTIFICATION INTEGRATION mv_error_notifyTYPE = QUEUE -ENABLED = TRUE -DIRECTION = OUTBOUND -NOTIFICATION_PROVIDER = AWS_SNS -AWS_SNS_TOPIC_ARN = 'am:aws:sns:us-east-2:111122223333:my_topic'AWS_SNS_ROLE_ARN = 'arn:aws:iam::111122223333:role/error sns role';ALTER TASK order_start SET ERROR_INTEGRATION = my_error_notify; Show Answer Answer Explanation Snowflake sends task-failure notifications through a notification integration assigned with ERROR_INTEGRATION on the root task of a task graph. A queue notification integration configured for outbound AWS SNS provides the required delivery mechanism; failures in child transformation tasks are relayed through the root task’s configured error integration.
Learn more
Community Discussion