QuestionQ11

VerifiedData Operations and Support

A company uses the AWS Glue Data Catalog to index data uploaded to an Amazon S3 bucket each day. The company uses daily batch processes in an extract, transform, and load (ETL) pipeline to upload data from external sources to the S3 bucket.

The company runs a daily report on the S3 data. On some days, it runs the report before all daily data has been uploaded to the S3 bucket. A data engineer must be able to send a message identifying any incomplete data to an existing Amazon Simple Notification Service (Amazon SNS) topic.

Which solution meets this requirement with the LEAST operational overhead?

Explanation

AWS Glue Data Quality can evaluate rulesets for Data Catalog data and publish evaluation-result events to Amazon EventBridge. An EventBridge rule can match failed data-quality evaluations, including failures caused by completeness rules, and send the event to an Amazon SNS topic. This provides a managed validation-and-notification path without administering Airflow or EMR clusters or implementing custom Lambda orchestration.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!