QuestionQ24

VerifiedData Ingestion and Transformation

A company runs AWS Glue ETL pipelines to process data and uses Amazon Athena to analyze data in an Amazon S3 bucket.

To gain a better understanding of shipping timelines, the company decides to collect and store shipping and delivery dates alongside order data. The company adds a data quality check to confirm that the shipping date is greater than the order date and that the delivery date is greater than the shipping date. Orders that do not pass the quality check must be stored in a second S3 bucket.

Which solution will meet these requirements in the MOST cost-effective way?

Explanation

AWS Glue Data Quality can define a custom rule that compares the three date columns and produces row-level pass/fail outcomes. The Glue ETL job can use those outcomes to route failed orders to a separate Amazon S3 bucket, avoiding an additional data-preparation service or a separate Athena-based validation process.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!