QuestionQ108

VerifiedData Operations and Support

A data engineer needs to validate the quality of files that are uploaded to an Amazon S3 bucket every day. The files are in CSV and JSON formats and schema variations exist. The data engineer needs a repeatable process to monitor data quality metrics such as null values, format inconsistencies, and outliers. The process must provide reusable rule-based profiling and scale across multiple datasets with minimal manual effort or custom code.

Which solution will meet these requirements?

Explanation

AWS Glue DataBrew provides built-in, no-code profiling jobs that compute data quality statistics (nulls, format inconsistencies, outliers) using reusable recipes, and these profiling jobs can be scheduled to run automatically against each dataset regardless of its schema. This requires far less custom code than building PySpark validation logic or dynamic frame schema-resolution jobs, and it avoids the manual review step that a purely dashboard-based approach would require. Learn more: Evaluating data quality with AWS Glue DataBrew

Community Discussion

No comments yet. Be the first to start the discussion!