QuestionQ107

VerifiedData Operations and Support

A company has a data lake in Amazon S3. The company uses AWS Glue to catalog data and AWS Glue Studio to build data extract, transform, and load (ETL) pipelines.

The company must ensure that data quality issues are checked every time the pipelines run. A data engineer must enhance the existing pipelines to evaluate data quality rules against predefined thresholds.

Which solution will meet these requirements with the LEAST implementation effort?

Explanation

AWS Glue Studio provides the built-in Evaluate Data Quality transform for ETL jobs. It evaluates rules written in Data Quality Definition Language (DQDL), including threshold-based rules, during job execution; therefore, it avoids implementing and maintaining custom SQL or third-party-library transforms.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!