QuestionQ31

VerifiedData Ingestion and Transformation

A company has created several AWS Glue extract, transform, and load (ETL) jobs that validate and transform data from Amazon S3. The ETL jobs load data into Amazon RDS for MySQL in batches once per day. The ETL jobs use a DynamicFrame to read the S3 data.

The ETL jobs currently process all data in the S3 bucket. However, the company wants the jobs to process only the daily incremental data.

Which solution will satisfy this requirement with the LEAST coding effort?

Explanation

AWS Glue job bookmarks maintain state for data successfully processed in prior job runs. With supported Amazon S3 DynamicFrame sources, enabling bookmarks lets a subsequent scheduled run filter for new data rather than reprocess the complete dataset, without building custom object-status tracking.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!