QuestionQ33

VerifiedData Operations and Support

A data engineer must troubleshoot an AWS Glue job that reads from Amazon S3 and writes to Amazon Redshift. The data engineer enabled the bookmark feature for the AWS Glue job.

The data engineer set the AWS Glue job's maximum concurrency to 1.

The AWS Glue job is successfully writing output to Amazon Redshift. However, Amazon S3 files loaded in earlier AWS Glue job runs are reprocessed by later runs.

What is the likely reason that the AWS Glue job is reprocessing the files?

Explanation

AWS Glue saves job-bookmark state when job.commit() is called. Without that required commit statement, the job does not persist the source-file state needed to exclude files processed in earlier successful runs, so subsequent runs reprocess them. Maximum concurrency of 1 does not cause the documented concurrent-run bookmark conflict.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!