QuestionQ6

VerifiedData Operations and Support

A company uploads .csv files to an Amazon S3 bucket. The company’s data platform team has configured an AWS Glue crawler for data discovery and for creating the tables and schemas.

An AWS Glue job writes processed data from the tables to an Amazon Redshift database. The AWS Glue job performs column mapping and creates the Amazon Redshift tables in the Redshift database as appropriate.

If the company runs the AWS Glue job again for any reason, duplicate records are added to the Amazon Redshift tables. The company requires a solution that updates the Redshift tables without duplicates.

Which solution meets these requirements?

Explanation

Loading rows into a Redshift staging table and using SQL to update rows that match existing target records prevents a rerun from inserting another copy of those records. Redshift supports merge patterns that use a staging table to update matching target rows and insert only rows that do not already match.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!