AWS Certified Data Engineer - Associate DEA-C01Demo

By Amazon · Question Mode

QuestionQ148

Data Ingestion and Transformation

A company uses Amazon S3 to store semi-structured data in a transactional data lake. Some data files are small, while other data files are tens of terabytes.

A data engineer must perform a change data capture (CDC) operation to identify changed data from the data source. The data source sends a full snapshot as a JSON file each day and ingests the changed data into the data lake.

Which solution will capture the changed data MOST cost-effectively?

  • A Create an AWS Lambda function to identify the changes between the previous data and the current data. Configure the Lambda function to ingest the changes into the data lake.
  • B Ingest the data into Amazon RDS for MySQL. Use AWS Database Migration Service (AWS DMS) to write the changed data to the data lake.
  • C Use an open source data lake format to merge the data source with the S3 data lake to insert the new data and update the existing data.
  • D Ingest the data into an Amazon Aurora MySQL DB instance that runs Aurora Serverless. Use AWS Database Migration Service (AWS DMS) to write the changed data to the data lake.
Explanation

Open-source data lake formats such as Apache Hudi, Apache Iceberg, and Delta Lake support transactionally consistent incremental processing for data stored in Amazon S3. Their merge or upsert capabilities can apply new and changed records directly to the existing data lake at large scale, avoiding the cost and impracticality of processing tens-of-terabyte files in Lambda or staging snapshots in a relational database for AWS DMS.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!