AWS Certified Data Engineer - Associate DEA-C01Demo

By Amazon · Question Mode

QuestionQ145

Data Ingestion and Transformation

A manufacturing company is setting up an IoT monitoring system that generates large, complex data streams. The company wants to store the data in an Amazon S3 data lake for real-time and historical analysis. The company needs a solution that can process data quickly, provide short query times, and use resources efficiently without slowing down data ingestion.

The solution must use a Spark streaming extract, transform, and load (ETL) job on Amazon EMR that is configured to write data to an Iceberg table.

Which solution will meet these requirements?

  • A Use Amazon Kinesis Data Streams to ingest the data. Configure the Iceberg table with copy on write (CoW) mode. Enable the AWS Glue Data Catalog compaction optimizer.
  • B Use Amazon Managed Streaming for Apache Kafka (Amazon MSK) to ingest the data. Configure the Iceberg table with copy on write (CoW) mode. Schedule an AWS Glue job for compaction to optimize the Iceberg table.
  • C Use Amazon Kinesis Data Streams to ingest the data. Configure the Iceberg table with merge on read (MoR) mode. Enable the AWS Glue Data Catalog compaction optimizer.
  • D Use Amazon Data Firehose to ingest the data. Use an AWS Lambda function to handle nested schema. Write the data to an Iceberg table with merge on read (MoR) mode in an Amazon S3 table bucket.
Explanation

Kinesis Data Streams feeds the Spark streaming job on EMR with low ingestion latency, and configuring the Iceberg table for merge-on-read keeps writes fast, since new data is simply appended as delete/data files rather than rewriting existing files, which is what keeps ingestion from slowing down. The AWS Glue Data Catalog's built-in compaction optimizer then periodically compacts those small files in the background so query times stay short. Copy-on-write instead forces a full rewrite of affected files on every write, which would slow the very streaming ingestion this workload depends on, and using Data Firehose with a Lambda transform does not satisfy the requirement to write via the Spark streaming ETL job on EMR.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!