AWS Certified Data Engineer - Associate DEA-C01Demo

By Amazon · Question Mode

QuestionQ181

Data Operations and Support

A gaming company uses Amazon Kinesis Data Streams to gather clickstream data. It uses Amazon Data Firehose delivery streams to store that data in JSON format in Amazon S3. The company’s data scientists use Amazon Athena to query the newest data for business insights.

The company wants to lower Athena costs without recreating the data pipeline.

Which solution satisfies these requirements with the LEAST management effort?

  • A Change the Firehose output format to Apache Parquet. Provide a custom S3 object YYYYMMDD prefix expression and specify a large buffer size. For the existing data, create an AWS Glue extract, transform, and load (ETL) job. Configure the ETL job to combine small JSON files, convert the JSON files to large Parquet files, and add the YYYYMMDD prefix. Use the ALTER TABLE ADD PARTITION statement to reflect the partition on the existing Athena table.
  • B Create an Apache Spark job that combines JSON files and converts the JSON files to Apache Parquet files. Launch an Amazon EMR ephemeral cluster every day to run the Spark job to create new Parquet files in a different S3 location. Use the ALTER TABLE SET LOCATION statement to reflect the new S3 location on the existing Athena table.
  • C Create a Kinesis data stream as a delivery destination for Firehose. Use Amazon Managed Service for Apache Flink (previously known as Amazon Kinesis Data Analytics) to run Apache Flink on the Kinesis data stream. Use Flink to aggregate the data and save the data to Amazon S3 in Apache Parquet format with a custom S3 object YYYYMMDD prefix. Use the ALTER TABLE ADD PARTITION statement to reflect the partition on the existing Athena table.
  • D Integrate an AWS Lambda function with Firehose to convert source records to Apache Parquet and write them to Amazon S3. In parallel, run an AWS Glue extract, transform, and load (ETL) job to combine the JSON files and convert the JSON files to large Parquet files. Create a custom S3 object YYYYMMDD prefix. Use the ALTER TABLE ADD PARTITION statement to reflect the partition on the existing Athena table.
Explanation

Amazon Data Firehose can natively convert JSON input to Apache Parquet before delivery to Amazon S3. Parquet is a columnar format that reduces storage and enables Athena to scan less data, and date-based S3 prefixes enable partition pruning for time-filtered queries. Increasing the delivery buffer also produces fewer, larger files. Converting and compacting the historical JSON files with AWS Glue extends those same benefits to existing data without replacing the streaming pipeline.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!