AWS Certified Data Engineer - Associate DEA-C01Demo

By Amazon · Question Mode

QuestionQ184

Data Operations and Support

A sales company uses AWS Glue ETL to collect, process, and ingest data into an Amazon S3 bucket. The AWS Glue pipeline creates a new file in the S3 bucket each hour. File sizes range from 200 KB to 300 KB. The company wants to build a sales-prediction model using data from the preceding 5 years. The historical data contains 44,000 files.

The company creates a second AWS Glue ETL pipeline using the smallest worker type. This second pipeline retrieves the historical files from the S3 bucket and processes them for downstream analysis. The company observes significant performance problems with the second ETL pipeline.

The company needs to improve the performance of the second pipeline.

Which solution will meet this requirement in the MOST cost-effective way?

  • A Use a larger worker type.
  • B Increase the number of workers in the AWS Glue ETL jobs.
  • C Use the AWS Glue DynamicFrame grouping option.
  • D Enable AWS Glue auto scaling.
Explanation

AWS Glue DynamicFrame file grouping lets an ETL task read multiple small Amazon S3 files as a single in-memory partition, reducing excessive Spark tasks, partitions, and S3 request overhead. For fewer than 50,000 input files, grouping must be explicitly enabled with groupFiles set to inPartition; this is appropriate for 44,000 files of 200–300 KB each.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!