QuestionQ14

VerifiedData Operations and Support

A company receives test results from testing facilities located worldwide. The company stores the test results as millions of 1 KB JSON files in an Amazon S3 bucket. A data engineer must process the files, convert them to Apache Parquet format, and load them into Amazon Redshift tables. The data engineer uses AWS Glue to process the files, AWS Step Functions to orchestrate the processes, and Amazon EventBridge to schedule jobs.

The company has recently added more testing facilities. The time needed to process the files is increasing. The data engineer must reduce the data-processing time.

Which solution will MOST reduce the data-processing time?

Explanation

AWS Glue DynamicFrame file grouping combines many small S3 input files into larger in-memory partitions. For JSON input, this reduces per-file and task overhead and is specifically intended to improve performance when processing a large number of small files, while preserving the existing Glue transformation to Parquet and subsequent Amazon Redshift load.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!