QuestionQ174
Data Ingestion and TransformationA company must build a data pipeline that processes a 1-TB file in an Amazon S3 bucket. The pipeline must create three DataFrames according to business logic and save all three DataFrames concurrently to a second S3 bucket. The company must configure the pipeline as the target of an Amazon EventBridge rule that matches file uploads to the source S3 bucket.
Which solution satisfies these requirements with the LEAST maintenance overhead?
- A Configure an Apache Spark Streaming application on Amazon EMR to process data from the S3 source bucket in batches, create DataFrames, and save the output to the destination S3 bucket.
- B Configure three AWS Lambda functions to process the business logic and to save the DataFrames to the destination S3 bucket in parallel.
- C Configure an AWS Glue workflow to run three AWS Glue jobs in parallel to process the file.
- D Configure an AWS Step Functions state machine to initiate an AWS Glue workflow to run three AWS Glue jobs in parallel to process the file.
Community Discussion