QuestionQ103

VerifiedData Ingestion and Transformation

A company is creating a new application that ingests CSV files into Amazon Redshift. The company has developed the application's frontend.

The files are kept in an Amazon S3 bucket. No file is larger than 5 MB.

A data engineer is building the extract, transform, and load (ETL) pipeline for the CSV files. The data engineer has configured a Redshift cluster and an AWS Lambda function that copies the data from the files into the Redshift cluster.

Which additional steps should the data engineer take to meet these requirements?

Explanation

Amazon S3 can deliver object-created events to Amazon EventBridge. An EventBridge rule can match those events and invoke an AWS Lambda function as its target, so each newly uploaded CSV file triggers the existing load function without requiring polling or an additional migration or streaming service.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!