QuestionQ309

Data Ingestion and Transformation

A company keeps CSV files in an Amazon S3 bucket. A data engineer must process the data in the CSV files and save the processed data to a new S3 bucket.

The process must:

  • rename a column;
  • remove specified columns;
  • skip the second row of every file;
  • create a new column from values in the first row of the data; and
  • filter the results according to a numeric value in a column.

Which solution satisfies these requirements with the LEAST development effort?

  • A Use AWS Glue Python jobs to read and transform the CSV files.
  • B Use an AWS Glue custom crawler to read and transform the CSV files.
  • C Use an AWS Glue workflow to build a set of jobs to crawl and transform the CSV files.
  • D Use AWS Glue DataBrew recipes to read and transform the CSV files.
Explanation

AWS Glue DataBrew provides no-code recipe steps for data preparation, including column renaming and removal, derived-column transformations, row removal, and condition-based filtering. A recipe can be run against data in Amazon S3 and write its output to another S3 location, avoiding the custom transformation code required by a Glue Python job. AWS Glue crawlers catalog data, and workflows orchestrate jobs; neither is the primary transformation mechanism.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!