QuestionQ141

Data Ingestion and Transformation

A data engineer has a one-time requirement to read data from objects in an Amazon S3 bucket that use the Apache Parquet format. The data engineer must query only one column of the data.

Which solution meets these requirements with the LEAST operational overhead?

  • A Configure an AWS Lambda function to load data from the S3 bucket into a pandas dataframe. Write a SQL SELECT statement on the dataframe to query the required column.
  • B Use S3 Select to write a SQL SELECT statement to retrieve the required column from the S3 objects.
  • C Prepare an AWS Glue DataBrew project to consume the S3 objects and to query the required column.
  • D Run an AWS Glue crawler on the S3 objects. Use a SQL SELECT statement in Amazon Athena to query the required column.
Explanation

Amazon S3 Select can run a SQL SELECT query directly against an Apache Parquet object in Amazon S3 and return only the requested subset of data, including a single column. This avoids custom Lambda code, pandas processing, Glue DataBrew setup, and the Glue crawler/Data Catalog resources required for the Athena approach.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!