QuestionQ5

VerifiedData Store Management

A data engineer uses an AWS Glue crawler to catalog data stored in an Amazon S3 bucket. The S3 bucket contains both .csv and .json files. The data engineer configured the crawler to omit the .json files from the catalog.

When the data engineer runs Amazon Athena queries, the queries also process the omitted .json files. The data engineer wants to resolve this issue and needs a solution that will not change the access requirements for the .csv files in the source S3 bucket.

Which solution meets this requirement with the SHORTEST query times?

Explanation

Amazon Athena does not honor AWS Glue crawler exclude patterns; it reads the files in the table's S3 location. Placing the .json files in a separate S3 path prevents Athena from scanning them while leaving access to the .csv files unchanged. AWS documents this exact CSV/JSON scenario and recommends placing files to be excluded in a different location.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!