AWS Certified Data Engineer - Associate DEA-C01Demo

By Amazon · Question Mode

QuestionQ161

Data Ingestion and Transformation

A company must partition the Amazon S3 storage it uses for a data lake. The partitioning will use an S3 object-key path in this format: s3://bucket/prefix/year=2023/month=01/day=01.

A data engineer must ensure that the AWS Glue Data Catalog stays synchronized with the S3 storage when the company adds new partitions to the bucket.

Which solution meets these requirements with the LEAST latency?

  • A Schedule an AWS Glue crawler to run every morning.
  • B Manually run the AWS Glue CreatePartition API twice each day.
  • C Use code that writes data to Amazon S3 to invoke the Boto3 AWS Glue create_partition API call.
  • D Run the MSCK REPAIR TABLE command from the AWS Glue console.
Explanation

The AWS Glue CreatePartition operation creates the partition metadata directly in the Data Catalog. Invoking it from the code that writes each new S3 partition registers that partition as part of the write workflow, avoiding the polling or periodic delay inherent in crawlers, manual runs, or MSCK REPAIR TABLE.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!