AWS Certified Data Engineer - Associate DEA-C01Demo

By Amazon · Question Mode

QuestionQ127

Data Ingestion and Transformation

A data engineer has configured an AWS Glue Data Catalog for data stored in Amazon S3 buckets. The data engineer must configure the Data Catalog to receive incremental updates.

The data engineer configures event notifications for the S3 bucket and creates an Amazon Simple Queue Service (Amazon SQS) queue to receive the S3 events.

Which combination of steps should the data engineer take to satisfy these requirements with the LEAST operational overhead?

Choose two
  • A Create an S3 event-based AWS Glue crawler to consume events from the SQS queue.
  • B Define a time-based schedule to run the AWS Glue crawler, and perform incremental updates to the Data Catalog.
  • C Use an AWS Lambda function to directly update the Data Catalog based on S3 events that the SQS queue receives.
  • D Manually initiate the AWS Glue crawler to perform updates to the Data Catalog when there is a change in the S3 bucket.
  • E Use AWS Step Functions to orchestrate the process of updating the Data Catalog based on S3 events that the SQS queue receives.
Explanation

An AWS Glue crawler can be configured for Amazon S3 event mode to consume S3 event messages from the specified SQS queue. After the initial crawl, scheduled crawler runs use those events to locate changed data rather than recrawling the entire target, updating the Data Catalog incrementally. This uses managed Glue functionality and avoids maintaining custom Lambda or Step Functions workflows.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!