AWS Certified Data Engineer - Associate DEA-C01Demo

By Amazon · Question Mode

QuestionQ248

Data Ingestion and Transformation

A company uses Amazon S3 and the AWS Glue Data Catalog to manage a data lake that holds customer contact information. The company uses PySpark and AWS Glue jobs with a DynamicFrame to run a workflow that processes data in the data lake.

A data engineer discovers that the workflow is producing errors because of the way customer postal codes are stored in the data lake. Some postal codes contain extraneous numbers or invalid characters.

The data engineer needs a solution that resolves the errors and corrects the postal codes in the data lake.

  • A Create a schema definition for PySpark that matches the format the processing workflow requires for postal codes. Pass the schema to the DynamicFrame during processing.
  • B Use AWS Glue workflow properties to allow job state sharing. Configure the AWS Glue jobs to read values from the postal code column by using the properties from a previously successful run of the jobs.
  • C Configure the column.push_down_predicate setting and the catalogPartitionPredicate settings for the postal code column in the DynamicFrame.
  • D Set the DynamicFrame additional_options parameter ‘useS3ListImplementation’ to True.
Explanation

AWS Glue DynamicFrames can represent schema inconsistencies and support resolving them to a concrete type for downstream processing. Defining the expected processing schema for the postal-code field is the applicable approach among the choices; workflow properties, partition predicates, and the S3 listing implementation do not cleanse or reconcile the postal-code data format.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!