QuestionQ81

VerifiedData Operations and Support

A data engineer creates an AWS Glue Apache Spark ETL job that performs transformations on a dataset. When the data engineer runs the job, it returns the error: “No space left on device.”

The data engineer must identify the error’s source and provide a solution.

Which combination of steps will satisfy this requirement in the MOST cost-effective way?

Choose two
Explanation

Spark shuffle operations write intermediate data to executor local disks; insufficient executor disk space can produce a No space left on device error. The Spark UI and AWS Glue metrics can reveal skew, shuffle activity, and disk spilling. The AWS Glue S3 shuffle manager, enabled by --write-shuffle-files-to-s3, moves shuffle storage to Amazon S3, avoiding local-disk exhaustion; salting distributes records for skewed keys more evenly across partitions.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!