QuestionQ48

VerifiedData Preparation for Machine Learning (ML)

An ML engineer needs to run a training job on Amazon SageMaker AI. The training job will train a neural network using multiple GPUs. The training dataset is in Parquet format.

The ML engineer finds that the Parquet dataset includes files that are too large to fit in the memory of the SageMaker AI training instances.

Which solution will resolve the memory issue?

Explanation

Smaller repartitioned Parquet files can be loaded as separate input partitions by training workers, avoiding an attempt to load a single oversized file into an instance's memory. AWS documentation notes that Parquet files are read as partitions and recommends splitting data into smaller files for distributed processing.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!