QuestionQ82

VerifiedAutomating and orchestrating ML pipelines

While profiling your TensorFlow model’s training time, you identify a performance problem caused by inefficiencies in the input data pipeline for a single 5-terabyte CSV-file dataset on Cloud Storage. You need to optimize input-pipeline performance. Which action should you try first to improve your pipeline’s efficiency?

Explanation

Remote storage can provide high aggregate bandwidth, but reading one file may use only a small portion of it. Sharding the dataset into multiple sufficiently large CSV files enables concurrent reads; using parallel Dataset.interleave overlaps data loading and reduces input-processing time.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!