QuestionQ64

VerifiedScaling prototypes into ML models

You recently built a deep learning model with Keras and are now trying different training strategies. You initially trained the model on a single GPU, but training was too slow. You then distributed training over 4 GPUs with tf.distribute.MirroredStrategy (making no other changes), but did not see a reduction in training time. What should you do?

Explanation

tf.distribute.MirroredStrategy divides the global batch among the GPU replicas. Increasing the global batch size lets each GPU process a sufficiently large per-replica batch, making better use of the additional compute capacity; the learning rate may also need retuning.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!