QuestionQ73

VerifiedServing and scaling models

You trained a TensorFlow DNN regressor to predict housing prices from predictive features. Your default precision is tf.float64, and you use a standard TensorFlow estimator:

Question Image

The model performs well, but immediately before production deployment you find that serving latency is 10 ms at the 90th percentile and that you currently serve on CPUs. Production requires model latency of 8 ms at the 90th percentile. You are willing to accept a small reduction in performance to meet the latency requirement.

Therefore, you plan to improve latency while evaluating the amount by which the model's predictions decrease. What should you try first to quickly reduce serving latency?

Explanation

Post-training quantization reduces numerical precision without retraining, shrinking the model and reducing inference computation and memory use; any resulting prediction-quality change can be evaluated after conversion. TensorFlow documents post-training quantization as a technique to reduce model size and improve latency, and float16 quantization is intended to incur minimal accuracy loss.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!