QuestionQ62

VerifiedMonitoring AI solutions

An ML model was deployed to production one year ago. Each month, you collect all raw requests sent to the model prediction service during the prior month. You send a subset of those requests to a human labeling service to assess the model’s performance.

After one year, you observe that model performance sometimes degrades substantially after a month, whereas at other times it takes several months to detect a performance decrease. The labeling service is expensive, but large performance degradations must also be avoided. You need to decide how often to retrain the model in order to maintain high performance while minimizing cost. What should you do?

Explanation

Training-serving skew detection compares aggregate feature distributions in recent serving data with the training-data baseline, allowing potential data-distribution changes to be identified before paying to obtain labels. Running these checks frequently and sending recent serving data to human labeling only when skew is detected focuses the expensive performance evaluation and subsequent retraining decisions on periods where degradation is more likely.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!