When to Use
When a model is live in production and needs ongoing health tracking to catch degradation before users notice.
Core Jobs
1. Monitor Data Drift
Input distribution changes → model becomes unreliable.
- Univariate drift: monitor each feature's distribution (KS test, PSI)
- Multivariate drift: monitor joint distribution (MMD, PCA shift)
- Tools: Evidently AI, WhyLogs, Alibi Detect
- Threshold: PSI > 0.2 = alert, > 0.25 = investigate retraining
2. Monitor Prediction Drift
- Distribution of model outputs changing
- Useful when ground truth is delayed (common in production)
- Alert if output distribution shifts significantly from baseline
3. Monitor Ground Truth (when available)
- Compare predictions vs actuals as labels arrive