PART 5 · MODULE 6Keeping AI reliable
Monitoring & Evaluation
Monitoring AI systems in production
Loading module...
Monitoring AI systems in production
Once your model is in production, the real work starts. The world changes, user behavior shifts, and data evolves. Without monitoring, your model quietly degrades until someone notices it is giving terrible answers.
The inputs your model sees in production look different from what it was trained on. A fraud detection model trained on pre-pandemic spending patterns will struggle when everyone starts shopping online.
Training data: mostly in-store purchases. Production data: mostly online orders.
The relationship between inputs and outputs changes. What used to be a "normal" transaction is now suspicious, or vice versa. The rules of the game changed, but your model is still playing by the old ones.
"Luxury goods purchases" used to signal high-value customers. Now it signals potential fraud.
Model performance degrades over time without monitoring
Set thresholds on key metrics. When accuracy drops below 90% or latency exceeds 500ms, get notified automatically. Catch problems before users report them.
Compare model versions with real users. Send 50% of traffic to the old model and 50% to the new one. The data tells you which is better -- not your gut feeling.
Some qualities only humans can judge -- is this response helpful? Is it safe? Is it biased? Automated metrics are fast but incomplete. Human review catches what numbers miss.
Use one LLM to evaluate another. Automated benchmarks test capabilities, while LLM-as-judge approaches rate response quality at scale. Faster than human review, though less nuanced.
Monitoring is not just about watching dashboards. It feeds a continuous improvement cycle.
Deploying a model is not the end -- it is the beginning of a continuous improvement cycle. The best ML teams treat monitoring as a first-class concern, not an afterthought. They know that a model without monitoring is a ticking time bomb -- it will fail silently, and by the time someone notices, the damage is already done.