PART 2 · MODULE 5Is your model any good?
Evaluating Models
Measuring model performance
Loading module...
Measuring model performance
A model that's 99% accurate sounds impressive. But if it's a disease-screening model and 1% of the population has the disease, a model that always says "healthy" would be 99% accurate while catching zero cases. Accuracy alone can be misleading.
That's why we need multiple ways to measure how well a model really performs, and we need to test it on data it has never seen before.
The data the model learns from. Like the textbook and practice problems a student studies.
Data the model has never seen. Like the final exam. You don't grade yourself on homework you've already seen.
Imagine a student who memorizes every practice test word-for-word but can't solve a slightly different problem. That's overfitting. The model performs brilliantly on training data but fails on anything new.
Overfitting is like studying only past exams and acing them, then bombing the real test because the questions are slightly different. Good models generalize; they learn the underlying principles, not just the specific examples.
How often are you right?
Correct predictions / Total predictions
When you say "yes", how often are you correct?
True Positives / (True Positives + False Positives)
Of all actual "yes" cases, how many did you find?
True Positives / (True Positives + False Negatives)
The balance between precision and recall
Harmonic mean of Precision and Recall
A confusion matrix is a simple table that shows exactly where your model gets things right and where it gets confused. Here's one for a spam filter:
CONFUSION MATRIX: SPAM FILTER RESULTS (1,000 EMAILS)
With a spam filter, high precision means almost no good emails end up in spam. High recall means almost no spam slips through. You usually can't max out both.
When false alarms are costly. You'd rather let some spam through than accidentally block an important email from a client.
When missing a case is dangerous. A cancer screening should catch every potential case, even if it means more false positives.
Choose your metrics based on what mistakes cost the most. A spam filter and a cancer detector both classify things into two categories, but the consequences of getting it wrong are vastly different. Always ask: "Which type of error hurts more?"