Measuring a Model
Ninety-nine percent accurate can still be useless. How to judge a model honestly with confusion matrices, precision, recall, and the ROC curve.
You have built a model. Is it any good? It is tempting to answer with one number: accuracy, the fraction of predictions it got right. But accuracy hides more than it reveals, and trusting it blindly is one of the most expensive mistakes in machine learning. This page shows why, then gives you the tools professionals actually use to judge a model: the confusion matrix, precision and recall, the F1 score, and the ROC curve. Move the slider in the demo and watch every one of them shift.
Why accuracy lies
Imagine a test for a rare disease that affects 1 person in 100. Here is a "model": always predict healthy. It never spots a single sick person, yet it is 99 percent accurate, because 99 percent of people really are healthy. Accuracy looks brilliant; the model is worthless.
This is the accuracy paradox. When one class is much rarer than the other (imbalanced data), a model can score high accuracy while completely failing at the job that matters. Fraud, disease, and defect detection are all like this: the interesting cases are rare, so accuracy alone tells you almost nothing.
The confusion matrix
Every honest evaluation starts here. A confusion matrix breaks predictions into four boxes by comparing what the model said against the truth. For a yes-or-no (positive-or-negative) classifier:
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True Positive (TP): correct hit | False Negative (FN): a miss |
| Actually negative | False Positive (FP): a false alarm | True Negative (TN): correct all-clear |
Those four numbers contain everything. Accuracy is just (TP + TN) / total, but now you can see what it throws away: it treats a missed cancer (FN) and a false alarm (FP) as equally bad, when in real life they are worlds apart. Every richer metric below is built from these four counts.
Precision and recall
Two metrics pull apart the two kinds of error, and they are the most important numbers on the page.
- Precision punishes false alarms. High precision means when the model says positive, you can trust it.
- Recall punishes misses. High recall means the model catches most of the real positives.
The catch is that they trade off. Flag everything as positive and recall hits 100 percent, but precision collapses under false alarms. Flag only the surest cases and precision soars, but you miss many real ones, so recall drops. You cannot usually max both, so you choose based on the cost of each error.
The F1 score
Sometimes you want a single number that balances precision and recall. The F1 score is their harmonic mean:
The harmonic mean is used, rather than a plain average, because it stays low unless both precision and recall are high. A model with 100 percent precision but 1 percent recall averages to 50 percent, which flatters it, but its F1 is about 2 percent, which tells the truth. F1 rewards balance and refuses to be fooled by one strong number masking a weak one.
The ROC curve and AUC
Most classifiers do not output a hard yes or no. They output a score (a probability), and you choose a threshold: above it is positive, below it is negative. Every threshold gives a different confusion matrix, so how good is the model across all of them? The ROC curve answers this.
As you sweep the threshold from high to low, you plot two things: the true positive rate (recall, on the vertical axis) against the false positive rate (false alarms among the negatives, on the horizontal axis). This traces a curve from the bottom-left to the top-right.
- A curve that hugs the top-left corner is excellent: high recall with few false alarms.
- The diagonal line is random guessing, no skill at all.
- The AUC (area under the curve) sums this up in one number: 1.0 is perfect, 0.5 is a coin flip.
Move the threshold
Now make it concrete. Below, each dot is one example with a model score, blue dots are truly negative and red dots truly positive, spread along the score axis. Drag the threshold. Everything to its right is predicted positive. Watch the confusion matrix, all four metrics, and the point on the ROC curve move together in real time.
Red dots are truly positive, blue dots truly negative, placed by the model's score. The line is your threshold.
| Pred + | Pred - | |
|---|---|---|
| Actual + | 0TP | 0FN |
| Actual - | 0FP | 0TN |
Validating honestly
One last, vital point: where you measure matters as much as what you measure. A model scored on the same data it trained on will look far better than it really is, because it has seen the answers. To measure honestly you must test on data the model has never seen.
- Train, validation, and test split: train on one slice, tune on a second, and report the final score on a third, untouched slice.
- Cross-validation: split the data into several folds, train on all but one and test on the held-out fold, rotate, and average. This uses the data efficiently and gives a more stable estimate.
These same ideas apply to models that predict numbers rather than classes (regression), which use their own measures such as mean absolute error and root mean squared error, the average size of the prediction error. Whatever the metric, always report it on unseen data.
Key points and practice
- Accuracy alone misleads, especially on imbalanced data (the accuracy paradox).
- The confusion matrix splits results into TP, FP, FN, TN, the basis for everything else.
- Precision = TP/(TP+FP) punishes false alarms; recall = TP/(TP+FN) punishes misses; they trade off.
- F1 is their harmonic mean, a single score that stays low unless both are high.
- The ROC curve plots recall against the false positive rate across thresholds; AUC summarises it.
- Always evaluate on unseen data using a test split or cross-validation.
Because the disease is rare, a model that simply predicts "healthy" for everyone is automatically 99 percent accurate, yet it never detects a single case, so high accuracy hides complete failure. This is the accuracy paradox on imbalanced data. Instead you should measure recall (the fraction of real cases caught), precision (the fraction of positive predictions that are correct), and their combination via the F1 score or the ROC curve, which reveal how well the model handles the rare positive class.
Precision is TP/(TP+FP), the fraction of positive predictions that are actually positive, so it penalises false alarms. Recall is TP/(TP+FN), the fraction of actual positives that were correctly identified, so it penalises misses. Recall should be favoured in cancer screening, where missing a real case is far worse than a false alarm. Precision should be favoured in a spam filter, where wrongly binning a genuine email is worse than letting some spam through.
The ROC curve plots the true positive rate (recall) against the false positive rate as the decision threshold is varied across all its values, showing how the model trades catches against false alarms at every threshold. The AUC is the area under this curve; an AUC of 0.5 corresponds to the diagonal line and means the model is no better than random guessing, while 1.0 would be a perfect classifier.