Lesson 3 of 9
Module 4 · Machine learning
96.5% accuracy sounds excellent. A clinician would ask: how many cancers did it miss? This lesson answers that, and shows how the threshold changes the answer.
Before you start. This lesson carries on in the notebook from Your first classifier. If you open a new notebook, or Colab has restarted, run this setup cell first. It repeats the loading, the split and the model from that lesson.
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
data = load_breast_cancer(as_frame=True)
df = data.frame
df["diagnosis"] = df["target"].map({0: "malignant", 1: "benign"})
X = df[data.feature_names]
y = (df["diagnosis"] == "malignant").astype(int)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)The confusion matrix
A confusion matrix is a small table that counts every kind of right and wrong answer a classifier gives. For two classes it has four cells, and each has a name. A true positive (TP) is a cancer the model called malignant. A true negative (TN) is a benign mass it called benign. A false positive (FP) is a benign mass it called malignant: a false alarm. A false negative (FN) is a cancer it called benign: a missed cancer. Every other measure in this lesson is built from these four counts.
from sklearn.metrics import confusion_matrix, roc_auc_score, roc_curve
import matplotlib.pyplot as plt
pred = model.predict(X_test)
print(confusion_matrix(y_test, pred))
tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()
print("sensitivity", round(tp / (tp + fn), 3), "specificity", round(tn / (tn + fp), 3))
print("PPV", round(tp / (tp + fp), 3), "NPV", round(tn / (tn + fn), 3))What this does
- The first line imports three functions for judging a classifier; the second imports matplotlib's plotting interface, used later for the ROC curve.
model.predict(X_test)gives a 0 or 1 for each of the 114 test masses. It uses the default threshold: a mass is called malignant (1) when its predicted probability is above 0.5.confusion_matrix(y_test, pred)compares the truth with the predictions. The order of the arguments matters: truth first, prediction second..ravel()flattens the 2 × 2 table into four numbers, read row by row. Because class 0 (benign) comes first, they arrive in the order true negatives, false positives, false negatives, true positives, and the line unpacks them into four named variables.- The last two lines turn the counts into four rates, each rounded to three decimal places. Each is a fraction: a count of correct answers divided by the group it belongs to.
Reading the output
The first two lines are the matrix, printed as a nested list. The first row is the 72 truly benign masses: 71 predicted benign, 1 predicted malignant. The second row is the 42 truly malignant masses: 3 predicted benign, 39 predicted malignant. The diagonal from top left to bottom right holds the correct answers (71 + 39 = 110), which matches the 0.965 accuracy from Your first classifier. The last two lines give the four rates as decimals: 0.929 is 92.9%, and so on.
A common pitfall is reading the matrix the wrong way round. scikit-learn puts the truth in rows and the prediction in columns, and orders the classes by sorting them, so 0 comes before 1. Other software, and many textbooks, put the positive class first or the prediction in rows. If your labels are words such as "benign" and "malignant", they are sorted alphabetically, which here happens to put benign first but will not always suit you. Passing labels=[0, 1] to confusion_matrix fixes the order explicitly, and saying "rows are the truth" in any report removes the doubt.
The four counts are also more honest than any rate built from them. "Sensitivity 92.9%" sounds precise; "39 of 42" tells the reader at once how few cancers the estimate rests on. This is the same principle as showing every point in Showing distributions: when numbers are small, show the numbers.
Worked example · reading the matrix
Rows are the truth, columns the prediction: first the 72 benign masses, then the 42 malignant.
malignant: 39 caught, 3 missed
sensitivity = 39 ÷ 42 = 92.9% · specificity = 71 ÷ 72 = 98.6%
Three cancers were missed. In a real service each is a delayed diagnosis, which is why a model like this would support, not replace, a pathologist.
Step by step. Row totals first: 71 + 1 = 72 benign and 3 + 39 = 42 malignant. Sensitivity divides the cancers caught by all the cancers: 39 ÷ 42 = 0.929, or 92.9%. Specificity divides the benign masses cleared by all the benign masses: 71 ÷ 72 = 0.986, or 98.6%. Now the columns: the model called 1 + 39 = 40 masses malignant, and 39 of them were, so PPV = 39 ÷ 40 = 97.5%. It called 71 + 3 = 74 benign, and 71 were, so NPV = 71 ÷ 74 = 95.9%.
What the answer means. The model rarely raises a false alarm, but it misses about 1 cancer in 14. Whether that is acceptable depends on what happens next. If every "benign" call still goes to a pathologist, a miss is caught later; if the model were used to decide who needs no further tests, those 3 patients would go home with cancer.
How to read the table below
Each row names a measure and the plain question it answers. The first two start from the truth: given a real cancer, or a real benign mass, how often is the model right? PPV and NPV start from the prediction: given what the model said, how far can you trust it? That second pair is what a patient wants to know. Recall and precision are the names used in machine learning for sensitivity and PPV; they are the same calculations.
| Measure | Question it answers |
|---|---|
| Sensitivity (recall) | Of the real cancers, what share did it catch? |
| Specificity | Of the benign masses, what share did it clear? |
| PPV (precision) | When it says cancer, how often is it right? Depends on how common cancer is. |
| NPV | When it says benign, how often is it right? |
| AUC | Across every threshold, how well does it rank cancers above benign masses? |
Why does PPV depend on how common cancer is? Take the same sensitivity (92.9%) and specificity (98.6%) and imagine a screening setting where only 100 of 10,000 people have cancer. The model catches about 93 of the 100. It also wrongly flags 1.4% of the 9,900 healthy people, about 139. Of the roughly 232 people it calls positive, only 93 have cancer: a PPV of about 40%, not 97.5%. The test set here is 37% malignant, so its PPV looks far better than it would in screening. NPV moves the other way, for the same reason.
Clear short explanations of these measures are Altman and Bland’s 1994 Statistics Notes in the BMJ, “Diagnostic tests 1: sensitivity and specificity” and “Diagnostic tests 2: predictive values”. When positives are rare, Saito and Rehmsmeier (2015, PLOS ONE) show that precision–recall plots are more informative than ROC curves.
Report uncertainty as well. A sensitivity of 39 out of 42 is 92.9%, but with only 42 cancers its 95% confidence interval runs from roughly 81% to 98%. A confidence interval is the range of values consistent with the data at a stated level of confidence. A second test set of similar size could easily give a noticeably different sensitivity.
Every threshold: ROC and AUC
The threshold of 0.5 is only one choice. Lower it and the model calls more masses malignant: it catches more cancers but raises more false alarms. An ROC curve (receiver operating characteristic, a name from radar engineering) shows this trade-off at every possible threshold at once. It plots the true positive rate (sensitivity) against the false positive rate (1 − specificity, the share of benign masses wrongly flagged). The AUC is the area under that curve, between 0 and 1.
probs = model.predict_proba(X_test)[:, 1]
print("AUC", round(roc_auc_score(y_test, probs), 3))
fpr, tpr, thresholds = roc_curve(y_test, probs)
plt.plot(fpr, tpr, color="#D9481F", label="Logistic regression")
plt.plot([0, 1], [0, 1], "--", color="grey", label="Chance")
plt.xlabel("False positive rate"); plt.ylabel("True positive rate")
plt.legend(); plt.show()
# what if we lower the threshold to catch more cancers?
print(confusion_matrix(y_test, (probs > 0.3).astype(int)))What this does
probsholds the predicted probability of malignancy for all 114 test masses. AUC needs these probabilities, not the 0/1 predictions, because it looks at how the masses are ranked.roc_auc_score(y_test, probs)computes the AUC.roc_curvetries every useful threshold and returns three arrays of the same length: the false positive rate, the true positive rate, and the threshold that produced each pair.- The first
plt.plotdraws the curve in vermilion. The second draws a dashed grey diagonal from (0, 0) to (1, 1): the line a model with no skill would follow. The next lines label the axes, add a legend and display the plot. - The last line applies a new threshold of 0.3.
probs > 0.3givesTruefor every mass whose probability is above 0.3;.astype(int)turns that into 1s and 0s, and the confusion matrix is printed again.
Reading the output
The AUC is 0.996, very close to the maximum of 1. One way to read it: pick one malignant and one benign mass from the test set at random, and the model gives the malignant one the higher probability 99.6% of the time. The new matrix, at a threshold of 0.3, has the same top row (71 cleared, 1 false alarm), but the bottom row is now 1 missed and 41 caught. Two of the three missed cancers had probabilities between 0.3 and 0.5.
What does a poor AUC look like? An AUC of 0.5 means the model ranks a cancer above a benign mass no more often than a coin toss: its ROC curve lies along the dashed diagonal. An AUC below 0.5 usually means something is upside down, such as the wrong probability column or the classes coded the other way round. On real clinical data, an AUC between 0.7 and 0.85 is common, and an AUC as high as 0.996 is a reason to check for leakage before celebrating. Here it reflects how cleanly these 30 measurements separate the two groups in a single, curated dataset.
AUC also has limits. It summarises every threshold, but a clinic uses one, so a model with a high AUC can still perform poorly at the threshold you actually need. And when positives are rare, the ROC curve can look excellent while precision is low, for the reason given in the PPV example above. In that case, plot precision against recall as well: precision_recall_curve and average_precision_score, both in sklearn.metrics, work exactly like roc_curve and roc_auc_score.
Draw ROC curves honestly, as in Honest axes and scales. Both axes are rates, so both should run from 0 to 1 with equal scaling, and the chance diagonal should be shown. Cropping the axes to zoom in on the top-left corner makes small differences between models look large.
At a threshold of 0.3, the model catches 41 of 42 cancers, with still only one false alarm on this test set. Choosing a threshold is a clinical decision about which error matters more, not a technical default. Choose it on training data or a separate validation set, not by looking at the test set.
A validation set is a third slice of the data, kept apart from both the training and the test set, used only to make choices such as this one. The reason is the same as before: if you try thresholds on the test set and keep the best, the test set has helped build the model, and its score is no longer an honest estimate. Cross-validation and data leakage shows how cross-validation can play the part of a validation set when data are scarce.
One more question sits behind any threshold: can the probabilities be taken at face value? A model is well calibrated when, among masses given a probability of about 0.3, about 30% really are malignant. Many models are not. Their rankings can be excellent, giving a high AUC, while the probabilities themselves are too extreme or too timid. If a clinician will read the number as a risk, check calibration first: calibration_curve, from sklearn.calibration, groups the predictions into bins and compares each bin's average probability with the share that were truly positive. With only 114 test masses, such a check is rough, but it can still reveal a model that is badly off. Tuning, interpreting and reporting runs this check on a tuned model.
Go deeper: choosing a threshold without touching the test set
You can choose a threshold from the training data alone. cross_val_predict(model, X_train, y_train, cv=5, method="predict_proba")[:, 1], from sklearn.model_selection, gives every training mass a probability from a model that did not see it, called an out-of-fold prediction. Store the result as oof.
Pass those probabilities to roc_curve(y_train, oof) to get fpr, tpr, thr. Suppose the clinical team wants at least 97% sensitivity. Because tpr rises along the array while thr falls, thr[np.argmax(tpr >= 0.97)] is the highest threshold that reaches it. Fix that number, then apply it once to the test set. Since version 1.5, scikit-learn also offers TunedThresholdClassifierCV, which automates this search by cross-validation.
Key point
Always report sensitivity and specificity together, with the counts behind them. A single accuracy figure hides which mistakes the model makes.
A second example: two models can both score 95% accuracy on 100 masses, 37 of them malignant. One misses 5 cancers and raises no false alarms; the other misses none and raises 5 false alarms. Accuracy cannot tell them apart, yet for a cancer service they are very different tools. Sensitivity of 32 ÷ 37 against 37 ÷ 37 shows the difference at once.
Further reading · Module 4Chicco 2017: ten practical tips for machine learning in biology. · All reading for this course
Check your understanding
1. At the default threshold, how many malignant masses were missed?
2. Lowering the threshold from 0.5 to 0.3 will usually:
3. An AUC of 0.5 means the model:
Up next
Lesson 4
Cross-validation and data leakage