Lesson 4 of 9
Module 4 · Machine learning
One train–test split is one roll of the dice. Cross-validation rolls it several times. This lesson also shows the most common way to fool yourself, using data that contain no signal at all.
Before you start. This lesson carries on in the notebook from Your first classifier. If you open a new notebook, or Colab has restarted, run this setup cell first. It repeats the loading, the split and the model from that lesson.
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
data = load_breast_cancer(as_frame=True)
df = data.frame
df["diagnosis"] = df["target"].map({0: "malignant", 1: "benign"})
X = df[data.feature_names]
y = (df["diagnosis"] == "malignant").astype(int)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)Cross-validation
In five-fold cross-validation the data are cut into five parts. The model is trained on four and tested on the fifth, five times, so every sample is tested once.
Each of the five parts is called a fold. Why go to this trouble? A single test set of 114 masses is one draw. A different random split could put slightly easier or harder masses in the test set and move the score up or down. With five folds you get five scores from five different test sets, and you can see how much the result moves. Every sample is used for testing exactly once and for training four times, so no data are wasted, which matters most when samples are scarce.
from sklearn.model_selection import cross_val_score scores = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc") print(scores.round(3), round(scores.mean(), 3))
What this does
cross_val_scoreruns the whole loop for you. It takes the model (the pipeline from Your first classifier), the featuresX_trainand labelsy_trainfor the 455 training masses, and two settings. The 114 test masses from Your first classifier stay sealed.cv=5asks for five folds. When the model is a classifier and you give a plain number, scikit-learn uses stratified folds, so each fold keeps about the same share of malignant masses, as in Your first classifier.scoring="roc_auc"scores each test fold by AUC rather than accuracy.- For each round, a fresh copy of the model is trained on four folds and scored on the fifth. The result,
scores, is an array of five numbers. The print line shows them to three decimal places, followed by their mean.
Reading the output
The part in square brackets is the five fold scores, in order. NumPy prints the second as 1. followed by spaces to keep the columns lined up; it means an AUC of 1.000 after rounding, a perfect ranking in that fold. The single number after the brackets is the mean, 0.995. The lowest fold is 0.984 and the highest 1.000, a narrow range.
Five different splits give consistently high AUC, so the result is not a lucky draw. Report the mean and the spread across folds. Because model is a pipeline, the scaler is refitted inside each fold, which is exactly what we want.
The spread is usually reported as the standard deviation of the fold scores, or as their lowest and highest values. A mean of 0.995 with folds from 0.984 to 1.000 tells a reader something quite different from the same mean with folds from 0.90 to 1.00. Trees and ensembles prints the standard deviation for you.
Be clear about what cross-validation measures. It does not test one finished model; it tests the recipe: this scaler, this model, these settings, trained on data of this size. The five fitted copies are thrown away. Once you are happy with the recipe, you fit it once more on all the training data, and that is the model you keep. That is why the code above cross-validates the training set only. The test set from Your first classifier stays sealed for one final check of the model you keep; if you cross-validated all 569 masses, those test masses would already have helped you choose the recipe.
How many folds? Five and ten are the usual choices. More folds mean bigger training sets in each round, closer to the size of the final model, but more fits and noisier single-fold scores. One pitfall hides in the default: when you pass cv=5, scikit-learn does not shuffle the rows before cutting them into folds. If the file happens to be sorted, by hospital or by date for example, each fold can end up quite unlike the others. Shuffling explicitly avoids that:
from sklearn.model_selection import StratifiedKFold cv5 = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores = cross_val_score(model, X_train, y_train, cv=cv5, scoring="roc_auc")
For a steadier estimate on small data, RepeatedStratifiedKFold repeats the whole five-fold procedure several times with different shuffles and gives you every score.
Fooling yourself with noise
Here are 60 “patients” with 5,000 random “genes” and labels that have nothing to do with them. Any honest method should score about 50%. A very common mistake is to pick the most informative genes using all the data, and only then cross-validate.
Feature selection means keeping only some of the features, usually the ones that look most related to the label. With thousands of genes and few samples it is tempting, and often sensible. The danger lies in when you do it. The code runs the same experiment twice, once in the wrong order and once in the right one.
from sklearn.feature_selection import SelectKBest, f_classif
rng = np.random.default_rng(0)
X_noise = rng.normal(size=(60, 5000)) # 60 "patients", 5,000 random "genes"
y_noise = np.array([0, 1] * 30) # labels unrelated to X_noise
# WRONG: choose the 20 best genes using all the data, then cross-validate
X_sel = SelectKBest(f_classif, k=20).fit_transform(X_noise, y_noise)
wrong = cross_val_score(LogisticRegression(max_iter=1000), X_sel, y_noise, cv=5)
print("selected before CV:", wrong.mean().round(2))
# RIGHT: selection inside the pipeline, learned within each training fold
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
right = cross_val_score(pipe, X_noise, y_noise, cv=5)
print("selected inside CV:", right.mean().round(2))What this does
SelectKBestkeeps thekfeatures with the highest scores.f_classifis the score it uses here: an F-test (the test behind one-way ANOVA) that measures how far apart a feature's averages are in the two classes, relative to the spread within each class.np.random.default_rng(0)creates a random number generator with a fixed starting point, the seed 0, so the "random" data are the same every time.rng.normal(size=(60, 5000))fills a table of 60 rows and 5,000 columns with random numbers from a bell-shaped (normal) distribution. Nothing in it is related to anything.[0, 1] * 30repeats the two-item list 30 times, giving 60 labels that alternate 0, 1, 0, 1. There are 30 of each, and they have no link to the numbers.- Wrong order:
fit_transformscores all 5,000 genes using all 60 labels and returns only the best 20 columns,X_sel. Cross-validation then runs on those 20 columns. Without ascoringargument, a classifier is scored by accuracy. - Right order: the selection step goes into a pipeline with the model. Inside each fold, the pipeline picks its 20 genes using only the training rows, then predicts the held-out rows.
Reading the output
The wrong order reports 0.9, a mean accuracy of 90%, on data that contain no signal at all. The right order reports 0.42, or 42%. That is a little below the 50% you would expect from guessing, but with only 12 samples in each test fold, a few lucky or unlucky guesses move the score a lot; 42% is what chance looks like at this size.
Change the seed in default_rng(0) and both numbers will move, but the pattern stays: the wrong order reports a score far above chance, and the right order hovers around 50%. The same wrong-order result can also be reached without SelectKBest. Picking "the genes that came out significant in our differential expression analysis" from the full dataset and then training a classifier on them is exactly the same mistake, written in different words. It is common because each step looks reasonable on its own.
The shuffled-label check from the key point below can be automated. permutation_test_score, from sklearn.model_selection, reruns cross-validation many times with the labels shuffled and tells you how often chance alone matches your real score. If your pipeline is honest, shuffled labels give scores around chance, and your real score stands clearly above them.
Worked example · why 90% from pure noise
choosing the top 20 with all 60 labels bakes the test labels into the features
result: 90% “accuracy” on data with no signal · done properly: 42%, about chance
This is the classic error in omics papers with many more features than samples. Any step that looks at the labels, including feature selection, must happen inside each training fold.
Step by step. If you test 5,000 unrelated genes for a difference between two groups at the usual 5% significance level, about 5% of them, roughly 250 genes, will pass by chance alone. The top 20 are the genes whose random values happen to line up best with all 60 labels, including the 12 labels in each future test fold. When cross-validation later hides a fold, it is too late: those 12 samples already helped choose the genes, so the genes "work" on them. Inside the pipeline, the genes are chosen from 48 training samples only; on the 12 unseen samples their chance pattern does not hold, and the score falls back to about 50%.
What the answer means. A high cross-validated score proves nothing if any step before the cross-validation looked at the labels. The more features you start with, the easier it is to find a chance pattern, which is why the problem is worst in omics.
The noise experiment repeats a classic demonstration. Ambroise and McLachlan (2002), “Selection bias in gene extraction on the basis of microarray gene-expression data”, PNAS, showed that choosing genes on all the samples before cross-validation could make error rates on published cancer datasets look close to zero. Kaufman et al. (2012), “Leakage in data mining”, ACM Transactions on Knowledge Discovery from Data, give a general account of how leakage happens.
When you read a paper, check where feature selection, scaling and tuning happened. Many methods sections do not say clearly. If you cannot tell whether a step was inside the loop, treat the reported score with caution.
Other kinds of leakage
Data leakage is any route by which information about the test samples, or about the answer, reaches the model during training. The feature-selection error is one route. Here are three more.
- Repeated patients: two samples from one patient in training and test. Split by patient, with
GroupKFold. - Time: training on later data to predict earlier events. Split by date for anything clinical.
- Proxy features: a column that encodes the answer, such as a treatment given only after diagnosis.
Why each one leaks:
- Repeated patients. Two biopsies from the same patient share that person's genetics and many other traits. If one is in training and one in testing, the model can recognise the patient rather than the disease.
GroupKFoldis the scikit-learn splitter that takes a group label, here a patient ID, and keeps all of a group's samples in the same fold. - Time. A model will be used on future patients. Testing it on patients seen before the training patients asks an easier question than the real one, because practice, equipment and populations change over time. Train on the earlier period and test on the later one.
- Proxy features. A column such as "chemotherapy started" is only filled in after a cancer diagnosis, so it predicts the diagnosis perfectly and is useless for a new patient who has not yet been diagnosed. For every feature, ask: would I know this value at the moment I want to make the prediction?
Two further routes are common in omics. The first is batch effects: differences caused by when or where samples were processed, not by biology. If most cancer samples were sequenced in one batch and most controls in another, a model can learn the batch and still score well in cross-validation, because the same confusion runs through every fold. Check how cases and controls are spread across batches before modelling. The second is preprocessing on the whole dataset: normalising, scaling or filling in missing values using all samples before the split. As in Your first classifier, every such step belongs inside the pipeline.
Duplicates are a quieter version of the repeated-patient problem. In Cleaning and combining data you saw how a merge can copy rows. A sample that appears twice can land once in training and once in testing, and the model is then marked on an answer it has already seen. Check for duplicate sample IDs, and for identical rows, before you split.
Go deeper: splitting by patient and by time in code
Splitting by patient needs one extra piece of information: a list with one patient ID per row. Then cross_val_score(model, X, y, groups=patient_ids, cv=GroupKFold(n_splits=5)) keeps every row from one patient in the same fold. If you also want each fold to keep the share of each class, StratifiedGroupKFold does both at once.
For time, sort the rows by date and use TimeSeriesSplit(n_splits=5). Each round trains on an earlier block and tests on the block that follows, so the model never learns from the future. All three splitters live in sklearn.model_selection. One check is worth running every time: after splitting, confirm that no patient ID appears in both a training and a test part. With ids as a NumPy array and train, test taken from the splitter’s split(), set(ids[train]) & set(ids[test]) should be empty.
Key point
If a result looks too good, assume leakage until you have ruled it out. Rerun the whole pipeline on shuffled labels: an honest pipeline scores about chance.
Shuffling the labels breaks any real link between features and answers while keeping everything else the same. It is the noise experiment above, applied to your own data. A second example of a result that looks too good: an imaging model that scores almost perfectly because every image from the cancer clinic carries that clinic's label in one corner. On shuffled labels it would fall to chance, and on images from another hospital it would fail; both checks point to the same leak.
Further reading · Module 4Chicco 2017: ten practical tips for machine learning in biology. · All reading for this course
Check your understanding
1. In 5-fold cross-validation, how many times is each sample used for testing?
2. Why did random data score 90%?
3. A dataset has three biopsies per patient. How should you split it?
Up next
Lesson 5
Trees and ensembles