Lesson 4 of 12
Module 8 · AI for biology
Tell two leukaemias apart from 7,129 gene measurements in 72 patients, test the model on patients it has never seen, find the genes it relies on, and watch a common mistake invent a signal from nothing.
The question
In the lessons from Where AI fits: traditional methods and AI methods side by side to A first sequence classifier, evaluated honestly every model learned from a sequence. Much of real omics work starts somewhere else: a table of measurements, with one row per patient and one column per gene. The classic question is whether that table can tell two diseases apart, and which genes do the telling.
This lesson uses one of the most famous datasets in the field. In 1999 Golub and colleagues measured the activity of 7,129 genes in the bone marrow or blood of 72 people with acute leukaemia. Two kinds were present: acute lymphoblastic leukaemia (ALL), which starts in lymphoid cells, and acute myeloid leukaemia (AML), which starts in myeloid cells. They need different treatment, and in 1999 telling them apart relied on a specialist reading cells under the microscope. Could gene expression do it automatically?
The table has a shape that is normal in omics and unusual in the rest of machine learning: far more measurements than samples, 7,129 genes for 72 patients. With so many columns, a model can find a pattern that separates the training patients perfectly by chance. Statisticians write this shape as p ≫ n: the number of features, p, is much larger than the number of samples, n. Everything in this lesson is about getting an honest answer anyway. The code runs in Google Colab; scikit-learn, NumPy and pandas are already installed, and the data download from OpenML, a public archive of datasets for machine learning.
import numpy as np, pandas as pd
from sklearn.datasets import fetch_openml
d = fetch_openml("leukemia", version=1, as_frame=True, parser="auto")
X, y = d.data, d.target
print(X.shape)
print(y.value_counts().to_dict())Reading the output
72 rows (patients) and 7,129 columns (gene probes), with 47 ALL and 25 AML patients. Each column name, such as X95735_at, is a probe: one spot on the Affymetrix chip used in the study, designed to match one gene. Module 2 showed how to look identifiers up; here the probe names are kept and a few are translated to genes below.
The same questions apply to the RNA-seq count tables you built in Module 5, in Reads to a count table, but the cleaning differs. Counts are whole numbers that depend on sequencing depth, so they are first scaled for library size and then log-transformed, usually after adding a small constant so that zeros can be logged. The microarray steps below are specific to fluorescence readings; the principle behind them, putting values on a scale where equal steps mean equal fold changes, is the same.
Cleaning and splitting the table
The values are raw fluorescence readings from the chip. Some are negative and some run into the tens of thousands, because very weak signals are noisy and very strong ones saturate. Three steps, close to those in the original paper, make them usable:
- Floor and ceiling. Values below 100 are set to 100 and values above 16,000 to 16,000, the range in which the chip measures reliably.
- Log scale. Taking log10 turns a tenfold change into a step of 1, so a gene going from 100 to 1,000 counts as much as one going from 1,000 to 10,000. This is the same idea as the log fold change of Module 1.
- Drop genes that barely change. A gene with the same value in every patient cannot separate anything. Genes whose highest value is less than five times their lowest are removed.
The split comes first, and it is the most important line in the lesson. The original study used the first 38 patients to build its method and kept the other 34 as an independent test. This lesson keeps that split, and every decision, including which genes to drop, is made on the 38 training patients only.
X = np.log10(X.clip(lower=100, upper=16000)) # floor, ceiling, then log10
y = (y == "AML").astype(int) # 1 = AML, 0 = ALL
X_train, X_test = X.iloc[:38], X.iloc[38:] # the original study's split
y_train, y_test = y.iloc[:38], y.iloc[38:]
spread = X_train.max() - X_train.min() # decided on training data only
keep = spread[spread > np.log10(5)].index # at least a five-fold range
X_train, X_test = X_train[keep], X_test[keep]
print("genes kept:", len(keep), "of", X.shape[1])
print("train:", X_train.shape, "AML", y_train.sum(), "| test:", X_test.shape, "AML", y_test.sum())What this does
clip(lower=100, upper=16000)applies the floor and ceiling to every value;np.log10then takes the logarithm. Neither step looks at the labels, and both treat every value the same way, so they cannot leak information.(y == "AML").astype(int)turns the labels into numbers: 1 for AML, 0 for ALL.iloc[:38]andiloc[38:]take the first 38 rows and the rest.spreadis each gene's highest minus lowest log value in the training patients. log10(5) ≈ 0.70, sospread > np.log10(5)keeps genes whose range is more than five-fold.
Reading the output
3,384 of the 7,129 probes vary enough to keep. The training set has 38 patients, 11 with AML; the test set has 34, with 14 AML. The test set is not just the leftovers: in the original study it was deliberately more varied, which makes it a tougher and fairer check.
How to read this figure
Cross-validation (A first sequence classifier, evaluated honestly) splits the 38 training patients into five parts, trains on four and scores on the fifth, five times over. Each highlighted box is a step that learns something from the data it is given. Putting all of them in one pipeline means that, in every fold, they are fitted on the four training parts only. If any of them saw the fifth part first, the score would be measuring a model that had already peeked at its test.
Two models, scored honestly
Two familiar models are compared. The first is a pipeline: drop genes that are constant within the fold, scale every gene to mean 0 and standard deviation 1, keep the 50 genes that best separate ALL from AML in that fold, and fit a logistic regression. The second is a random forest of 500 trees given all 3,384 genes; trees do not need scaling, and each split already picks a gene, so it needs no separate gene choice.
The 50 is a choice, not a fact about the data. Keeping 10 genes or 200 would give a slightly different model. If you try several values and keep the one with the best cross-validation score, that score is again optimistic, for the same reason as in A first sequence classifier, evaluated honestly. The fix, which Measuring a model honestly: confusion matrix, ROC, precision, recall and imbalance shows in code, is nested cross-validation: an inner loop on the training folds picks the value, and an outer loop scores the whole procedure, choice included. With 38 patients, the simpler course is to fix a sensible value in advance and say so.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import VarianceThreshold, SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
models = {
"logistic regression": make_pipeline(VarianceThreshold(), StandardScaler(),
SelectKBest(f_classif, k=50),
LogisticRegression(max_iter=5000)),
"random forest": RandomForestClassifier(n_estimators=500, random_state=0),
}
for name, m in models.items():
s = cross_val_score(m, X_train, y_train, cv=cv)
print(f"{name:20s} CV accuracy {s.mean():.3f}")Reading the output
Both models get about 97% right in cross-validation: across the five folds, one training patient out of 38 was misclassified. That is encouraging but not the final answer. Cross-validation reuses the same 38 patients, and they all came from the same study design. The real test is the 34 patients the models have never seen.
for name, m in models.items():
m.fit(X_train, y_train) # train once on all 38
right = (m.predict(X_test) == y_test).sum()
print(f"{name:20s} test: {right} of {len(y_test)} correct")Reading the output
The logistic regression gets 33 of the 34 new patients right, close to its cross-validation score. The random forest drops to 29 of 34. Same training data, same cross-validation score, different behaviour on new patients. Measuring a model honestly: confusion matrix, ROC, precision, recall and imbalance looks at exactly which patients it missed and why a single accuracy number hides that. For now, note the habit: report the score on data the model never saw, and expect it to be lower than the cross-validation score, sometimes much lower.
Go deeper: why so few patients can still work
With 3,384 genes and 38 patients, almost any two groups of patients could be separated by some gene by luck. The leukaemia data work because the biology is strong: ALL and AML cells come from different blood lineages, and hundreds of genes differ between them by large amounts. A model only has to find a few of them.
Most clinical questions are much harder, for example predicting who will respond to a drug, where the signal is weak and spread across many genes. There, the same code on 38 patients would usually give an unstable model and a test score close to guessing. The number of samples you need depends on how strong the signal is, which is why small studies with impressive accuracy deserve suspicion until they are tested on new patients.
Which genes drive the prediction
A model that separates two cancers is useful; knowing which genes it relies on can also point to biology and to marker genes, genes whose level alone helps identify a disease. Random forests report a feature importance for each column: how much, on average, splits on that gene made the trees' groups purer.
rf = models["random forest"]
imp = pd.Series(rf.feature_importances_, index=X_train.columns)
imp = imp.sort_values(ascending=False)
print(imp.head(8).round(4).to_string())
lr = models["logistic regression"]
chosen = lr[:-1].get_feature_names_out(X_train.columns) # the 50 genes it kept
print("top 8 forest genes also in the 50:", len(set(imp.index[:8]) & set(chosen)))What this does
rf.feature_importances_holds one number per gene, adding up to 1 across all 3,384. Wrapping it in a pandasSerieswith the gene names as the index lets it be sorted and printed with names.lr[:-1]is the logistic regression pipeline without its last step, the model itself;get_feature_names_outreturns the names of the 50 genes that survived the gene-choice step.- The last line counts how many of the forest's top 8 genes are also among those 50.
Reading the output
The top gene, X95735_at, is the probe for zyxin (ZYX), a protein of cell adhesion. M27891_at is cystatin C (CST3) and M23197_at is CD33, a protein on the surface of myeloid cells that is still used today to recognise AML cells and is the target of an antibody drug for AML. Zyxin and CD33 are among the genes the original study reported as most informative. Seven of the forest's top 8 are also chosen by the completely different logistic regression pipeline, a good sign that they reflect the data rather than one model's quirks. Look the other probes up yourself with the Module 2 methods: the chip is the Affymetrix Hu6800, listed in GEO as platform GPL80.
Two cautions. Importance says the model uses a gene, not that the gene causes the disease. And when two genes carry the same information, a forest may split the importance between them, so a gene can matter biologically and still rank low.
A third caution is stability. With 38 patients, the list of top genes can change when the random seed or the training patients change. A useful check is to refit the forest with a few different random_state values, or on resampled sets of patients, and see which genes stay near the top every time. Genes that keep reappearing, and that are also picked by a different model, as here, are the ones worth taking forward to the enrichment methods of Module 7.
Go deeper: SHAP values, and why importance is not explanation
A forest's feature importance is one number per gene for the whole dataset. SHAP values, from Lundberg and Lee (2017), go further: for each patient, they split the model's prediction into a contribution from every gene, based on an idea from game theory for dividing a reward fairly among players. A SHAP plot can show that, for one patient, high CD33 pushed the prediction towards AML by a certain amount. The shap Python package computes them for most scikit-learn models (!pip install shap).
They answer the question "why did the model say this?", not "why does the disease happen?". If the training data contain a batch effect, for example AML samples processed in a different week, SHAP will faithfully report that the model relies on batch-related genes. Interpretation tools are for checking a model, and the checking still needs biology.
The leakage trap, demonstrated
The pipeline put gene choice inside cross-validation. It is tempting to skip that: pick the 50 genes that best separate the two groups using all the samples, then cross-validate a model on those 50. The code below shows why that is wrong, using the clearest possible test. The labels are shuffled at random, so no gene can truly predict them, and any honest method should score no better than guessing.
rng = np.random.default_rng(1)
y_fake = pd.Series(rng.permutation(y_train.values), index=y_train.index) # no real signal
# WRONG: choose the 50 "best" genes using every sample, then cross-validate
best = SelectKBest(f_classif, k=50).fit(X_train, y_fake)
X_leaky = X_train.loc[:, best.get_support()]
plain = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
print("genes chosen outside CV:", round(cross_val_score(plain, X_leaky, y_fake, cv=cv).mean(), 3))
# RIGHT: gene choice is a step inside the pipeline, redone in every fold
honest = make_pipeline(VarianceThreshold(), StandardScaler(),
SelectKBest(f_classif, k=50), LogisticRegression(max_iter=5000))
print("genes chosen inside CV: ", round(cross_val_score(honest, X_train, y_fake, cv=cv).mean(), 3))
print("always guessing ALL: ", round(1 - y_fake.mean(), 3))Reading the output
With labels that mean nothing, choosing genes outside cross-validation reports 89% accuracy: a model that looks excellent and has learned nothing. Choosing genes inside cross-validation gives 65%, below the 71% you would get by always guessing the more common label, which is the honest answer for meaningless labels. The first method fooled itself because, with 3,384 genes, some will separate any grouping of 38 patients by chance, and it picked those genes while looking at the patients it was later tested on.
Gene choice is the most dramatic case, but not the only one. Normalising all samples together before splitting, correcting batch effects with the test samples included, or removing "outlier" patients after looking at the results all let the test data shape the model. Each of them is easy to do by accident in a notebook, because the leaky version is usually the shorter code. The guard is always the same: decide on the split first, and let only the training data inform each step.
Worked example · how big is the fake signal?
accuracy of always guessing the commoner label = 27 ÷ 38 = 0.711
apparent gain from leakage alone = 0.893 − 0.711 ≈ 18 percentage points
Step by step. After shuffling, 27 of the 38 training labels are ALL, so guessing ALL every time scores 27 ÷ 38 ≈ 0.711. The leaky method scores 0.893 on the same meaningless labels. The whole 18-point gap is leakage, not biology. In a real study, that gap would be added on top of whatever true signal exists, and nobody could tell the two apart from the score alone.
The data are from Golub et al. (1999, Science), "Molecular classification of cancer: class discovery and class prediction by gene expression monitoring", one of the first demonstrations that expression profiles can classify cancers. The leakage shown here has a name in this literature, selection bias: Ambroise and McLachlan (2002, Proceedings of the National Academy of Sciences) re-analysed published microarray classifiers and showed that choosing genes before cross-validation could make error rates look close to zero even when the true error was much higher. When you read a gene-expression classifier paper, check whether gene selection was repeated inside every fold, and whether the final test used patients from a separate cohort.
Key point
On an expression table, split first, put every step that learns from data (filtering, scaling, gene choice, the model) inside one pipeline, score it with cross-validation, then test once on patients the model has never seen. Feature importance then shows which genes the model relies on: a starting point for biology, not proof of it.
Further reading · Module 8scikit-learn: Common pitfalls and recommended practices: data leakage and how pipelines prevent it. · OpenML dataset 1104: the leukaemia data used here. · All reading for this course
Check your understanding
1. Why must gene selection happen inside each cross-validation fold?
2. Both models scored 97% in cross-validation. On the 34 new patients the random forest got 29 right. What does that show?
3. A random forest ranks a gene first by feature importance. What can you conclude?
Up next
Lesson 5
Measuring a model honestly: confusion matrix, ROC, precision, recall and imbalance