Lesson 10 of 12
Module 8 · AI for biology
Most results that fail to hold up fail for a handful of reasons. Learn the checklist: how the data were split, what the baseline was, whether the probabilities mean anything, and whether anyone could rerun it.
Why so many results do not hold up
A paper reports that a model predicts a disease from blood gene expression with 98% accuracy. Another reports a deep learning method that beats every rival on a public benchmark. A third announces a model that predicts drug response from a tumour's genome. Some of these results are real. Many shrink or vanish when someone else tries them on new data.
This lesson is a set of questions to ask, of other people's work and of your own. None of them requires you to be a machine learning specialist. Most of the failures are not in the mathematics; they are in how the data were split, what the model was compared with, and what was reported.
The scale of the problem is easiest to see in one field at one moment. During the COVID-19 pandemic, hundreds of models were published that claimed to diagnose the disease from chest X-rays or CT scans. A systematic review by Roberts and colleagues (2021, Nature Machine Intelligence) examined the most detailed of them and concluded that none was ready for clinical use, mostly because of the problems in this lesson: datasets that mixed sources in revealing ways, patients on both sides of the split, and no external testing. The models were not badly programmed; they were badly evaluated.
A second reason is selection. A team that tries twenty variants of a method and reports the best one has, without meaning to, tuned the result to its particular test set. Journals and readers prefer striking numbers, so the results that reach print are, on average, the luckiest ones. Expect a reported score to fall somewhat when anyone else measures it, and treat a large fall as normal rather than surprising.
Leakage: the biggest single cause
Leakage is information about the test set reaching the model before it is tested. You met the simplest form in A first sequence classifier, evaluated honestly: scaling or choosing features using the whole dataset before splitting. Biology adds three more forms that random splitting cannot prevent.
- Homology leakage. Protein and DNA sequences come in families. Split a sequence dataset at random and close relatives of the test sequences end up in the training set, so the model needs only to recognise a familiar family. The result looks like prediction and is really lookup. The fix is to cluster sequences by similarity and keep whole clusters together, so a test sequence has no near relative in training.
- Patient-level leakage. One patient often contributes several samples: repeat biopsies, multiple images, several cells, left and right eye. If rows from the same patient fall on both sides of the split, the model can identify the patient rather than the disease. Split by patient, not by row.
- Time leakage. Predicting something that unfolds over time, such as which patients will deteriorate, using measurements recorded after the event, or training on later data and testing on earlier data. Clinical records are full of variables that are filled in only once the answer is known: a treatment code, a transfer to intensive care, even which scanner was used. Split by time, and ask when each variable was actually recorded.
Two more that catch people out: duplicate records, since public databases hold the same sequence or sample under several identifiers, and target leakage through a proxy, such as a hospital identifier that happens to correlate with the diagnosis.
How do you spot leakage you did not plan for? Two signs are worth checking. The first is a score that is too good: if a task that experts find hard comes out at 99%, look for leakage before celebrating. The second comes from asking which features the model relies on most, as you did in Machine learning on an expression table: from patients to marker genes. If the top features include a sample identifier, a date, a plate number or a count of how many tests a patient had, the model has probably found a shortcut. For sequence data, tools such as CD-HIT and MMseqs2 cluster sequences by identity, and the threshold you choose, often somewhere between 30% and 50% identity for proteins, decides how strict the family split is.
Benchmarks, baselines and test-set reuse
A benchmark is a shared dataset with a fixed task, used to compare methods. Benchmarks are valuable and they wear out. Every time a community tries new ideas on the same test set, keeps what scores well and publishes it, the collective result drifts upwards without anything really improving. This is test-set reuse, and it is why competitions such as CASP, which you met in AI structure prediction: what it changed, and its limits, hold back answers nobody has seen.
A baseline is the simplest sensible method for the same task. Without one, a number means nothing. For a classifier, start with the share of the largest class: a test set that is 95% healthy makes 95% accuracy worthless. Then try something simple but real, such as logistic regression on a handful of obvious features, or, for sequences, a similarity search of the kind you met in Module 4. A great many deep learning results in biology have turned out to be within the uncertainty of such a baseline.
The protection against test-set reuse is simple to state and hard to keep: set aside a final test set at the start, make every choice of features, model and settings without looking at it, and score it once. The strongest form of testing goes further, to data that did not exist when the model was built, such as patients recruited after the model was frozen. This is called prospective validation, and it is the test CASP applies to structure prediction every two years.
Go deeper: good scores, few true alarms
A model can have strong sensitivity and specificity and still be wrong most of the times it raises an alarm. The reason is prevalence, how common the condition is. Take a test with 90% sensitivity and 95% specificity, used where 1% of patients have the condition. In 10,000 patients, 100 have it, and the test flags 90 of them. Of the 9,900 without it, 5% are flagged wrongly: 495.
So 90 + 495 = 585 alarms, of which only 90 are real. The positive predictive value, the share of alarms that are true, is 90 ÷ 585, about 15%. The same model in a group where 20% have the condition gives a far higher figure. This is why scores measured on a balanced test set, like the 50:50 sets in this module, cannot be carried straight into a clinic. Ask for performance at the real prevalence.
Calibration
A model that outputs probabilities is calibrated when those probabilities mean what they say: among the cases given a 70% chance, about 70% really are positive. Accuracy and AUC do not check this. A model can rank cases perfectly while being systematically overconfident.
Calibration matters most where the number is used to make a decision, which in medicine is nearly always. "A 3% chance of this variant being pathogenic" invites a different action from "a 30% chance", so if the model's 3% is really 30%, the ranking being correct is no comfort. Calibration is checked by grouping predictions into bands and plotting the predicted chance against the observed rate in each band, and it can often be repaired on held-out data without retraining the model.
Read a calibration plot against the diagonal line from (0, 0) to (1, 1). Points below the line mean the model is overconfident: cases it calls 80% likely turn out positive less often than that. Points above it mean underconfidence. A common pitfall is to check calibration once and assume it lasts. Calibration depends on prevalence, which, as the box above shows, changes between settings, so a model calibrated in one hospital may be miscalibrated in the next, even if its ranking, and so its AUC from Measuring a model honestly, stays the same.
Reproducibility
A result you cannot rerun is a claim, not a finding. The things that make a result reproducible are unglamorous: the code, in a public repository with a version tag; the data, or an exact description of how to obtain it, including the version of the database; the environment, as a list of packages and versions, as in Module 3; the random seeds; and the splits themselves, saved as files rather than regenerated. Record the date you downloaded any public database, since UniProt, Ensembl and ClinVar all change between releases.
Two less obvious traps. In a notebook, cells can be run out of order, so a result may depend on a variable that a deleted cell once created; before sharing, restart the notebook and run it from top to bottom. And some deep learning calculations on graphics processors are not exactly repeatable even with a fixed seed, so small differences between runs are expected. Report the spread over several runs rather than the best one.
Ethics and bias
A model learns the population it was trained on. Most human genomic data come from people of European ancestry, so polygenic scores and variant classifiers transfer poorly to everyone else, and the gap is not fixed by a larger model. Clinical datasets also carry the health system that produced them: if a group historically received fewer tests, their records show fewer findings, and a model trained to predict "diagnosis recorded" learns the access pattern rather than the disease.
Three practical habits. Report performance separately for each group you have enough data for, and say so when you do not. State the intended use and the populations the model was tested on. And ask what happens when the model is wrong, which is a different question for a research hypothesis than for a clinical decision.
Genomic data raise one further duty. A genome identifies the person it came from and tells you something about their relatives, who never consented. That is why much human genomic data sits in controlled-access archives, such as the European Genome-phenome Archive, where researchers apply for access and agree to conditions of use. Respect those conditions when you share models as well as data, because a model trained on a small cohort can sometimes reveal information about individual participants.
| Ask | Good sign | Warning sign |
|---|---|---|
| How were the data split? | By sequence family, patient or time, and stated explicitly | "Randomly split 80/20", with no more detail |
| What is the baseline? | A simple method, run by the authors on the same split | Only comparisons with other complex models, or none |
| Is the class balance given? | Class sizes reported; a measure suited to imbalance | Accuracy alone on an unnamed mixture |
| Was the test set used once? | Settings chosen by cross-validation on training data | Test scores reported for many variants of the model |
| Are probabilities calibrated? | A calibration check, or probabilities not used for decisions | Confident probabilities with no check |
| Can it be rerun? | Code, data version, environment, seeds and splits available | "Available on reasonable request" |
| Who was it trained on? | Population described; results broken down by group | No description of the cohort |
| Was it tested elsewhere? | An external dataset, from another centre or cohort | One dataset, one split, one number |
How to read the table. Work down the first column while reading a paper or a preprint; each row is one question to answer from the methods section. The middle column is what a careful study reports, the right column is what is often written instead. A warning sign is not proof of a mistake, but two or three together mean the headline number should be treated as an upper limit.
The ancestry gap is documented in Popejoy & Fullerton 2016, Nature, which found that about 81% of participants in genome-wide association studies up to 2016 were of European descent. Martin et al. 2019, Nature Genetics, showed that polygenic scores built mainly from such data predict several times less accurately in people of African ancestry, with East Asian ancestry also affected. The share of non-European participants has grown since, so cite these figures with their year. The underlying point still holds: report the ancestry of training and test cohorts, and do not assume a score transfers.
Leaky and proper splits, side by side
This block needs numpy and scikit-learn. The data are made up so that the right answer is known: 400 imaginary proteins in 40 families, where the label was assigned to each family at random. Because the labels are random, nothing in a sequence can really predict them, and an honest evaluation must land at chance.
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score, KFold, GroupKFold
rng = np.random.default_rng(1)
n_families, per_family, n_features = 40, 10, 20
family = np.repeat(np.arange(n_families), per_family) # 400 made-up proteins
centres = rng.normal(0, 1, size=(n_families, n_features)) # each family has its own profile
X = centres[family] + rng.normal(0, 0.2, size=(len(family), n_features)) # close relatives
y = rng.integers(0, 2, size=n_families)[family] # label set per family, at random
model = KNeighborsClassifier(n_neighbors=3)
leaky = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0))
proper = cross_val_score(model, X, y, cv=GroupKFold(5), groups=family)
print("Random split, accuracy: ", leaky.mean().round(3))
print("Family split, accuracy: ", proper.mean().round(3))
print("Share of the larger class:", round(max(y.mean(), 1 - y.mean()), 3))What this does
family = np.repeat(np.arange(40), 10)gives 400 proteins, ten in each of 40 families.centresholds one random 20-number profile per family; adding a small amount of noise (scale 0.2) to the right centre gives each protein a profile very close to its relatives'. This stands in for the resemblance of real family members.y = rng.integers(0, 2, size=40)[family]draws a label 0 or 1 per family and gives it to all ten members. The labels have nothing to do with the profiles.KNeighborsClassifier(n_neighbors=3)predicts by majority vote of the three most similar proteins: the simplest possible stand-in for a similarity search.KFold(5, shuffle=True)splits the 400 rows at random.GroupKFold(5)withgroups=familykeeps every family whole, so no test protein has a relative in training.
Reading the output
The random split reports perfect accuracy, 1.000. The family-aware split reports 0.480, below the 0.525 you would get by always guessing the larger class. The honest answer is "no better than chance", which is exactly right, because the labels were random by construction. The difference between 1.000 and chance is pure leakage: in the random split, nearly every test protein had relatives in the training set, and a vote among its three nearest neighbours simply read their shared label.
Real datasets rarely leak this completely, but the mechanism is the same, and the lesson transfers: when a split-aware evaluation drops sharply, the drop is the honest number.
On real proteins the families are not given in advance; you create them by clustering, and the identity threshold sets how far a test protein must be from anything in training. A stricter threshold gives a lower score and a more honest picture of how the model will do on a genuinely new protein. Which threshold is right depends on the intended use. If the model will only ever be applied to well-studied families, a looser split may match reality; if it is meant to annotate newly sequenced organisms, the strict split is the one to report.
How to read the figure. Each circle is a protein and each colour a family. In the top row, the same colours appear on both sides of the split, so the model has seen close relatives of every test protein. In the bottom row, a colour is on one side or the other. The accuracies on the right are the two numbers from the output.
Worked example: critique an abstract
Worked example · reading a claim like a reviewer
"We present DeepSepsis, a deep learning model that predicts sepsis from routine blood tests. We collected 42,000 records from one hospital, split them 80/20 at random, and trained a neural network with 14 million parameters. The model reached 99.1% accuracy and an AUC of 0.97, outperforming previously published models. DeepSepsis could be deployed to alert clinicians early. Code and data are available on reasonable request."
Step by step. Take the table above, row by row.
- Split. "Split 80/20 at random", on hospital records. One patient usually has many records, so the same patients are almost certainly on both sides: patient-level leakage. Nothing is said about when each blood test was taken relative to the sepsis diagnosis, so time leakage is possible too.
- Baseline. It compares only with other published models, on its own data. No simple baseline, such as logistic regression on a few vital signs, or an existing clinical score.
- Class balance. Sepsis is uncommon in routine records. If 1% of records are sepsis, predicting "no sepsis" every time scores 99% accuracy, which is almost exactly the figure reported. Accuracy is the wrong measure here, and the class sizes are not given.
- Test set. One split, one number, from one hospital. No external validation, so nothing shows the model would transfer.
- Calibration. The model is proposed for alerting clinicians, but no calibration is reported.
- Reproducibility. "Available on reasonable request" is not availability.
- Population. One hospital, no description of the patients, no results by group.
What the answer means. The abstract may describe careful work, but as written it supports no conclusion about sepsis prediction. Seven of the eight rows raise a warning sign, and the 99.1% is consistent with a model that has learned the base rate. The constructive version of this critique is a list of requests: split by patient and by time, add a simple baseline and a clinical score, report sensitivity at a fixed alert rate, test on another hospital, show a calibration curve, and publish the code with the splits.
Key point
Judge a claim by how it was tested, not by how high the number is. Ask how the data were split, what the baseline was, how the classes were balanced and whether anyone else could rerun it. A high score with a random split on grouped biological data is the single most common way to be wrong.
Apply the same list to your own work, and apply it before you see your results rather than after. Write down the split, the baseline and the measure in advance; then a disappointing honest number is a finding, not a failure to be explained away.
Further reading · Module 8scikit-learn: common pitfalls: leakage and how to avoid it. · Cross-validation with groups. · Probability calibration. · All reading for this course
Check your understanding
1. A study predicts disease from biopsy images, splitting images at random; many patients contributed several images. What is the main problem?
2. A classifier reports 96% accuracy on a dataset in which 95% of cases are negative. What does that tell you?
3. A model ranks patients by risk very well (high AUC) but its stated probabilities are far from the observed rates. What is missing?
Up next
Lesson 11
AI for gene editing: designing CRISPR guides