LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 11 lessons

Lesson 3 of 3

Module 1 · Foundations

How a medical AI model is built

How a medical AI model is built

How a medical AI model is built

Most failures of medical AI happen before the model is trained: in the data, the labels and the way it is tested. This lesson follows a model from question to test.

⏱ 12 min readModule 1 · FoundationsFree
Lesson at a glance3 steps · about 12 min
  1. Five steps
  2. Three piles of data
  3. Where the data comes from matters

Five steps

In Health data: the raw material you saw how routine records carry the marks of the care that produced them. This lesson follows that data through the steps that turn it into a model, and shows where in those steps the problems you met are caught or missed.

  1. Define the question. “Does this retinal photo show referable diabetic retinopathy?” is a question; “Can AI help with eyes?” is not.
  2. Collect data from patients like those the model will meet.
  3. Label it. The model can never be better than these labels.
  4. Train. The model adjusts its internal numbers to match the labels.
  5. Test on data it has never seen, ideally from another hospital.

The steps are listed in order, but real projects loop back. Testing may show that the labels were inconsistent, which sends the team back to step 3, or that the question was too broad, which sends them back to step 1. A project that runs straight from step 1 to step 5 without any loop is unusual, and a report that describes no changes along the way may not be telling the whole story.

What each step involves

  • The question fixes the input (one retinal photo), the output (referable or not) and the patients (people with diabetes attending screening). "Referable" means the eye disease is bad enough that the person should be sent to an eye specialist. A question this precise tells you what data to collect and how to judge success.
  • The data must look like the data the model will meet in use. A model trained only on photos from high-end hospital cameras may struggle with photos from a mobile screening van, even if the patients are the same.
  • The labels are the answers the model learns from, sometimes called the reference standard. For retinal photos, each image might be graded by two trained graders, with a senior grader settling any disagreement. If the graders often disagree, the model is learning from a moving target, and its best possible score is limited by that disagreement.
  • Training is the step where the model's internal numbers, its parameters or weights, are adjusted again and again to reduce its errors on the training images. It is the step people picture when they hear "AI", but it is usually the shortest part of the work.
  • Testing measures how well the finished model does on images it has never seen. That is the only number that predicts how it will behave on the next patient.

Two pieces of work sit around these steps and are easy to forget. Before collecting data, the team needs permission to use it: patients' records are confidential, and using them to build a model needs a lawful basis, approval from the bodies that oversee research or data access, and steps to remove or protect identifying details. After testing, the team needs to decide how the output will be shown and acted on: a probability, a yes-or-no flag, or a highlighted area on an image. That choice changes how clinicians use the model, and it is part of the design, not an afterthought.

A second example shows how the question shapes everything else. Suppose the aim is "predict deterioration on the ward". The team must decide what counts as deterioration (admission to intensive care, a cardiac arrest call, death, or any of these), over what time (the next 12 hours or the next 48), and at what moments the prediction is made (every hour, or each time new observations are entered). Each choice gives a different dataset, a different set of labels and, in the end, a different model. Two models both described as "predicting deterioration" may not be comparable at all.

Three piles of data

The training set teaches the model. The validation set tunes it and decides when to stop. The test set is locked away and used once, to report performance.

Why three piles and not two? While building a model, developers make many choices: how long to train, how complex to make the model, which threshold to use. Each choice is made by checking which option does best on the validation set. After dozens of such choices, the model has been fitted to the validation set too, a little, and its validation score is slightly flattering. The test set has played no part in any choice, so its score is a fair estimate.

The usual mistake is to look at the test set, change something, and look again. Each look turns the test set into another validation set, and the reported score becomes optimistic.

The sizes of the piles are a trade-off. A larger training set gives the model more to learn from; larger validation and test sets give more trustworthy estimates of how well it does. Splits such as 70/15/15 or 80/10/10 are common, but what matters most is the number of patients with the outcome in each pile. A test set with only 20 positive cases cannot say precisely how often the model finds the disease, however many negative cases it holds.

When data is scarce, developers often use cross-validation instead of a single validation set. The training data is divided into, say, five parts. The model is trained five times, each time leaving out a different part and checking on it, and the five results are averaged. This uses the data more efficiently, but it does not replace a locked test set: the averaged score has still been used to make choices, so it is still slightly flattering.

0.000.350.701.051.400102030405060EpochLossEarly stop · epoch 40TrainingValidation
Training error keeps falling, but validation error starts rising: the model has begun to memorise, called overfitting, so training stops there. Illustrative data.

Reading the figure. The horizontal axis shows how long the model has trained, counted in passes through the training data. The vertical axis shows its error. Early on, both lines fall together: the model is learning patterns that hold for all patients. Later, training error keeps falling while validation error turns upwards. That is overfitting: the model is learning details peculiar to the training images, such as the exact appearance of one patient's photo, which do not help with new patients. The best point to stop is near the lowest point of the validation line.

Stopping early is only one defence. Others include using a simpler model with fewer weights, collecting more training data, and regularisation, which means adding a penalty during training that discourages very large weights. All of them aim at the same thing: a model that has learned the general pattern rather than the quirks of the examples it happened to see. Overfitting is a particular risk when the number of inputs is large compared with the number of patients, as it often is with omics data or images.

Worked example · split by patient, not by image

800 skin images from 200 patients, four each. We want 70% for training and 15% each for validation and testing.

split by image: 560 / 120 / 120 images at random
→ the same patient appears in training and testing
split by patient: 140 / 30 / 30 patients = 560 / 120 / 120 images
→ every test patient is new to the model

With an image-level split, the model can recognise a patient's skin rather than the disease, so its test score is too good. This data leakage is one of the most common errors in published medical AI.

Step by step. By image: 70% of 800 = 560 images for training, and 15% of 800 = 120 each for validation and testing; 560 + 120 + 120 = 800. By patient: 70% of 200 = 140 patients, and 15% of 200 = 30 patients each. Because every patient has four images, 140 × 4 = 560 and 30 × 4 = 120. The image counts come out the same; what differs is which images travel together.

What the answer means. With a random image split, a patient's four images are scattered, so most test patients also have images in training. The model may score well by recognising a familiar mole, skin tone or camera angle. With a patient split, every test image comes from someone the model has never seen, which is the situation it will face in clinic. The same rule applies whenever one patient contributes several records: several admissions, several scans or several blood tests.

How to spot it in a paper. Look in the methods for the words "split at the patient level" or "no patient appeared in more than one set". If a paper reports numbers of images in each set but never numbers of patients, ask how the split was done. A test score far higher than that of similar published models is another prompt to check.

How a medical AI model is built · LSMA free course Top row: images are shared out at random into training, validation and test sets, and the four images of patient A end up in all three, so the test set contains a patient the model has seen. Bottom row: whole patients are shared out, and all four images of patient A sit in the test set, so there is no leakage. 1 · SPLIT BY IMAGE, AT RANDOM2 · SPLIT BY PATIENT Training · 70%Validation · 15%Test · 15% Training · 70%Validation · 15%Test · 15% Patient A is intraining and test→ leakage All of patient Ais in the test set→ no leakage
Each small square is one image; the orange squares are the four images of one patient, patient A. In the bottom row each 2 × 2 block is one patient’s four images, kept together. Scaled down from the worked example to 20 patients and 80 images, with the same 70/15/15 split. Schematic, not to scale.
Go deeper: leakage that a patient split does not catch

Splitting by patient fixes one kind of leakage, but there are others. The first is leakage in time. If a model meant to warn of sepsis is allowed to use whether antibiotics or a blood culture had been ordered, it is using information that only exists because a clinician already suspected sepsis. It then predicts what staff already know. Every input must be something available at the moment the prediction would be made.

The second is leakage through preparation. Steps such as scaling values or choosing which variables to keep should be worked out on the training data only. If they are worked out on the whole dataset first, the test set has quietly influenced the model. The third is duplicates: the same scan can appear in two public datasets under different names, and so end up in both training and testing.

Where the data comes from matters

A model trained in one hospital learns that hospital: its scanners, its patients, its coding. Elsewhere, performance often drops. This dataset shift is why external validation, on data from other sites, is such strong evidence.

Dataset shift means that the data a model meets in use differs from the data it learned from. It can take several forms. The patients may differ: older, sicker, or from different ethnic groups. The equipment may differ: another make of scanner, other settings. The recording may differ: other coding habits or other tests. And the proportion of patients with the disease may differ, which changes how often a positive result is right, as Module 2 shows.

External validation means testing the finished, unchanged model on data from somewhere it was not built, and reporting the result. A related check, temporal validation, tests the model on later patients from the same hospital, which shows whether it survives changes over time. If a model's score holds up at several other hospitals, you can be more confident it has learned the disease and not the setting.

When performance drops at a new site, it does not always mean the model is useless there. Sometimes the model still ranks patients correctly, putting higher-risk patients above lower-risk ones, but its predicted risks are too high or too low overall, because the new site's patients have more or less of the disease. This can sometimes be corrected by adjusting the model on a sample of local data, a step called recalibration. Any such adjustment is a change to the model and needs to be tested again before use.

A practical pitfall is to treat "tested at another hospital" as a single box to tick. Ask which hospitals, how similar they were to the development site, and whether they resemble your own. A model validated at three large teaching hospitals may still behave differently in a small district hospital with older equipment and a different mix of patients.

A clear demonstration of hospital-level shortcuts is Zech and colleagues (PLOS Medicine, 2018). They trained a deep learning model to detect pneumonia on chest X-rays from several US hospital systems. Performance on images from a hospital system not used in training was worse than internal performance in most of their comparisons, and a model could tell which hospital system an X-ray came from almost perfectly. When citing it, note that the labels were taken automatically from report text, so some were wrong, and the study was retrospective. It shows that shortcuts are possible and common, not how much any particular product will drop.

Key point

Find out how a model was tested. Same hospital, random split? Caution. Separate hospitals, patient-level split? Much more convincing.

A second example: a model that predicts readmission is trained on 2018 to 2020 data from one trust and tested on a random 20% of the same admissions. Its test score says little about 2025 patients in another trust. Ask instead how it did on a later year, or at a different trust, with each patient appearing in only one of the piles.

What to take from Module 1

  • Most medical AI is supervised machine learning: weights learned from labelled examples.
  • Routine health data records care as well as illness, and models learn both.
  • A good model starts with a precise question and trustworthy labels, and is judged on data it has never seen, split by patient, ideally from elsewhere.

Module 1 has given you the parts: what a model is, what it learns from and how it should be tested. Module 2, Judging a model, builds on them by putting numbers to the question every buyer and user must ask: how good is this model, and how strong is the evidence for it? It starts with Measuring performance, where the test set you met here produces the figures that appear in every product leaflet and paper.

End of Module 1: FoundationsYou can stop here: this module is complete on its own. Or carry on to Module 2, Judging a model (Intermediate).

Further reading · Module 1Rajkomar, Dean & Kohane 2019: a clear overview of machine learning in medicine, for clinicians. · All reading for this course

Check your understanding

1. Which data set is used only once, to report final performance?

2. Images of the same patient are in both training and test sets. The risk?

3. What is the strongest evidence that a model will work in your hospital?

Up next

Lesson 1

Measuring performance

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.