LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 11 lessons

Lesson 2 of 3

Module 1 · Foundations

Health data: the raw material

Health data: the raw material

Health data: the raw material

Every medical AI model is built from data collected for another purpose: care, billing, research. Knowing where it comes from, and how it can mislead, is the most useful skill in judging AI.

⏱ 12 min readModule 1 · FoundationsFree
Lesson at a glance3 steps · about 12 min
  1. Six kinds of health data
  2. Codes and standards
  3. How data misleads

Six kinds of health data

Almost none of this data was collected to train AI. It was recorded to treat a patient, to claim payment, to run a service or to answer a research question. That history shapes what the data contains and what it leaves out, and so it shapes every model built from it.

In What AI is, and what it is not you saw that a machine learning model takes its weights from its training data. This lesson looks at that data itself: what kinds there are, how they are coded, and the ordinary ways in which they mislead.

How to read the table below

Each row is one kind of data. The middle column gives familiar examples, and the last column the kind of AI task that data most often feeds. Read across a row and ask: where in my own service is this data produced, and who records it?

That question matters because the person and the system that record the data decide its quality. A heart rate typed in by a nurse at a set time differs from one streamed every second by a monitor: the first is sparse but checked, the second is dense but includes moments when a lead slipped off. A model trained on one kind may not cope with the other, even though both are called "heart rate".

TypeExamplesTypical AI use
Structured recordsDiagnoses, lab results, vital signs, prescriptionsRisk scores, early warning
ImagesX-ray, CT, MRI, retinal photos, pathology slidesDetection and triage
SignalsECG, EEG, continuous monitoringRhythm and event detection
TextClinic letters, discharge summaries, reportsSummaries, coding, extraction
OmicsTumour sequencing, expression profilesSubtyping, treatment selection
WearablesHeart rate, activity, sleepMonitoring outside hospital

Terms in the table

  • Structured data sits in fixed fields with set formats: a potassium of 4.2 mmol/L, a heart rate of 96, a diagnosis code. It is the easiest kind for a computer to use, because every value is already in a known place.
  • Unstructured data, such as the free text of a clinic letter, has no fixed fields. The same fact can be written in many ways ("no chest pain", "denies CP", "chest pain absent"), so a model must first work out what the text means.
  • Signals are measurements taken many times a second or minute. An ECG records the heart's electrical activity as a continuous trace; an EEG does the same for the brain.
  • Omics data measures many molecules at once, for example sequencing a tumour's DNA to find its mutations, or an expression profile that measures how active thousands of genes are.
  • Triage means sorting cases by urgency, so that the most urgent are seen first. Subtyping means dividing one disease into groups that behave differently.

Each kind also brings its own practical burdens. Images are large files, and labelling them needs specialist time. Text contains names, addresses and other identifying details scattered through it, so it is harder to make anonymous than a table of numbers. Wearable data is collected by consumer devices whose accuracy varies between makes and between users, and it comes mostly from people who chose to buy one, who may not resemble the patients a service most needs to reach.

The rows are not fixed boxes. A single admission can produce all six: blood results, a chest X-ray, a monitor trace, a discharge letter, perhaps a tumour report and, increasingly, data from the patient's own watch.

Codes and standards

Records use shared codes so that different systems mean the same thing. In the NHS, clinical terms are recorded with SNOMED CT, and hospital diagnoses are classified with ICD-10. Medical images are stored in the DICOM format, and HL7 FHIR is a modern standard for exchanging records between systems. Models trained on one hospital's codes can break when coding habits differ elsewhere.

The value of a standard is that a field means the same thing wherever it appears. Without one, a model built at one site would need its inputs translated by hand before it could be used anywhere else, and every translation is a chance for error. Even with standards, sites use them differently: which codes are chosen, how quickly they are entered, and whether a field is filled in at all. A standard fixes the language, not the habits of the people using it.

What each standard does

  • SNOMED CT is a very large clinical vocabulary. Each concept, such as a symptom, a finding or a procedure, has its own code, so "heart attack" and "myocardial infarction" typed in two systems can point to the same concept.
  • ICD-10, the International Classification of Diseases, 10th revision, is maintained by the World Health Organization. It groups diagnoses into categories, and is used for statistics and for counting hospital activity. Because it groups conditions, it is coarser than SNOMED CT.
  • DICOM (Digital Imaging and Communications in Medicine) stores each image together with information about it: the patient, the date, the type and make of scanner and its settings. That extra information is useful, and it is also one way a model can learn the scanner instead of the disease.
  • HL7 FHIR (Fast Healthcare Interoperability Resources) describes records as standard building blocks, such as a patient, an observation or a medication, so that systems from different suppliers can exchange them.

To see how coding habits matter, imagine two hospitals that treat the same kind of patients. One codes many infections as sepsis; the other usually codes only the underlying infection, such as pneumonia. A model trained to predict "sepsis" at the first hospital will appear to fail at the second, even though the patients are alike. The difference is in the recording, not the illness.

A related pitfall is the unit or reference range. One laboratory may report a blood result in different units from another, or use an analyser that reads slightly higher. If values from several sites are pooled without checking, the model can learn which hospital a patient attended from the numbers alone. Before training, each input should be checked for units, ranges and changes of method across sites and years.

Health data: the raw material · LSMA free course Four stages from left to right. Care happens: a patient is unwell, a clinician worries and a test is ordered. The care is recorded with shared codes: SNOMED CT, ICD-10, DICOM and HL7 FHIR. A dataset is built as a grid of patients and variables, with gaps where tests were not done. A model learns from it both the illness and the care given. 1 · CARE2 · RECORD3 · DATASET4 · MODEL Patient is unwell Clinician worries Test is ordered SNOMED CTICD-10DICOMHL7 FHIR shared codes rows = patients gaps = not tested the illness + the care given the model learns both
Read left to right. In the dataset, the orange cells are a test such as lactate that was done only for some patients; the dashed cells are where it was not ordered. That pattern of gaps reflects clinical worry, and a model can learn it. Schematic, not to scale.
Go deeper: codes lose detail when they are converted

Records are often translated from one code system to another, for example from the detailed SNOMED CT terms used at the bedside to the broader ICD-10 categories used for hospital statistics. Many detailed concepts map to one category, so the translation throws information away and cannot be reversed. A model trained on the converted codes never sees the detail that was dropped.

Codes also change over time. When the United States switched from ICD-9 to ICD-10 for diagnoses in October 2015, the counts of some conditions jumped or fell overnight because the categories were drawn differently, not because patients changed. A model trained across such a change, or used after one, can see a false trend. Before trusting a pattern in coded data, check which code system and which version was in use for each year.

How data misleads

  • Missing data is rarely random. A lab test is ordered because a clinician is worried, so whether a test was done carries information on its own.
  • Labels are noisy. A diagnosis code recorded for billing may not match what the patient actually had.
  • Practice changes. New guidelines, tests or coding rules change the data over time, even if patients do not.
  • Shortcuts. A model may learn the scanner, the hospital or a ruler in the image rather than the disease.

Each problem, with an example

  • Missingness. "Missing" here means a value that was never recorded. In a well patient, nobody checks blood gases, so the field is empty; that empty field is itself a sign the patient looked well. The worked example below puts numbers on this.
  • Noisy labels. A label is the answer a model is trained to predict. If the label is "had a urinary infection", taken from a diagnosis code, some coded patients will not have had one, because the code was entered early on suspicion and never revised. The model then learns from wrong answers.
  • Changing practice. If a hospital introduces a new, more sensitive blood test, more patients will be diagnosed with the condition it detects from that month onwards. A model trained on the earlier years sees an apparent rise in disease that is really a change in testing.
  • Shortcuts. A shortcut is a feature that predicts the label in the training data for a reason unrelated to the disease. Clinicians often photograph a worrying skin lesion next to a ruler, to record its size. So in a collection of skin photos, rulers appear more often beside lesions that turned out to be cancer, and a model can learn "ruler means cancer".

The usual mistake is to treat these as rare technical faults. They are normal features of routine data, and a careful evaluation assumes they are present until shown otherwise.

Each problem has a matching check. For missingness, compare how often each value is recorded in each group of patients, and ask whether a model would still work in a service that tests more or less often. For noisy labels, take a sample of records and have someone read the full notes to see how often the label is right. For changing practice, plot each input and the outcome over time and look for sudden steps. For shortcuts, look at what the model attends to, and test it on data where the suspected shortcut is absent, such as skin photos without rulers.

None of these checks is complicated, but they take time and need people who know how the data was produced. That is why the most useful members of an AI project team are often not the programmers but the clinicians, coders and data managers who can explain why a field looks the way it does.

Worked example · when missingness predicts

In a dataset of 10,000 admissions, lactate (a blood test) was measured in 1,500.

deaths among patients with a lactate test: 150 of 1,500 = 10%
deaths among patients without one: 170 of 8,500 = 2%

A model can learn that “lactate was measured” predicts death, without using the lactate value at all. It has learned the doctor's worry, not the biology, and it will behave differently in a hospital with other testing habits. The numbers are for illustration.

Step by step. First check the groups add up: 1,500 tested + 8,500 not tested = 10,000 admissions. In the tested group, 150 ÷ 1,500 = 0.10, or 10%. In the untested group, 170 ÷ 8,500 = 0.02, or 2%. Overall, 150 + 170 = 320 deaths in 10,000 admissions, which is 3.2%.

What the answer means. Patients who had a lactate test died five times as often as those who did not (10% against 2%). Lactate is a marker of poor blood flow to the tissues, but none of that biology is needed to see the difference: the bare fact that someone ordered the test separates the groups. Now imagine a hospital that measures lactate on every admission. There, "lactate was measured" is true for everyone and says nothing, so a model that relied on it would lose much of its apparent skill.

What to do about it. Leaving out the "was it measured" information is not always right either: in a hospital that keeps the same testing habits, it is genuinely predictive. The safer course is to decide on purpose. If the model will be used where testing habits are stable and similar, the signal may be acceptable. If it may move to other settings, or if the model is meant to prompt testing in the first place, a model built on testing habits will defeat its own purpose.

Gene 1Gene 2Gene 3Gene 4Gene 5Gene 6Gene 7Gene 8Gene 9Gene 10Gene 11Gene 12+2−2NormalTumour
Health data as a grid of patients and variables. Patterns, including patterns of missing values, are what models learn from. Illustrative data.

Reading the figure. In a grid like this, each row is usually one patient and each column one variable, such as age, heart rate or a blood result. The colour of each cell shows the value. Look for whole blocks of rows that share a pattern, and for columns that are often empty: both are things a model will notice, whether or not they mean anything clinically.

The idea that the timing of care carries signal has been measured directly. Agniel, Kohane and Weber (BMJ, 2018) studied laboratory tests in the records of two Boston hospitals. Whether a test was ordered, and at what time of day, was linked to three-year survival for most tests, and in many cases carried more information than the result itself. When citing it, remember it is an observational study from one US health system: it shows that the care process leaves strong traces in records, not how large that effect will be in another hospital with other habits.

Key point

Health data records the care that was given, not just the illness. A model learns both, so always ask how the data was produced.

A second example: a model built to predict hospital admission might learn that patients seen in one particular clinic are admitted more often. The reason may be that this clinic sees the most severe cases referred from elsewhere. The model has learned the referral pathway, and its predictions will mean something different in a hospital organised another way.

The next lesson, How a medical AI model is built, shows how data like this is turned into a model, and how to test it so that these problems show up before the model reaches patients rather than after.

Questions to ask of any dataset

  • Why was this data originally collected: care, payment, audit or research?
  • Who decided which tests were done, and what made them decide?
  • How were the labels produced, and by whom? Were they checked?
  • Did practice, coding or equipment change during the years covered?
  • Could anything other than the disease, such as the scanner, the ward or a marker in the image, separate the groups?

Further reading · Module 1Rajkomar, Dean & Kohane 2019: a clear overview of machine learning in medicine, for clinicians. · All reading for this course

Check your understanding

1. Why might “a test was ordered” predict a bad outcome?

2. Which format stores medical images?

3. A skin model learned that rulers in photos predict cancer. What is this?

Up next

Lesson 3

How a medical AI model is built

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.