LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 11 lessons

Lesson 4 of 6

Module 3 · AI in practice

Safety, bias and regulation

Safety, bias and regulation

Safety, bias and regulation

A model that works on average can still fail particular patients. This lesson covers how bias enters, how AI is regulated, and the questions to ask before any tool reaches a patient.

⏱ 12 min readModule 3 · AI in practiceFree
Lesson at a glance3 steps · about 12 min
  1. How bias gets in
  2. How AI is regulated
  3. Ten questions to ask before using a tool

How bias gets in

In this lesson, bias means a systematic error that falls more heavily on some groups of patients than on others. It is not about anyone's intentions: a model can be biased even if everyone who built it meant well, because it copies patterns from the data it was given.

Much of this lesson draws together ideas you have already met. The shortcuts and noisy labels of Health data: the raw material, the dataset shift of How a medical AI model is built and the measures of Measuring performance all reappear here, now asking a sharper question: who bears the errors?

  • Unrepresentative data. A skin-lesion model that learned mostly from lighter skin may perform worse on darker skin.
  • Biased measurements. Pulse oximeters have been shown to miss low oxygen levels more often in Black patients. A model trained on those readings inherits the error.
  • The wrong target. A widely used US algorithm predicted future healthcare costs as a stand-in for health need. Because less had been spent on Black patients, it underestimated how ill they were. A 2019 study exposed this, and showed that predicting health need instead of cost greatly reduced the bias.

Why each one happens

  • Unrepresentative data arises because a model learns most about the patients it sees most. Skin lesions can look different on darker skin, and if those images are rare in training, the model has had little chance to learn them. The same applies to children, very old patients or rare conditions whenever they are scarce in the data.
  • Biased measurements start before any model exists. A pulse oximeter estimates oxygen levels by shining light through the skin, usually of a finger, and skin pigment affects how that light is absorbed. If a device reads too high in some patients, a model trained on its readings will treat those patients as better oxygenated than they are, and no amount of data from the same device will correct it.
  • The wrong target comes from using a proxy: something readily measured that stands in for what you really care about. Cost is recorded for every patient, while "health need" is hard to measure, so cost was used instead. But spending depends on access to care as well as on illness. Where two patients were equally ill and less had been spent on one of them, the model ranked that patient as lower need, and so less likely to be offered extra support.

Two further routes are worth knowing. Label bias arises when the outcome itself is recorded less reliably for some groups: if a condition is diagnosed later or less often in women, for example, a model trained on recorded diagnoses learns to expect it less in women and so flags them less. Deployment bias arises after the model is built: a tool may work equally well for everyone, yet be installed only in better-funded sites, or be used less with patients who need an interpreter, so its benefits are shared unequally.

Each route suggests its own remedy. Unrepresentative data calls for collecting more examples from under-represented groups, or at least for reporting clearly whom the model was tested on. Biased measurements call for checking the instruments themselves before trusting what they feed into a model. The wrong target calls for asking, at the very first step of building a model, whether the label really measures what the tool is meant to serve. None of these remedies is a technical patch applied at the end; each belongs to a stage of the work described in Module 1.

Safety, bias and regulation · LSMA free course Four stages from left to right. Patients: most training patients come from one group. Measurement: a reading such as pulse oximetry is less accurate in some groups. Target: the model predicts cost, which is not the same as need. The model learns from all three and inherits each bias. 1 · PATIENTS2 · MEASURE3 · TARGET4 · MODEL Most patientsfrom one group Readings lessaccurate insome groups needcost Predicts cost,not need Learns all threeand repeats them each stage can add bias before any algorithm is chosen
The three examples from the list above, in the order data passes through them: who is included, how they are measured, and what the model is asked to predict. None of them is a fault in the algorithm, and none is fixed by a better one. Schematic, not to scale.
0%25%50%75%100%Sensitivity91%Group A89%Group B92%Group C74%Group D90%Group EOverall 88%
Overall sensitivity is 88%, but one group is much lower. Reporting by subgroup reveals it. Illustrative data.

Reading the figure. The figure shows the sensitivity within each group of patients alongside the overall figure, which is worked out across everyone. A subgroup is any group defined by a shared characteristic, such as sex, age band, ethnicity or the hospital where patients were seen. Compare each group with the overall figure: the gap for the lowest group is what a single headline number hides. The worked example below uses its own illustrative numbers, which is why its overall figure is slightly different.

Worked example · what an average hides

majority group: 828 of 900 caught = 92%
smaller group: 60 of 100 caught = 60%
overall: 888 of 1,000 = 89%

An 89% headline hides that four in ten patients in the smaller group are missed. Ask for results by sex, age, ethnicity and site.

Step by step. Majority group: 828 ÷ 900 = 0.92, or 92%. Smaller group: 60 ÷ 100 = 0.60, or 60%. Overall: 828 + 60 = 888 caught out of 900 + 100 = 1,000 patients with the condition, and 888 ÷ 1,000 = 0.888, which rounds to 89%. Missed: 900 − 828 = 72 in the majority group (8%), and 100 − 60 = 40 in the smaller group (40%).

What the answer means. The overall figure sits close to the majority group's figure because that group makes up 90% of the patients. A patient from the smaller group is five times as likely to be missed (40% against 8%). The smaller the group, the less it moves the average, so the patients most at risk of being failed are the ones a headline number hides best. Subgroups also need to be large enough for their results to be reliable; a sensitivity worked out from ten patients can swing widely by chance.

Groups also overlap. A model may do well for women overall and for older patients overall, yet poorly for older women with a particular skin type. Checking every combination quickly runs into groups too small to measure, so evaluators usually choose in advance a set of subgroups that matter most for the condition and the population, and report those with their uncertainty.

Go deeper: fairness measures can pull against each other

"Fair" can mean several things, and they cannot always all be true at once. Take a model with 90% sensitivity and 90% specificity in two groups of 1,000 people. In group A, 20% have the disease: 180 of 200 are caught and 80 healthy people are flagged, so the PPV is 180 ÷ 260, about 69%. In group B, only 5% have it: 45 of 50 are caught and 95 are flagged, so the PPV is 45 ÷ 140, about 32%.

The model treats both groups equally by sensitivity and specificity, yet a flag means much less in group B. Adjusting the threshold to equalise the PPV would make sensitivity unequal instead. Mathematical work in 2016 and 2017 showed that, when prevalence differs, no imperfect model can equalise all of these at once. Choosing which kind of fairness matters is a clinical and ethical decision, and it should be stated openly.

How AI is regulated

Software that diagnoses, predicts or guides treatment is usually regulated as a medical device. In the UK this is overseen by the MHRA; in the US by the FDA, which publishes a list of authorised AI-enabled devices. The EU AI Act, in force since 2024, adds requirements for high-risk AI, which includes AI medical devices. Intended use, evidence and post-market monitoring matter as much as the algorithm. Patient data is also covered by data protection law, in the UK the UK GDPR.

Regulators do not treat all medical software alike. Devices are placed in risk classes, and the class depends largely on how serious the decisions are that the software informs and how directly it drives them. Software that only displays information for a clinician to interpret usually sits in a lower class than software that gives a diagnosis for a serious condition with no one checking it. A higher class means more evidence is needed and more independent assessment before the product can be sold.

The terms in that paragraph

  • Medical device. Most people picture equipment such as a pacemaker or a blood pressure cuff. Software can be a medical device on its own when its purpose is medical, for example to detect a disease on a scan. Software that only books appointments usually is not.
  • The MHRA is the Medicines and Healthcare products Regulatory Agency, the UK regulator for medicines and medical devices. The FDA is the US Food and Drug Administration.
  • Intended use is the maker's statement of what the device is for, for which patients and in whose hands. A tool approved to triage adult chest X-rays is not approved for children's X-rays, even if it produces an answer for them.
  • Post-market monitoring is the maker's duty to keep watching how the device performs once it is in use, and to report and act on problems. It is the regulatory side of the drift monitoring in Deploying AI in a hospital.
  • Changes after approval. A model that is retrained or updated is, in effect, a changed device. Traditionally each significant change needed fresh review. Some regulators now allow a maker to agree in advance a plan describing which kinds of update it may make, how each will be tested, and how users will be told, so that well-controlled improvements need not wait for a new approval.
  • UK GDPR is the UK's general data protection law. It treats health data as particularly sensitive, which is why Using AI assistants safely at work comes back to it.

The usual mistake is to think regulation settles the question of whether to use a tool. Approval shows that a tool met requirements for its stated use; your hospital still has to ask whether that use, those patients and that evidence match its own situation.

Regulation is also changing quickly, and dates and details in this area go out of date. When a decision depends on a specific rule, check the regulator's own current guidance rather than relying on a summary, including this one.

The EU AI Act is Regulation (EU) 2024/1689. It entered into force on 1 August 2024, but its duties apply in stages. For AI that is a medical device needing a notified body's assessment, the high-risk rules were first due to apply from August 2027. A 2026 amendment (the "Digital Omnibus") delayed that date, so they now apply from 2 August 2028, alongside the existing medical device regulations, not in place of them. On the US side, the FDA's list of AI-enabled devices had passed 1,000 entries by 2025, most of them in radiology, and it keeps growing. When citing that list, treat the count as approximate and dated, as it is updated often. Most entries were cleared through the 510(k) route, which relies on similarity to an existing device. A listing is not evidence of better patient outcomes.

Ten questions to ask before using a tool

1What exactly is it intended to do, and for which patients?
2Is it approved as a medical device for that use, here?
3What data was it trained on, and does it resemble our patients?
4Was it validated externally, how high up the evidence ladder, and has NICE assessed it?
5What are its sensitivity, specificity and PPV at our prevalence?
6How does it perform in each subgroup?
7Is it well calibrated?
8Who acts on the output, and what happens when it is wrong?
9Where does patient data go, and who can access it?
10How will performance be monitored after go-live?

How to use the ten questions

Each question draws on a part of the course, so you can go back to the right lesson when an answer is unclear:

  • Questions 1 and 3: defining the use and the data, Health data: the raw material and How a medical AI model is built.
  • Question 2: regulation, this lesson.
  • Question 4: external validation and the evidence ladder, How a medical AI model is built and Evidence: from validation to trials.
  • Questions 5 and 7: performance at your prevalence and calibration, Measuring performance.
  • Question 6: subgroups, this lesson.
  • Questions 8 and 10: workflow, human factors and monitoring, Deploying AI in a hospital.
  • Question 9: data protection, this lesson and Using AI assistants safely at work.

A vague answer to any one of them is useful information in itself. The case study at the end of this module asks you to apply all ten to a fictional tool.

Some answers should prompt particular caution. A maker who gives only an AUC or an accuracy figure has not answered question 5. "Validated on over a million images" says nothing about question 4 if they all came from one source. "Our tool is fair" with no table of subgroup results has not answered question 6. And "the tool improves over time" raises question 10 rather than answering it: improves how, checked by whom, and with what notice to users?

The questions are not a pass-or-fail test. A tool with modest evidence may still be worth using with extra local checks, a silent pilot and close monitoring. What matters is that the gaps are known and planned for before patients are affected, not discovered afterwards.

Key point

Fairness and safety are measured, not assumed. If a vendor cannot show subgroup results and monitoring plans, that is the answer.

A second example: a vendor says its tool "does not use ethnicity, so it cannot be biased". Leaving a variable out does not remove bias, because other inputs, such as postcode, the hospital attended or a measurement like pulse oximetry, can carry the same differences. The only way to know is to measure performance in each group.

Further reading · Module 3Obermeyer et al. 2019: a widely used algorithm that was biased by its choice of label. · All reading for this course

Check your understanding

1. Where does bias in medical AI most often come from?

2. A model reports 90% sensitivity overall. What next?

3. Which UK body regulates AI software that is a medical device?

Up next

Lesson 5

Using AI assistants safely at work

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.