LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 11 lessons

Lesson 3 of 6

Module 3 · AI in practice

Deploying AI in a hospital

Deploying AI in a hospital

Deploying AI in a hospital

A good model can still fail in practice. This lesson covers what happens after the algorithm: fitting it into the workflow, the people who use it, and watching it once it is live.

⏱ 11 min readModule 3 · AI in practiceFree
Lesson at a glance4 steps · about 11 min
  1. Fit the workflow, not the other way round
  2. Human factors
  3. Monitoring after go-live
  4. Governance in the NHS

Fit the workflow, not the other way round

The earlier lessons of this module looked at what AI tools do and how well they do it. This lesson asks what happens when a hospital switches one on. Most failures in practice come not from the algorithm but from the way it is fitted into daily work, watched over time and owned by someone.

  • Who sees the output, when, and where? An alert in a system nobody opens is no alert.
  • What should they do with it? Every output needs an agreed action, and an agreed way to disagree.
  • What does it replace or add? New work needs time from somewhere.

A workflow is the sequence of steps, people and systems through which a piece of work passes, for example from a patient's X-ray being taken to the report reaching the clinician. Each of the three questions above can be answered concretely. Take a deterioration alert:

  • Who and where: the nurse in charge of the ward, on the screen already used for observations, at the moment the score is calculated. Sending it to a junior doctor's email inbox, read at the end of a shift, would make the same alert useless.
  • What to do: review the patient within an agreed time and follow the existing escalation pathway. "Disagreeing" might mean recording that the patient was reviewed and the alert judged not to need action, with a short reason, so those cases can be studied later.
  • What it costs: if the tool raises 20 alerts a day and each review takes 10 minutes, that is more than three hours of nursing time a day, which has to come from somewhere.

A second example is the imaging triage tool from AI across the patient journey. Its output is useful only if it reorders the list radiologists actually read from. If the flag appears in a separate program that radiologists must open themselves, few will, and the reading order will not change. Getting the flag into the existing worklist usually means connecting the tool to the hospital's imaging and record systems through the standards described in Module 1, which is often the slowest and most costly part of a deployment.

A common pitfall is to judge the workflow on a pilot run by enthusiasts. The staff who volunteer for a pilot tend to be the ones most interested in the tool, most willing to check its outputs and most forgiving of its quirks. When it spreads to every ward and every shift, including nights, weekends and temporary staff, the same tool may be used quite differently. A rollout plan should allow for this and check again once use is routine.

Human factors

Automation bias is the tendency to accept a computer's suggestion even when it is wrong, and to stop looking once it has spoken. Its opposite, ignoring a tool after a few bad alerts, wastes a good one. Training, clear displays of uncertainty and regular review of disagreements help with both.

Human factors is the study of how people interact with equipment, systems and each other, and how to design work so that mistakes are less likely. An example of automation bias: a tool marks one suspected fracture on an X-ray, and the reader, reassured, does not search the rest of the image, so a second, unmarked fracture is missed. The tool did not make that error; the way it was used did.

A "clear display of uncertainty" means showing how confident the tool is, for instance distinguishing a strong flag from a borderline one, so that the reader knows when to look harder. Regular review of disagreements, cases where the clinician and the tool differed, shows over time which of them tends to be right in which situations.

The design of an alert also changes how people respond. An interruptive alert stops the user until it is acknowledged; a non-interruptive one sits on the screen to be read when convenient. Interruptive alerts are noticed but, if they fire often, they quickly cause the alert fatigue described in AI across the patient journey. They are best kept for findings that need action now. Everything else can be shown more quietly, where staff look in the normal course of their work.

A longer-term concern is deskilling: if staff rely on a tool for a task over months or years, they may get less practice at doing it themselves, and find it harder when the tool is unavailable or wrong. Training should cover what to do when the tool is down, and teams may choose to keep some cases read without it, so that skills and a comparison group are both kept up.

The standard reference for automation bias is Parasuraman and Manzey (Human Factors, 2010), who reviewed complacency and bias in the use of automated aids. They found that both occur in experts as well as novices, and that experience or simple instructions do not reliably prevent them. Much of their evidence came from aviation and laboratory tasks, so for healthcare also cite Goddard, Roudsari and Wyatt (Journal of the American Medical Informatics Association, 2012), a systematic review of automation bias in clinical decision support. Both describe tendencies and factors that make them worse, such as heavy workload. Neither gives a fixed error rate that applies to your tool.

Monitoring after go-live

Performance changes when the world does: a new scanner, a new patient group, a change in testing. This drift can be silent, so live tools need ongoing measurement against agreed limits.

"Go-live" is the moment a tool starts being used in real care. Drift is the gradual or sudden change in a model's performance after that point, caused by changes in the data it receives. It is silent because the model keeps producing outputs that look as confident as before. Nothing on the screen says "this image came from a machine I have never seen".

Useful things to monitor include the tool's sensitivity and PPV against confirmed outcomes, how often it alerts, and how often staff override it. An agreed limit is a value fixed before go-live, below which someone must investigate. Agreeing it in advance matters: once a fall has happened, it is tempting to explain it away.

Monitoring also has to cover changes to the tool itself. Makers release updated versions, sometimes with retrained models. An update can fix one problem and create another, so a hospital should know which version is running, be told before it changes, and repeat key checks after each update. The question "which version produced this output?" should always have an answer.

Worked example · catching drift

A fracture-detection tool is audited monthly on 200 confirmed fractures.

months 1–6: about 180 of 200 caught = 90% sensitivity
month 7, after a new X-ray machine: 156 of 200 = 78%
agreed alert limit: below 85% → investigate

Without a monthly audit and a pre-agreed limit, the drop would go unnoticed. The numbers are for illustration.

Step by step. Sensitivity is fractures caught ÷ all confirmed fractures. Months 1–6: 180 ÷ 200 = 0.90, or 90%. Month 7: 156 ÷ 200 = 0.78, or 78%. The limit of 85% corresponds to 0.85 × 200 = 170 fractures caught. Month 7 is 14 fractures below that, and 180 − 156 = 24 fewer fractures caught than in a typical earlier month.

What the answer means. About 24 more patients in that month had a fracture the tool did not flag. The timing points to the new machine: its images may differ in brightness, contrast or processing from those the tool learned from. An investigation would check the images and settings, tell the users so they rely on the tool less until it is resolved, and ask the manufacturer whether the tool supports that machine. The audit itself depends on a reliable reference standard: someone must confirm which patients really had fractures.

Chance or real change? Monthly figures wobble even when nothing has changed. With 200 fractures a month and a true sensitivity of 90%, a single month could fall to around 86% or rise to around 94% by chance alone. That is why the limit sits at 85% rather than just below 90%: a limit too close to the usual value would trigger investigations for ordinary noise, while one too far away would let a real fall go unnoticed for longer. A drop to 78% is well outside the usual wobble, and its timing with the new machine makes a real change far more likely than chance.

Deploying AI in a hospital · LSMA free course Sensitivity of a fracture-detection tool, audited each month on 200 confirmed fractures. Months 1 to 6 sit at about 90 percent, above the alert limit of 85 percent. After a new X-ray machine arrives, month 7 falls to 78 percent, below the limit, which triggers an investigation. MONTHLY AUDIT ON 200 CONFIRMED FRACTURES sensitivity 100% 90% 80% 70% 1 2 3 4 5 6 7 month new X-ray machine limit 85% about 180 of 200 caught, about 90% 156 of 200, 78% investigate
Each point is one month’s audit. The dashed red line is the limit agreed before go-live; the first point below it starts an investigation. Months 1–6 are drawn at 90% for simplicity; real audits wobble a little from month to month. Numbers from the worked example above.
0.000.350.701.051.400102030405060EpochLossEarly stop · epoch 40TrainingValidation
Performance must be tracked over time, just as a model's errors are tracked while it trains. Illustrative data.

Reading the figure. This is the same kind of chart as in How a medical AI model is built, where error was tracked during training. The idea carries over: a model's quality is not fixed when it is finished; it has to be measured repeatedly. The difference is that, after go-live, the measurements come from real patients and confirmed outcomes, and a change in the line is a signal for people to act.

Go deeper: when the tool changes what it is measured against

Monitoring has a catch: once staff act on a tool, the outcomes it is checked against change. Suppose a deterioration alert fires, the team responds quickly, and the patient never deteriorates. In the audit, that alert looks like a false positive, although it may have been the most useful alert of the week. The better the response, the worse the tool can appear.

The reverse also happens. If staff come to rely on the tool, patients it stays silent about may be watched less closely, and some deteriorations then happen that would otherwise have been prevented.

So live results need careful reading. Teams often add checks that do not depend on the patient's later outcome, such as how often alerts are overridden, how alert rates move after changes in equipment or coding, and periodic review of a sample of cases by a clinician.

Governance in the NHS

In England, digital health tools are assessed against the Digital Technology Assessment Criteria (DTAC), covering clinical safety, data protection, technical security, interoperability and usability. Clinical safety standards require manufacturers (DCB0129) and the organisations deploying a tool (DCB0160) to manage its risks, each with a named clinical safety officer. NHS England refreshed DTAC in 2026: a shorter form replaced the old one from 6 April 2026. These standards are revised from time to time, so check the current versions before you rely on them.

Governance means the rules, roles and records by which an organisation makes sure a tool is used safely and responsibly. The five DTAC areas in plain words: clinical safety, whether risks to patients have been found and reduced; data protection, whether personal data is handled lawfully; technical security, whether the system is protected against misuse and attack; interoperability, whether it can exchange data with the systems already in use, using standards such as those in Health data: the raw material; and usability, whether staff and patients can use it as intended.

The two clinical safety standards divide the work. The manufacturer, under DCB0129, assesses the risks of the product itself. The hospital or practice, under DCB0160, assesses the risks of using it in its own setting: its own staff, systems and patients. Both typically keep a hazard log, a list of the ways the tool could cause harm and what has been done about each, and summarise their reasoning in a safety case. The clinical safety officer is the named person who oversees this work and signs it off. Other countries have their own arrangements, but the same questions apply everywhere.

An entry in a hazard log is short and practical. For the fracture tool, one hazard might be "tool unavailable because of a network failure". The possible harm is that urgent X-rays are not prioritised. The controls might be that the system shows clearly when the tool is not running, and that staff then fall back to the reading order used before the tool existed. Each hazard is given a rating for how likely and how serious it is, before and after the controls, and the log is updated whenever something changes or goes wrong.

Governance also covers the end of a tool's life. The plan made before go-live should say who can pause or switch off the tool, how staff will be told, how work will continue without it, and what happens to the data it holds. A tool that cannot be switched off safely has not been deployed safely.

Key point

Plan the human side and the monitoring before go-live. Decide who owns the tool, what “working” means, and what will trigger a review or switch-off.

A second example: before switching on an AI tool that drafts discharge letters, a team agrees that a senior clinician owns it; that "working" means at least the same accuracy as letters written without it, checked on a sample each month; and that any letter found with a wrong medicine will trigger a review, and two in one month will pause the tool. Writing this down before go-live turns vague worries into actions someone will take.

Further reading · Module 3Wong et al. 2021: a widely used sepsis model that performed much worse when tested independently. · All reading for this course

Check your understanding

1. What is automation bias?

2. Sensitivity falls from 90% to 78% after a scanner change. What is this?

3. Which standard covers clinical risk management for the organisation deploying a tool in England?

Up next

Lesson 4

Safety, bias and regulation

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.