LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 11 lessons

Lesson 2 of 6

Module 3 · AI in practice

Large language models in medicine

Large language models in medicine

Large language models in medicine

Chat assistants have reached clinics remarkably quickly. Used well, they save time; used carelessly, they produce confident errors. This lesson shows the difference.

⏱ 11 min readModule 3 · AI in practiceFree
Lesson at a glance3 steps · about 11 min
  1. What a language model does
  2. Good uses, and risky ones
  3. Privacy

What a language model does

In AI across the patient journey you saw that most mature clinical AI reads images or signals for one narrow task. Large language models are different: one general tool, used for many tasks, mostly with text. The ideas from earlier lessons still apply, from training data to checking outputs against a reference, but they need adapting.

A large language model (LLM) is trained on vast amounts of text to predict the next token, a word or part of a word. Repeated many times, this produces fluent paragraphs; further training teaches it to follow instructions.

A token is the unit of text the model works with. Common short words are usually one token each; a long or unusual word, such as a drug name, may be split into several pieces. During training, the model is shown a passage with the next token hidden, makes a guess, and has its internal numbers, its parameters, nudged so that the right token becomes a little more likely. Repeat that over a very large collection of books, websites and articles, and the model becomes very good at continuing text in a way that sounds right.

A model trained only that way would continue your question rather than answer it. So it is then trained further on examples of instructions paired with good answers, often with people rating which of two answers is better. That second stage is what turns a text predictor into a chat assistant that follows requests.

This first stage is the self-supervised learning mentioned in What AI is, and what it is not: the text supplies its own labels, because the hidden next token is the answer. No one has to label anything, which is why the model can learn from far more text than any team could ever annotate. The price is that it learns whatever the text contains, including errors, out-of-date advice and the biases of whoever wrote it.

When you use a chat tool, everything the model can take into account at once, your question, any document you paste in and the conversation so far, is called its context window. The model does not look things up in its training text while it answers. What it learned is spread across billions of parameters, not stored as documents it can open, so it cannot tell you reliably where a statement came from. A common pitfall is to assume a long document pasted into the context will be read with equal care throughout. On very long inputs, details can be missed or mixed up, so key facts should be checked however long the source.

Large language models in medicine · LSMA free course The text so far, "The patient was started on", is split into tokens. The model gives every possible next token a score; "antibiotics" scores highest, followed by "oral", "a", "IV" and "insulin". One token is chosen, added to the text, and the model predicts again. 1 · THE TEXT SO FAR, AS TOKENS Thepatientwasstartedon ? 2 · SCORES FOR THE NEXT TOKEN antibioticsoralaIVinsulin 3 · REPEAT Chosen: antibiotics It is added to the text,then the model predictsthe next token again.
Nothing in this loop looks anything up or checks a fact: the model picks a likely continuation, one token at a time. The bar lengths are schematic, not real scores, and tokens are often parts of words rather than whole words.

The key consequence: an LLM produces text that is likely, not text that is checked. When it does not know, it can still write something plausible. This is called a hallucination.

Here is a typical example. Asked for evidence on a treatment, an assistant may give a reference with real-sounding authors, a suitable journal, a year and a title, none of which exists as a paper. Each part is a likely continuation of "a reference on this topic looks like…", which is exactly what the model was trained to produce. The word "hallucination" can mislead: the model is not malfunctioning when this happens; it is doing what it always does, and this time the likely text happens to be false.

Some tools are connected to a search engine or to documents you supply, and asked to answer from those. This reduces errors, because the model can copy from a real source, but it does not remove them: it can still misread, mix up or add to what it was given.

This approach is often called retrieval-augmented generation: the tool first retrieves passages from a search or a document collection, then generates an answer from them. Its quality depends on both steps. If the search returns the wrong guideline, or an old version, the answer will faithfully repeat the wrong source. When a tool shows which passages it used, check that they say what the answer claims, and that they are current.

Go deeper: why the same question gets different answers

A chat tool does not always choose the single top-scoring token. Most pick at random among the likely ones, weighted by their scores, and a setting often called temperature controls how adventurous that choice is. So the same question can produce different answers on different occasions.

This has two practical effects. One good answer does not show that the next will be good. And getting the same answer twice is not a check, because a model can repeat a confident mistake as easily as a correct fact.

There is also a training cut-off: the model learned from text collected up to a certain date. A guideline revised after that date may be missing, or the model may give the old version with the same confidence, unless the tool is given the current document to work from.

Good uses, and risky ones

Lower risk
  • Drafting a clinic letter from notes, then checked by the clinician
  • Rewriting information for patients in plain language
  • Summarising a long record, checked against the source
  • Ambient scribes that draft notes from a recorded consultation
Higher risk
  • Asking for a dose, diagnosis or interaction without checking a trusted source
  • Asking for references, which may not exist
  • Pasting identifiable patient data into a public tool
  • Letting output reach a patient unchecked

What separates the two columns

Every lower-risk use has two things in common: the source is to hand, and a person who knows the subject checks the result before it is used. An ambient scribe is a tool that listens to a consultation, with the patient's agreement, and drafts the clinical note; the clinician then reads and corrects the draft. The drafting saves time, and the checking keeps it safe.

Each higher-risk use removes one of those safeguards. A dose asked for without a trusted source has nothing to be checked against. A reference may not exist. Identifiable data in a public tool has left your organisation's control. Output that reaches a patient unchecked has skipped the person who should have caught the error. The usual mistake is to move from the left column to the right one gradually: checking every letter at first, then fewer as the tool keeps getting things right.

A second example shows how a task can drift between columns. Rewriting a leaflet in plain language is lower risk when the clinician checks that no meaning has changed. The same request becomes higher risk if the tool is also asked to "add advice on what to do if symptoms get worse", because now it is writing new clinical content rather than rewording checked content. The safest prompts ask the model to work only from material you supply, and the safest habits check additions most carefully of all.

The intended purpose matters for regulation, too. In many countries, software intended to inform diagnosis or treatment can count as a medical device and must meet the rules for one. General-purpose chat tools are usually not approved for that purpose. Using one to decide a dose or a diagnosis puts it to a use it was never assessed for, which is part of why those uses sit in the right-hand column.

Worked example · checking a summary

An LLM summarises a 40-page record into 12 statements. A pharmacist checks each against the source.

supported: 10 · wrong date: 1 · not in the record: 1
needing correction = 2 of 12 = 17%

Ten correct statements make the two errors easy to miss. Treat every output as a draft, and keep a person responsible for what is signed.

Step by step. First check the total: 10 + 1 + 1 = 12 statements. Two need correcting, so 2 ÷ 12 = 0.167, which is about 17%. Put the other way, 10 ÷ 12, about 83%, were supported by the record.

What the answer means. The two errors are of different kinds. A wrong date is a distortion of something real; a statement "not in the record" is an addition, a hallucination. Both are written in the same confident style as the ten correct statements, so nothing in the text warns the reader. That is why checking means comparing each statement with the source, not reading the summary to see whether it sounds right. A wrong date could matter a great deal, for example the date a medicine was stopped.

Why one check is not a measure. One record with 2 errors in 12 statements says little about the tool's error rate in general. The ideas from Measuring performance apply: a fair evaluation checks many records, chosen to match real use, and counts the errors of each kind, with a range of uncertainty. Because a language model's answers have no single right form, evaluations usually count errors statement by statement, as here, or ask experts to rate each output, and they report how often an error would have been harmful, not only how often one occurred.

Claims that LLMs "pass medical exams" usually trace back to work such as Singhal and colleagues (Nature, 2023), which introduced Med-PaLM and reported that a version of Google's PaLM model scored about 67.6% on US licensing-style multiple-choice questions, ahead of earlier systems. Doctors also rated the answers, and found gaps. Later models scored higher: Med-PaLM 2 reached 86.5% (Singhal and colleagues, Nature Medicine, 2025), and GPT-4 with careful prompting passed 90% (Nori and colleagues, 2023). When citing exam scores, be careful. Multiple-choice questions test textbook knowledge, not safe work with real patients and messy records. Some test questions may also have appeared in training text. Models change quickly, so any score dates fast and says little about the tool in front of you.

A better question than "did it pass an exam?" is the one from Evidence: from validation to trials: which rung has this tool reached for the task I want to use it for? For drafting notes or letters, useful studies compare the time staff spend and the errors that reach the final, signed document, with and without the tool.

Privacy

Anything typed into a public chat tool may be stored outside your organisation. Use only tools your organisation has approved for patient information, and remove identifiers when the task does not need them.

An identifier is any detail that could point to a particular person, alone or together with other details: a name, an NHS number, a date of birth, an address, or even an unusual condition in a small town. A public tool is one offered to anyone, under the provider's own terms, which may allow the text to be kept, read by staff or used to train later models. An approved tool is one your organisation has assessed and agreed terms for. Using AI assistants safely at work turns these points into everyday habits.

Removing identifiers from text is harder than it looks. In Health data: the raw material you saw that free text has no fixed fields, and the same goes for identifying details: a name may appear in the middle of a sentence, a relative may be mentioned, or a rare job or event may point to one person. Deleting the obvious fields at the top of a letter is not enough. The safest rule is data minimisation: give the tool only the information the task needs, and nothing else.

When an organisation approves a tool for patient information, it usually checks questions such as where the data is processed and stored, how long it is kept, who at the provider can see it, and whether it may be used to train future models. Staff do not need to answer these themselves, but they should know which tools have been approved and use those, rather than a personal account on a public service that looks similar.

Key point

An LLM is a fast, fluent drafter, not a source of truth. Keep a human in the loop, check against primary sources, and never let fluency stand in for accuracy.

A second example: an assistant is asked to turn a discharge summary into a letter for the patient's GP. It writes a clear, well-organised letter, and in passing describes a medicine as "continued" when the summary says it was stopped. A primary source, here the discharge summary and the prescription record, is the only way to catch that. The letter's polish is no evidence of its accuracy.

Further reading · Module 3Thirunavukarasu et al. 2023: what large language models can and cannot do in medicine. · All reading for this course

Check your understanding

1. What is an LLM trained to do?

2. Which use carries the highest risk?

3. What is a hallucination?

Up next

Lesson 3

Deploying AI in a hospital

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.