LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 73 lessons

Lesson 7 of 12

Module 8 · AI for biology

Protein language models and embeddings

Protein language models and embeddings

Protein language models and embeddings

Protein language models learn the grammar of proteins by predicting hidden residues. Use the small ESM-2 model to fill in a masked residue, embed real proteins from two families, and compare them with cosine similarity.

⏱ 14 min readModule 8 · AI for biologyFree
Lesson at a glance4 steps · about 14 min
  1. From words to proteins
  2. Masked-language training
  3. Embeddings as summaries
  4. Limits and costs

From words to proteins

A language model is a model trained to predict missing or next words in text. To do that well, it has to pick up grammar, meaning and context: in "the cat sat on the ___", "mat" is likely and "theorem" is not. Nobody labels the text; the words themselves are the answers. That is why language models can learn from enormous amounts of raw text.

Learning in this way, where the answers are hidden pieces of the data itself rather than labels added by people, is called self-supervised learning. It matters for biology because labels are scarce and sequences are not. As Where AI fits: traditional methods and AI methods side by side put it, the slow, expensive part of most projects is collecting trustworthy answers; a self-supervised model sidesteps that by learning from the hundreds of millions of sequences that carry no label at all.

Proteins are also strings over an alphabet, of 20 amino acids, and they have their own "grammar". A buried position in a protein's core takes hydrophobic residues; two cysteines far apart in sequence may need to pair up in the folded shape; a catalytic site tolerates almost no change. A protein language model is a language model trained on protein sequences instead of sentences. To predict a missing residue well, it has to learn something about these constraints, and so, indirectly, about structure and evolution.

You have seen these constraints before in another form. In Protein families, domains and motifs, a profile built from an alignment of one family recorded which residues each position allows. A protein language model learns something similar, but for all families at once and without being given any alignments: the patterns of what is allowed where are absorbed from the sequences during training.

The model used here is from the ESM-2 family, released by Meta AI and trained on tens of millions of protein sequences from UniProt's clustered UniRef sets. The family ranges from 8 million to 15 billion parameters. We use the smallest, facebook/esm2_t6_8M_UR50D: 6 layers and 8 million parameters, small enough to run on an ordinary processor in seconds. The code uses the transformers and torch packages, both already in Colab; the model, about 30 MB, downloads on first use.

Masked-language training

ESM-2 was trained by masking: in each training sequence, about 15% of the residues are hidden, and the model must predict what they were from all the others. Each wrong guess slightly adjusts its parameters. After hundreds of thousands of such steps over millions of proteins, it is good at the task. The model is built from transformer layers, the same design used in text chatbots. Their key feature, called attention, lets every residue look at every other residue in the sequence when working out its own representation, so a residue can be influenced by partners that are far away in the chain.

Only a fraction of residues is hidden at a time, because the model needs the rest as context; with most of the sequence masked, there would be little to reason from. Keep in mind, too, that ESM-2 was trained on UniRef, which includes haemoglobin and almost every other well-studied protein. When it fills in a residue of HBB, part of its success may come from having seen this protein, or close relatives, during training. That is the homology leakage of A first sequence classifier, evaluated honestly in a new setting, and it is why claims about a language model's skill should be tested on proteins far from its training data.

We can ask the trained model to do its training task. Below, residue 7 of human haemoglobin beta (HBB) is hidden. That is the glutamate which becomes valine in sickle cell disease; it is often called position 6, because traditional numbering skips the first methionine.

Python
import requests, torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

def fetch(acc):
    text = requests.get(f"https://rest.uniprot.org/uniprotkb/{acc}.fasta").text
    return "".join(text.split("\n")[1:])          # drop the header line

proteins = {"HBB": "P68871", "HBA1": "P69905", "MB": "P02144",      # globins
            "INS": "P01308", "IGF1": "P05019", "IGF2": "P01344"}    # insulin family
seqs = {name: fetch(acc) for name, acc in proteins.items()}
print({name: len(s) for name, s in seqs.items()})

name = "facebook/esm2_t6_8M_UR50D"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForMaskedLM.from_pretrained(name).eval()

hbb = seqs["HBB"]
masked = hbb[:6] + tok.mask_token + hbb[7:]       # hide residue 7
inputs = tok(masked, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
pos = (inputs.input_ids[0] == tok.mask_token_id).nonzero().item()
top = logits[0, pos].softmax(-1).topk(3)
print("true residue 7:", hbb[6])
print([(tok.convert_ids_to_tokens(int(i)), round(p.item(), 2)) for p, i in zip(top.values, top.indices)])

What this does

  • fetch downloads each FASTA file from UniProt and joins the sequence lines, dropping the header. The six proteins are three globins (haemoglobin beta, haemoglobin alpha and myoglobin) and three members of the insulin family (insulin and insulin-like growth factors 1 and 2).
  • AutoTokenizer loads the model's tokeniser, which turns a sequence into the integer codes the model reads, one per residue plus a start and an end code. AutoModelForMaskedLM loads the model with its prediction layer; .eval() switches it from training to use.
  • hbb[:6] + tok.mask_token + hbb[7:] builds HBB with residue 7 replaced by the special mask token.
  • logits are the model's raw scores for every possible token at every position. pos finds where the mask sits; softmax turns its scores into probabilities and topk(3) keeps the three most likely residues.
{'HBB': 147, 'HBA1': 142, 'MB': 154, 'INS': 110, 'IGF1': 195, 'IGF2': 180} true residue 7: E [('E', 0.31), ('A', 0.2), ('D', 0.09)]

Reading the output

  • The first line gives the length of each protein as stored in UniProt, from insulin's 110 residues to IGF1's 195. In Colab you may also see download progress bars for the model; they are not part of the result.
  • The true residue 7 is E, glutamate. The model's top guess is also E, with a probability of 0.31, followed by alanine (0.20) and aspartate (0.09). Aspartate is chemically close to glutamate, so even its third choice makes sense.
  • A probability of 0.31 is not certainty. Many positions on a protein's surface tolerate several residues, and this is the smallest ESM-2 model. Larger models are more confident and more often right.
Go deeper: scoring a mutation with a masked model

The masking trick can also score changes. Hide a position, read the model's probability for the original residue and for a substitute, and take the difference of their logarithms: log p(substitute) − log p(original). A strongly negative value means the model finds the substitute much less plausible in that context. Meier and colleagues showed in 2021 that scores like this, from ESM models, track measured effects of mutations on protein function surprisingly well, with no training on mutation data.

Read such a score as evolutionary plausibility, not as harm. Sickle cell disease is a useful warning. The change from glutamate to valine at this surface position does not stop haemoglobin folding. The damage comes later: in low oxygen, the valine helps haemoglobin molecules stick together into long fibres. A model that judges one sequence on its own has no way to see that.

Embeddings as summaries

To predict masked residues, each layer builds, for every residue, a list of numbers describing that residue in its context. In this model each list has 320 numbers. These per-residue vectors from the last layer are the model's embedding of the protein. They are useful far beyond the training task, because they carry what the model learned about each position.

The last layer is not the only choice. It is shaped most directly by the masking task, while earlier layers hold somewhat different information, and for some downstream tasks a middle layer works as well or better. If embeddings are central to a project, it is worth comparing a few layers on your training data, with the same cross-validation, and reporting which one you used.

A protein of 147 residues gives 147 vectors. To get one vector per protein, the simplest way is average pooling: take the mean of the 147 vectors, number by number. The result is a fixed-length summary of 320 numbers, whatever the protein's length. That makes it a drop-in replacement for the hand-made features of Turning sequences into numbers: features for machine learning: you could feed it to the classifier of A first sequence classifier, evaluated honestly instead of composition.

To compare two embeddings, we use cosine similarity. It treats each embedding as an arrow in 320 dimensions and measures the angle between two arrows: 1 if they point the same way, 0 if they are at right angles. It ignores the arrows' lengths, so it compares the pattern of the numbers rather than their size.

Protein language models and embeddings · LSMA free course A protein sequence is split into one token per residue, with start and end tokens added. Six transformer layers turn each token into a vector of 320 numbers. The start and end vectors are dropped and the residue vectors are averaged into one 320-number embedding for the whole protein. Two embeddings are then compared with cosine similarity. FROM A SEQUENCE TO ONE EMBEDDING startMVHL…end 6 transformer layers (attention) one 320-number vector per residue; start and end dropped mean 1 × 320 Compare two proteins small angle cosine near 1 = similar direction
Each residue gets a vector in context; averaging gives one fixed-length embedding per protein; cosine similarity compares two embeddings by the angle between them. Schematic: the real vectors have 320 numbers, not drawn.

How to read this figure

Start at the top left. The sequence is split into tokens, one per residue, with a start and an end token added. The dark bar is the model. Below it, each residue now has its own vector (orange bars); the dashed grey ones belong to the start and end tokens and are dropped. The arrow labelled "mean" averages the residue vectors into one, the solid orange bar. On the right, two such embeddings are drawn as arrows; the smaller the angle between them, the closer their cosine similarity is to 1.

Python
import torch.nn.functional as F

def embed(seq):
    inputs = tok(seq, return_tensors="pt")
    with torch.no_grad():
        out = model(**inputs, output_hidden_states=True)
    per_residue = out.hidden_states[-1][0, 1:-1]   # drop start and end tokens
    return per_residue.mean(dim=0)                 # average-pool to one vector

E = torch.stack([embed(s) for s in seqs.values()])
print(E.shape)
sim = F.cosine_similarity(E[:, None], E[None, :], dim=-1)
names = list(seqs)
print("      " + " ".join(f"{n:>5}" for n in names))
for n, row in zip(names, sim):
    print(f"{n:>5} " + " ".join(f"{v:5.2f}" for v in row.tolist()))

What this does

  • output_hidden_states=True asks the model to return the vectors from every layer; hidden_states[-1] is the last layer. [0, 1:-1] takes the first (only) sequence and drops the start and end tokens.
  • .mean(dim=0) averages over the residues, leaving 320 numbers. torch.stack puts the six embeddings into one table.
  • F.cosine_similarity(E[:, None], E[None, :], dim=-1) compares every embedding with every other in one step, giving a 6 × 6 table. The loop prints it with the protein names as row and column labels.
torch.Size([6, 320]) HBB HBA1 MB INS IGF1 IGF2 HBB 1.00 0.99 0.98 0.86 0.87 0.91 HBA1 0.99 1.00 0.96 0.86 0.89 0.92 MB 0.98 0.96 1.00 0.85 0.87 0.90 INS 0.86 0.86 0.85 1.00 0.94 0.95 IGF1 0.87 0.89 0.87 0.94 1.00 0.97 IGF2 0.91 0.92 0.90 0.95 0.97 1.00

Reading the output

  • torch.Size([6, 320]): six proteins, 320 numbers each.
  • The diagonal is 1.00: each protein compared with itself. The table is symmetric: HBB against MB is the same as MB against HBB.
  • Every value is high, between 0.85 and 1.00. Average-pooled embeddings of natural proteins all point in broadly similar directions, so read the values relative to each other, not against 0.
  • Within the globins, the similarities are 0.96 to 0.99. Within the insulin family they are 0.94 to 0.97. Between the two families they are 0.85 to 0.92. Every same-family pair scores higher than every cross-family pair, so the embeddings have grouped the proteins by family without being told about families.

This grouping is not just shared letters. Myoglobin and haemoglobin beta share far fewer than half their residues, yet score 0.98, because they share a fold and the patterns that go with it. That is what lets embedding search (Where AI fits: traditional methods and AI methods side by side) find relatives that alignment struggles with. A lesson-3 style classifier trained on these 320 numbers, instead of composition, would often do better, and would need the same grouped evaluation.

Worked example · cosine similarity by hand

Real embeddings have 320 numbers. To see the arithmetic, use three made-up embeddings with 3 numbers each: a = (1, 2, 2), b = (2, 1, 2) and c = (2, −1, 0).

a · b = 1×2 + 2×1 + 2×2 = 8
length of a = √(1 + 4 + 4) = 3 length of b = √(4 + 1 + 4) = 3
cosine(a, b) = 8 ÷ (3 × 3) = 0.89
a · c = 1×2 + 2×(−1) + 2×0 = 0 cosine(a, c) = 0

Step by step. First multiply the two vectors number by number and add the products; this is the dot product, written a · b. For a and b it is 2 + 2 + 4 = 8. Next find each vector's length: square its numbers, add them, and take the square root. Both a and b have length 3. Divide the dot product by the product of the lengths: 8 ÷ 9 = 0.89. For a and c, the products are 2, −2 and 0, which add up to 0, so the cosine is 0 whatever the lengths.

What the answer means. a and b point in nearly the same direction, so their cosine is close to 1; a and c are at right angles, with nothing in common. Doubling every number in b would not change its cosine with a, because only direction counts. With 320 numbers the arithmetic is the same, just longer, and that is what the 0.98 between haemoglobin beta and myoglobin is.

The ESM approach was introduced by Rives et al. 2021, PNAS, which showed that structure and function information emerges in embeddings learned from unlabelled sequence. ESM-2 and ESMFold are described in Lin et al. 2023, Science. The training data come from UniRef, the clustered UniProt sets described by Suzek et al. 2015, Bioinformatics. When you use embeddings, cite the exact checkpoint, such as esm2_t6_8M_UR50D, the layer used and the pooling method. Embeddings from different checkpoints or layers are not interchangeable, and cosine values from one model cannot be compared with those from another.

Limits and costs

  • Bigger is better, and dearer. The 8-million-parameter model is a teaching model. The larger ESM-2 models give clearly better embeddings and predictions, but need far more memory and, in practice, a GPU. The 15-billion-parameter version is beyond a free notebook.
  • Long proteins. Memory grows quickly with sequence length, because every residue attends to every other. Very long proteins are usually cut into overlapping pieces.
  • An average hides detail. Average pooling throws away where things are. For questions about one site, such as a mutation or a binding residue, use the per-residue vectors instead.
  • Similar is not the same. A high similarity suggests relatedness or a shared fold, not identical function. Two enzymes of one family can act on different substrates.
  • The training data shape the model. It learned from natural sequences in public databases, which over-represent some families and organisms. It is less reliable for rare families, designed proteins and sequences unlike anything it saw.
  • No context. The model reads one sequence. It knows nothing of the cell type, partners or modifications, which often decide what a protein does.

The same models power newer tools. ESMFold, for example, predicts a protein's 3D structure from the language model's internal representations, using a single sequence and no search for relatives, which makes it much faster than methods that need one.

A practical habit follows from these costs. Computing embeddings is the slow step, and the result for a given sequence never changes for a given checkpoint and layer. So compute them once, save them to a file with the protein identifiers, and load that file for every model you try afterwards. This saves time, and it guarantees that every comparison uses exactly the same features.

Key point

A protein language model learns from unlabelled sequences by predicting hidden residues. Its embeddings are ready-made features: proteins of the same family end up close together, even when their sequences differ a lot. Compare them relatively, and evaluate any model built on them as carefully as in A first sequence classifier, evaluated honestly.

A second example: you have 300 newly sequenced proteins from an unusual microbe, and BLAST finds no relatives for many of them. Embedding all 300 and looking for their nearest neighbours among embedded, annotated proteins can suggest a family for some. Treat each suggestion as a hypothesis to check, for example with a predicted structure or an experiment, not as an annotation.

Further reading · Module 8The esm2_t6_8M_UR50D model page: the model used here and its larger siblings. · Transformers documentation: ESM: how to load and use ESM models in Python. · All reading for this course

Check your understanding

1. How is a protein language model such as ESM-2 trained?

2. All cosine similarities in the lesson's table were between 0.85 and 1.00. How should you read them?

3. You want to predict the effect of a single amino acid change at one site. Why is the average-pooled embedding a poor choice?

Up next

Lesson 8

AI structure prediction: what it changed, and its limits

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.