Guided project
Module 8 · AI for biology · Self-checked
Build a dataset of kinases, proteases and ion channels from UniProt, classify them with k-mer features and with ESM-2 embeddings, and find out how much of the accuracy survives a family-aware split.
Research question
Can a protein's function class be predicted from its sequence alone, and how much of that ability is real prediction rather than recognising a familiar family?
How to approach this
You will build a small dataset of reviewed human proteins in three function classes, describe them with two kinds of features, and compare the results under two ways of splitting the data. The second half of the question is the important one: a classifier that only recognises families will look excellent under a random split and collapse under a family-aware one, exactly as in Judging AI claims in biology.
The question is a practical one. Every newly assembled genome, of the kind you built in Module 6, yields thousands of predicted genes, and most of their proteins will never be studied in a laboratory. Their function labels come from computational prediction, usually by similarity to proteins that have been studied, which is the annotation you met in Finding genes and annotating a genome. A model that predicts function from sequence is useful only for the proteins similarity search cannot already handle, so the family-aware score, not the random-split score, is the one that tells you whether the model adds anything.
Work in one notebook, top to bottom, so anyone can rerun it. A few habits make that possible.
- Record every URL you download from, in a text cell next to the code. UniProt changes between releases, so also record the date. Your counts may differ slightly from the checkpoints for this reason; the pattern should not.
- Fix every seed the tasks give you (
random_state=0throughout). Your numbers should then be close to the checkpoints. - Write each number down as you get it, in a text cell under the code. Task 6 is quick when every figure is already recorded.
- Keep it small. About 120 proteins per class and the small ESM-2 model are enough; the whole notebook runs in a few minutes on the free Colab CPU. There is no need for a GPU.
Three practical points before you start. UniProt keywords are controlled terms attached to reviewed entries, so keyword:KW-0418 asks for proteins annotated as kinases; the alternative is an EC number, such as ec:2.7.* for transferases that move a phosphate. Ask the REST API for TSV with the fields you need; the search URLs in A first sequence classifier, evaluated honestly show the pattern. And the family labels you will group by come from UniProt's own Protein families field: a real study would instead cluster the sequences by identity, which is stricter, but the annotation is good enough to show the effect.
Decide in advance what a good outcome looks like. It is not the highest accuracy. A good outcome is a pair of numbers you can explain: how well each feature set does when it may recognise families, how well it does when it may not, and why the two differ. A project that ends with "0.94 under a random split, 0.76 when whole superfamilies are held out, and here is what the errors have in common" has answered the question; one that ends with 0.94 alone has not.
Build the dataset and explore lengths
Finding genes and annotating a genome; A first sequence classifier, evaluated honestly
- Download reviewed human proteins for three function classes from the UniProt REST API, about 120 each, and record the URLs.
- Report the length distribution per class and how many protein families each class spans.
Hint. Use keyword queries: keyword:KW-0418 (kinase), keyword:KW-0645 (protease) and keyword:KW-0407 (ion channel), each with reviewed:true AND organism_id:9606 AND fragment:false and a length range such as length:[100 TO 1000]. Ask for format=tsv with the fields accession,gene_primary,protein_families,keywordid,length,sequence, and read the result straight into pandas. Drop any protein carrying more than one of the three keywords, then sample the same number from each class with a fixed seed so the classes are balanced. Keep the Protein families column: task 4 needs it.
Why this step. Balanced classes make the accuracy easy to judge against chance, 0.333 for three classes. A protein carrying two of the keywords has no single right answer, so keeping it would reward the model for whichever label happened to be chosen. Exploring lengths first rules out a trivial shortcut: if one class were much longer than the others, a classifier could score well from length alone and appear to understand function.
Checkpoint: 360 proteins, 120 per class. Mean lengths are similar across classes (about 530–575 amino acids), so length alone carries little information. The 360 proteins span 164 families, but unevenly: 32 families for kinases against 70 for ion channels, and the protein kinase superfamily alone accounts for 95 proteins.
Composition and k-mer features
Turning sequences into numbers: features for machine learning; A first sequence classifier, evaluated honestly
- Build amino-acid composition (20 numbers) and composition plus dipeptide frequencies (420 numbers) for every protein.
- Score both with scaled logistic regression and 5-fold stratified cross-validation, next to a length-only baseline.
Hint. Composition is the share of each of the 20 amino acids; dipeptides are all 400 ordered pairs, counted over the sequence and divided by the number of pairs. Use itertools.product to make the list of pairs once, outside the loop. Put StandardScaler and LogisticRegression in a pipeline, as in A first sequence classifier, evaluated honestly, and use a small C such as 0.1 to keep 420 features under control with 360 proteins. The baseline matters: with three balanced classes, chance is 0.333.
Why this step. Composition and dipeptides are the hand-built features of Turning sequences into numbers: features for machine learning: easy to compute and easy to interpret, so they set the bar a learned representation must clear. Scaling matters because logistic regression with a penalty shrinks all coefficients by the same rule, which is only fair if every feature is on a similar scale. Keep the scaler inside the pipeline, so that it is fitted on the training folds only; scaling on all 360 proteins first is the small leak you met in A first sequence classifier, evaluated honestly.
Checkpoint: length alone about 0.336, which is chance; composition alone about 0.74; composition plus dipeptides about 0.79. The k-mer features add something, but not much.
ESM-2 embeddings
Protein language models and embeddings
- Embed every sequence with facebook/esm2_t6_8M_UR50D and average over residues to get one vector per protein.
- Score them with the same model and the same folds.
Hint. Reuse the embedding code from Protein language models and embeddings. Load the tokeniser and model once, outside the loop, call .eval(), and wrap the loop in torch.no_grad() so no gradients are stored. Drop the first and last token (the start and end markers) before averaging. The model returns 320 numbers per residue, so each protein becomes 320 numbers. Everything downstream is identical to task 2; only the feature matrix changes, which is the point of the comparison.
Why this step. Averaging over residues turns proteins of different lengths into vectors of the same size, which the classifier needs. The average loses the order of the residues, so the embedding must already carry, at each position, information about its surroundings; that is what the language model was trained to provide. ESM-2 accepts at most about 1,000 residues at once, so the length limit in task 1 also keeps every sequence inside the model's window.
Checkpoint: a 360 by 320 matrix, built in under a minute on CPU; cross-validated accuracy about 0.94, clearly ahead of the k-mer features.
Compare, with a leakage check
Judging AI claims in biology
- Measure how often a random split puts a family member in both parts.
- Rescore both feature sets with
GroupKFoldgrouped by family, and again by the top-level group only.
Hint. The family column has the form "superfamily, family, subfamily"; keep the first two parts as the family group and the first part alone as the stricter top-level group. For the overlap figure, loop over the random split's folds and ask what share of test proteins have their family represented in the training part (np.isin does this in one line). GroupKFold needs groups= passed to cross_val_score, and it does not shuffle, so it needs no seed. Expect the group folds to be less balanced than the stratified ones; that is unavoidable when whole families must stay together.
How to read the result. Compare the drops, not only the scores. Holding out families costs the embeddings very little and the k-mer features a little more, so neither was relying only on recognising close relatives. Holding out whole superfamilies is the hard test: the protein kinase superfamily, for example, then sits entirely in one fold, and the model must recognise a kinase without having seen anything like it. Both feature sets lose a great deal there, and that larger drop is the honest measure of how far either generalises to distant proteins.
Checkpoint: in the random split, about 68% of test proteins have a family member in training. Accuracy for composition plus dipeptides: 0.79 random, 0.74 by family, 0.56 by top-level group. For ESM-2: 0.94, 0.93, 0.76. Both drop, the k-mer features further.
Look at the errors
A first sequence classifier, evaluated honestly; Judging AI claims in biology
- Get out-of-fold predictions for the ESM-2 features under the family split and build a confusion matrix.
- List the misclassified proteins with their gene names and families, and look for a pattern.
Hint. cross_val_predict with the same folds gives one prediction per protein, each made by a model that never saw that protein's family. pd.crosstab turns true and predicted labels into a matrix. Then look the genes up: are the wrong ones unusual members of their class, or ordinary ones? Pay attention to where the protein works as well as what it does, since a language model sees only the sequence and membrane proteins look alike.
Read the matrix as you did in Measuring a model honestly: confusion matrix, ROC, precision, recall and imbalance: each row is a true class, each column a predicted one, and the numbers off the diagonal are the errors. Look first for an asymmetry, such as many proteases called ion channels but few ion channels called proteases, because a one-way confusion usually points to a property the two groups share.
Checkpoint: about 24 errors out of 360. Several misclassified "kinases" are not protein kinases at all (nucleoside diphosphate kinases, inositol and lipid kinases), and a group of intramembrane proteases is called ion channels, which suggests the model partly keys on being a membrane protein.
Write a short honest report
Judging AI claims in biology
- In text cells: question, answer, methods, results, errors and limits, in that order.
- Use the numbers you recorded, and say plainly which split each one came from.
Hint. Answer the research question in the first two sentences, with the random-split and family-split numbers side by side and chance (0.333) next to them. Give the methods in one sentence each: the queries and date, the two feature sets, the model, the two splitting schemes. For the limits, work through the checklist in Judging AI claims in biology against your own study: one species, three hand-picked classes, families taken from annotation rather than sequence identity, no external test set, and a database that changes between releases. End with what you would do next, and be specific.
Checkpoint: your report states the accuracy under each split, says what the gap means, and does not claim the model would work on a protein family it has never seen.
Go deeper: clustering by identity for the stricter check
The "Take it further" step asks you to cluster by sequence identity. With MMseqs2, one command does it: mmseqs easy-cluster proteins.fasta clu tmp --min-seq-id 0.3 -c 0.8. This groups proteins at 30% identity or more, with the alignment covering at least 80% of the sequences, and writes a file, clu_cluster.tsv, that lists each cluster's representative next to each member. Use the representative as the group label for GroupKFold.
One subtlety matters for leakage. By default, MMseqs2 assigns each protein to one representative it resembles, but two proteins in different clusters can still be more than 30% identical to each other. The method from A first sequence classifier, evaluated honestly, joining any similar pair and chaining relatives of relatives, is stricter. MMseqs2 can do this too, with --cluster-mode 1, which takes connected groups. Expect fewer, larger groups, and a lower but more honest score.
Take it further
Share the notebook on GitHub with a short README giving the UniProt queries and the date you ran them. Then try a stricter leakage check: cluster the sequences by identity instead of trusting the family annotation, and split on those clusters.
Other extensions, in rough order of effort: add a fourth class and see which pairs confuse the model; swap the small ESM-2 for a larger one and ask whether the family-split accuracy improves as much as the random-split one; or keep the per-residue embeddings instead of averaging, and see whether a protein's domains can be located rather than only its class.
This project closes the course, and it uses nearly every part of it. The sequences come from the databases and file formats of the early modules, fetched and handled with the command line and Python. Comparing sequences gave you the idea of a family, which is what the leakage check depends on. Module 5 turned raw reads into a table you could trust, and Module 6 turned reads into a genome, the source of every predicted protein a model like this would one day label. Module 7 asked what genes do. Module 8 added the models, and with them the habit that matters most: asking how a number was measured before believing it. The same habit applies to every result in omics, with or without machine learning. Check the data, choose the honest comparison, and say plainly what your evidence can and cannot show.
Further reading · Module 8UniProt REST API queries and UniProt keywords. · ESM-2 (8M) model card. · Cross-validation with groups. · All reading for this course
Up next
Optional
Further reading