Lesson 3 of 12
Module 8 · AI for biology
Build a labelled set of mitochondrial and secreted human proteins from UniProt, train two models on composition features, and measure them with cross-validation. Then see how near-identical relatives inflate the score, and how grouped splits guard against it.
The question
Can a simple model tell, from sequence alone, whether a human protein works inside mitochondria or is secreted from the cell? This is a clean contrast with real biology behind it. Most secreted proteins start with a signal peptide: a short N-terminal stretch with a hydrophobic core that sends the new chain into the secretory pathway (you saw insulin's in Turning sequences into numbers: features for machine learning). Many mitochondrial proteins start instead with a presequence, or targeting peptide: an N-terminal stretch rich in positively charged arginine and in serine, and poor in acidic residues. Both tags sit at the start of the chain, so a feature set that looks at the first 30 residues should help.
The biology also sets a ceiling on any score. Not every mitochondrial protein has a presequence: many proteins of the outer membrane and of the space between the two membranes are sent there by signals inside the chain instead. A model that only looks at the start of the protein will miss those, however well it is trained. Knowing this before you start tells you which errors to expect, and stops you mistaking a biological limit for a bug.
In this lesson you build a small labelled dataset from UniProt, train two models, measure them with cross-validation, and then check whether the score is honest. The code needs Biopython (!pip install biopython); scikit-learn, NumPy and SciPy are already in Colab.
Building the dataset
UniProt's search can be run from code through its REST interface: a web address that returns results instead of a web page. The query below asks for reviewed (Swiss-Prot) human entries annotated as located in one compartment and not the other, between 80 and 1,000 residues long. Results are sorted by accession and only the first 200 of each are kept, so the query is fixed and anyone running it gets the same list, as long as the UniProt release is the same.
The length limits are a judgement call. Very short entries are often peptide hormones or fragments, and very long ones are usually large proteins with many domains, where an N-terminal tag is a tiny part of the whole. Removing both keeps the dataset closer to the question being asked. Whatever filters you use, save the list of accessions you ended up with, alongside the UniProt release, so the exact dataset can be rebuilt later even if the query returns something different.
import requests
from io import StringIO
from Bio import SeqIO
URL = "https://rest.uniprot.org/uniprotkb/search"
def fetch(place, not_place):
q = (f"reviewed:true AND organism_id:9606 AND cc_scl_term:{place} "
f"AND NOT cc_scl_term:{not_place} AND length:[80 TO 1000]")
r = requests.get(URL, params={"query": q, "format": "fasta",
"size": 200, "sort": "accession asc"})
print(r.url)
return [str(rec.seq) for rec in SeqIO.parse(StringIO(r.text), "fasta")]
mito = fetch("SL-0173", "SL-0243") # mitochondrion, not secreted
secr = fetch("SL-0243", "SL-0173") # secreted, not mitochondrion
seqs = mito + secr
labels = [1] * len(mito) + [0] * len(secr) # 1 = mitochondrial
print(len(mito), "mitochondrial,", len(secr), "secreted")What this does
reviewed:truekeeps Swiss-Prot entries, whose annotation a curator has checked.organism_id:9606is the NCBI taxonomy number for human.cc_scl_termsearches the subcellular location annotation.SL-0173is UniProt's term for Mitochondrion andSL-0243its term for Secreted.AND NOTremoves proteins annotated with both, so no protein carries two labels.length:[80 TO 1000]drops very short and very long entries.size: 200takes the first 200 results, andsort: accession ascfixes their order.requests.get(..., params=...)builds the full address from the pieces and fetches it;print(r.url)records exactly what was asked. The labels are 1 for mitochondrial and 0 for secreted.
Reading the output
The two printed addresses are the exact queries; paste one into a browser to see the same FASTA file. Each returned 200 sequences, giving a balanced set of 400. When this lesson was written, UniProt release 2026_04 held 1,237 entries matching the first query and 1,831 matching the second, so these are samples of the first 200 by accession, not random samples. A later release may give a slightly different list and slightly different numbers below.
Features, models and cross-validation
Each protein becomes 40 numbers: the composition of the whole sequence (20) and the composition of its first 30 residues (20), as in Turning sequences into numbers: features for machine learning. Two models are compared. Logistic regression learns one weight per feature and turns their weighted sum into a probability. A random forest builds hundreds of small decision trees, each asking questions like "is the share of arginine in the first 30 residues above 0.1?", and lets them vote.
With only 400 proteins, a single train/test split would leave a small, noisy test set. Cross-validation uses every protein for testing once. The data are cut into 5 equal parts, called folds; the model is trained on 4 and tested on the fifth, and this is repeated so each fold is the test set once. The 5 scores are averaged. Stratified means each fold keeps the 50:50 mix of the two classes.
Two models are compared on purpose. If a flexible model only just beats a simple one, the extra flexibility is not buying much, and the simple model is easier to explain. Accuracy is also only one number: it does not say whether the errors fall mostly on one class. Measuring a model honestly: confusion matrix, ROC, precision, recall and imbalance takes these predictions apart into the four kinds of right and wrong answer. For now, accuracy is enough to compare the two models on the same folds.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import StratifiedKFold, cross_val_score
AA = "ACDEFGHIKLMNPQRSTVWY"
def features(s):
whole = [s.count(a) / len(s) for a in AA]
start = [s[:30].count(a) / 30 for a in AA] # the N-terminal 30
return whole + start
X = np.array([features(s) for s in seqs])
y = np.array(labels)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
models = {"logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)),
"random forest": RandomForestClassifier(n_estimators=300, random_state=0)}
for name, model in models.items():
print(name, round(cross_val_score(model, X, y, cv=cv).mean(), 3))What this does
featuresreturns 40 fractions: the 20 for the whole protein followed by the 20 fors[:30].Xstacks them into a table with one row per protein.StratifiedKFold(n_splits=5, shuffle=True, random_state=0)sets up the five folds. Shuffling mixes the two classes, which arrive in two blocks; the fixedrandom_statemakes the folds the same on every run.- The logistic regression sits in a pipeline after
StandardScaler, which puts all features on the same scale. The random forest grows 300 trees and does not need scaling. cross_val_scoretrains and tests each model five times and returns five accuracies;.mean()averages them.
Reading the output
Guessing would score 0.5, because the classes are balanced. Logistic regression gets 83.5% of proteins right and the random forest 87.8%. That is a good result for 40 composition numbers and supports the biology: the N-terminal make-up carries much of the signal. The forest does better because it can combine features in ways a single weighted sum cannot, for example "high arginine at the start and few acidic residues". But is 87.8% honest?
Go deeper: why the scaler sits inside the pipeline
The code puts StandardScaler inside a pipeline with the model. That is not just tidiness. Scaling uses the mean and spread of each feature. If you scale the whole dataset first and then cross-validate, those numbers already include the test proteins, so a little information about the test set has reached training.
For scaling, the effect is usually small. For other steps it can be huge. The classic case is feature selection: picking the 50 features that best separate the classes using all the data, then cross-validating a model on those 50. With thousands of features and few samples, this can produce impressive accuracy on pure noise, because the features were chosen partly for how well they fit the test samples. The rule is simple: any step that learns from the data, including scaling, selection and tuning, must be repeated inside each training fold.
Homology leakage
Proteins come in families. Two members of a family, or two copies of a recently duplicated gene, can be almost identical. Now imagine one copy lands in the training folds and its near twin in the test fold. The model does not need to have learned anything general about mitochondria or secretion; it only needs to recognise a protein it has nearly seen before. The test score then measures memory, not understanding, and is too high for truly new proteins. This is homology leakage: information leaking from training to test through shared ancestry.
The guard is to split by family, not by protein. First group the proteins so that any two similar proteins fall in the same group; then make sure each whole group goes to one side of every split. Dedicated tools such as CD-HIT and MMseqs2 cluster sequences by percent identity for exactly this purpose. Here we use a quick stand-in. For each protein, list the set of 3-mers it contains. The Jaccard similarity of two proteins is the number of 3-mers they share divided by the number found in either: 0 means no shared words, 1 means the same set. For example, the made-up peptides MKVLAA and MKVLGA have 3-mers {MKV, KVL, VLA, LAA} and {MKV, KVL, VLG, LGA}; they share 2 of 6 distinct words, so their Jaccard similarity is 2 ÷ 6 = 0.33.
In Module 4, Comparing sequences, you measured similarity by aligning sequences and counting percent identity. The 3-mer Jaccard score is a rough, alignment-free substitute: it is quick to compute for every pair, but it only sees exact shared words. Two relatives whose letters have drifted, while keeping the same fold, may share few 3-mers and so land in different groups. That makes this stand-in a little too lenient; a real study would cluster with CD-HIT or MMseqs2 at a stated identity.
How to read this figure
Each circle is a protein and each colour is a family of similar sequences. White boxes are the training folds and orange boxes the test fold. On the left, the split ignores families: the dashed line marks a test protein whose close relative was in training. On the right, dashed outlines show the similarity groups, and each group is placed wholly in training or wholly in test. Only the right-hand split tests the model on families it has never met.
from scipy.sparse.csgraph import connected_components
from sklearn.model_selection import StratifiedGroupKFold, cross_val_predict
kmers = [{s[i:i + 3] for i in range(len(s) - 2)} for s in seqs]
J = np.array([[len(a & b) / len(a | b) for b in kmers] for a in kmers])
np.fill_diagonal(J, 0) # ignore each protein's match to itself
rf = models["random forest"]
pred = cross_val_predict(rf, X, y, cv=cv)
twin = np.zeros(len(y), bool) # has a near twin in its training folds?
for train, test in cv.split(X, y):
twin[test] = J[np.ix_(test, train)].max(axis=1) >= 0.5
print(twin.sum(), "with a near twin: accuracy", round((pred == y)[twin].mean(), 3))
print((~twin).sum(), "without: accuracy", round((pred == y)[~twin].mean(), 3))
n_groups, groups = connected_components(J >= 0.3)
gcv = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=0)
print(n_groups, "similarity groups")
print("grouped CV, random forest", round(cross_val_score(rf, X, y, cv=gcv, groups=groups).mean(), 3))What this does
kmersholds, for each protein, the set of its 3-mers.Jis a 400 × 400 table of Jaccard similarities:a & bis the set of shared words anda | bthe set of all words.fill_diagonal(J, 0)blanks each protein's similarity to itself.cross_val_predictruns the same five folds as before but returns each protein's prediction from the fold where it was tested.- For each test fold,
J[np.ix_(test, train)].max(axis=1)finds each test protein's most similar training protein. A protein counts as having a near twin if that similarity is 0.5 or more. The code then prints the accuracy separately for proteins with and without a near twin. connected_components(J >= 0.3)joins any two proteins with a similarity of 0.3 or more, and chains them, so relatives of relatives end up in one group.StratifiedGroupKFoldthen makes five folds that keep each group whole while keeping the classes balanced.
Reading the output
- 52 of the 400 proteins had a near twin in their training folds, and the forest got 98.1% of them right. For the other 348 it got 86.2%. Recognition, not understanding, explains much of that gap. 46 of the 52 are secreted immunoglobulin variable segments, the building blocks of antibodies, which come from large families of very similar genes.
- Chaining at a similarity of 0.3 gives 336 groups from 400 proteins: most proteins are alone, and a few families hold several members.
- Grouped cross-validation gives 0.87, against 0.878 for the random folds. Here the drop is small, because only 13% of the proteins have near twins and 40 composition numbers give a model little to memorise. With more family-rich data, or with models that can memorise more, such as nearest-neighbour search or large networks, the gap is often much wider.
How to read this figure
Each bar is an accuracy, from 0 on the left to 1 on the right; the dashed line is the 0.5 a coin toss would score. The top two bars split the same random-fold predictions into proteins that had a near twin in training and proteins that did not. The bottom two bars are the overall scores from random and from grouped folds. The orange bar is the inflated one.
The clustering tools named here are CD-HIT (Li & Godzik 2006, Bioinformatics) and MMseqs2 (Steinegger & Söding 2017, Nature Biotechnology). The wider case for reporting how data were split is made in the DOME recommendations, Walsh et al. 2021, Nature Methods. When you report a grouped evaluation, state the tool, the identity threshold and the coverage setting. Thresholds such as 30% or 40% are conventions, not laws. Even at 30% identity, remote relatives can still share structure, so a grouped score is fairer, but not a guarantee that the model works on truly new families.
Other ways a score can mislead
- Label noise. Some proteins work in several places, and annotations depend on the evidence available. Some entries in the mitochondrial set, such as polyubiquitin-B and polyubiquitin-C, are found in many places in the cell and carry the mitochondrion label among others.
- Artificial balance. We chose 200 of each. In a real proteome the two classes are not equal, and accuracy on a balanced set does not tell you how many false alarms you would get on all human proteins.
- A non-random sample. Taking the first 200 by accession is fixed and repeatable, but it is not random; older and newer entries may differ.
- Tuning on the test data. If you try many feature sets and keep the best cross-validation score, that score is optimistic. Keep a final, grouped test set aside for the very end.
- Features that contain the answer. If a feature is derived from the annotation itself, such as a UniProt keyword or the presence of an annotated transit peptide, the model can score well by reading the label back. Build features from the sequence alone when the claim is "predicted from sequence".
Worked example · how the twins inflate the score
The random-fold accuracy is a mixture of the two groups printed in block 3. How much does the share of near twins matter?
correct without twin = 0.862 × 348 ≈ 300
overall = (51 + 300) ÷ 400 = 351 ÷ 400 = 0.878
if half had twins: 0.5 × 0.981 + 0.5 × 0.862 ≈ 0.92
Step by step. Multiply each group's accuracy by its size to get the number of correct predictions: about 51 of the 52 proteins with a twin, and about 300 of the 348 without. Add them to get 351 correct out of 400, which is 0.878, the random forest's score from block 2. Now imagine a dataset where half the test proteins had a near twin in training: the average becomes 0.5 × 0.981 + 0.5 × 0.862, which is 0.49 + 0.43, about 0.92.
What the answer means. The same model, with the same real skill on new families (about 0.86), would report 0.878 on this dataset and about 0.92 on a more family-rich one. The reported score depends on how many twins the split happens to create, not just on the model. Splitting by similarity group removes that dependence, which is why a grouped score is the one to report.
Key point
For biological sequences, split by family, not by sequence. Cluster by similarity first, keep each cluster on one side of every split, and report the grouped score.
A second example: a published tool claims 95% accuracy at predicting enzyme class, tested on a random 20% of UniProt enzymes. Many test enzymes will have close relatives in training, often from other species, so 95% mostly reflects recognition. The fair question is how it does on enzymes with no relative above, say, 30% identity in its training set. Good papers report exactly that.
Further reading · Module 8UniProt help: programmatic access and queries: how to build search URLs like the ones above. · UniProt help: subcellular location: where the location labels come from. · scikit-learn user guide: cross-validation, including grouped splits. · All reading for this course
Check your understanding
1. Why does a random split of protein sequences often overstate a model's accuracy on truly new proteins?
2. What does a grouped split, such as StratifiedGroupKFold with similarity groups, guarantee?
3. In the lesson, grouped cross-validation scored 0.87 against 0.878 for random folds. Why was the drop so small?
Up next
Lesson 4
Machine learning on an expression table: from patients to marker genes