Lesson 9 of 12
Module 8 · AI for biology
Thousands of measurements per sample, and often more measurements than samples. See how dimensionality reduction and clustering find structure, and why a batch effect can quietly take over the result.
Wide data, few samples
Omics data are wide: thousands of measurements for each sample, and often far fewer samples than measurements. A single-cell experiment turns this around in one way, since it may hold hundreds of thousands of cells, but each cell still carries about 20,000 gene measurements, most of them zero. Either way, you cannot look at the columns one at a time and see the picture.
Wide data cause a specific problem for the supervised models of Machine learning on an expression table: from patients to marker genes. With 20,000 genes and 50 patients, some combination of genes will separate any two groups perfectly by chance, so a model can fit the training data exactly and still learn nothing. This is sometimes called the curse of dimensionality. The methods in this lesson are one response: describe the main structure in the data first, with a few well-behaved numbers, before asking any predictive question.
Two families of methods answer this. Dimensionality reduction squeezes many measurements into a few new numbers that keep most of the structure. Clustering groups samples or cells that look alike, without being told the groups in advance. Both are unsupervised: no labels go in, so what comes out is a description of the data, not a prediction.
PCA, and the UMAP idea
Principal component analysis (PCA) finds new axes through the data. The first, PC1, is the direction along which samples differ most; PC2 is the direction with the next most variation that is at right angles to PC1, and so on. Each principal component is a weighted mixture of the original genes, and each sample gets a score on it. Plot PC1 against PC2 and you see most of the spread in two dimensions. PCA is linear, which has two consequences: it is fast and predictable, and it cannot untangle structure that is curved rather than straight.
UMAP and t-SNE are non-linear and are what most single-cell figures show. The idea behind both: for each cell, find its nearest neighbours in the full high-dimensional space, then arrange points on a flat page so that neighbours stay close. Clusters separate far more cleanly than in PCA. The price is that the map is a drawing, not a measurement.
- Distances between well-separated clusters on a UMAP mean little. Two blobs on opposite sides of the page are not necessarily more different than two that touch.
- The shape of the map changes with its settings (how many neighbours, how tightly points are packed) and with the random seed. Always record them.
- Both are usually run on the first 20–50 principal components, not the raw genes, for speed and to cut noise.
Use PCA to check the data and to feed later steps; use UMAP to show the result. Never claim a biological difference from a UMAP alone.
"Check the data" means something concrete. Plot PC1 against PC2 and colour the points by every piece of information you have: condition, sex, batch, sequencing depth. If one sample sits far from all the others, it may be a failed library worth checking with the reports you learnt to read in Quality control. If the points separate by sequencing run rather than by condition, you have found a batch effect before it misleads you, the problem the later sections of this lesson turn to. A PCA plot made at the start of an analysis is one of the cheapest checks in omics.
Clustering cells
k-means is the plainest clustering method. You choose k, the number of groups, and the algorithm places k centres, assigns every point to its nearest centre, moves each centre to the average of its members, and repeats until nothing moves. Single-cell work more often uses graph-based clustering (Leiden or Louvain): build a network joining each cell to its nearest neighbours, then cut the network into densely connected communities. That needs no k, but it does need a resolution setting, which plays the same part: turn it up and you get more, smaller clusters.
In both cases the clusters are only shapes in the data. They become cell types when you look at which genes are high in each cluster and recognise known markers, by hand or by automatic reference mapping. The code below does this in miniature.
The choice of k, or of resolution, is not something the data settle for you. The same immune cells can honestly be split into three groups (T cells, B cells, monocytes) or into fifteen (several kinds of T cell, and so on). A useful habit is to try a few values and watch which clusters stay stable and which split or merge. Clusters that appear at every setting are more trustworthy than ones that exist only at a single value.
Go deeper: double dipping: testing clusters on the data that made them
After clustering, the usual next step is to find marker genes by testing each cluster against the rest. The p-values from that test are too small. The clusters were drawn precisely where cells differ most, so of course genes differ between them. This is called double dipping: using the same data both to form a hypothesis and to test it.
You can see the effect with pure noise. Cluster a matrix of random numbers into two groups, then test every gene between the groups. Many genes come out highly "significant", although there is no structure at all. So treat marker-gene p-values as a ranking, not as evidence that clusters are real. Stronger evidence comes from elsewhere: markers that match known biology, clusters that reappear in an independent dataset, or held-out data. Statisticians have developed corrected tests for this situation, but the simple rule is to be sceptical of p-values after clustering.
A small example
This lesson needs numpy, scikit-learn and pandas, all preinstalled in Colab. The matrix is made up, so you can see whether the methods recover groups that are known to be there. Real data would come from a count matrix of the kind you built in Module 5.
import numpy as np
from sklearn.decomposition import PCA
rng = np.random.default_rng(0)
n_per_type, n_genes = 60, 200
types = np.repeat(["T cell", "B cell", "monocyte"], n_per_type)
X = rng.normal(0, 1, size=(len(types), n_genes)) # background noise
X[types == "T cell", 0:10] += 3 # 10 marker genes per type
X[types == "B cell", 10:20] += 3
X[types == "monocyte", 20:30] += 3
pca = PCA(n_components=10, random_state=0)
pcs = pca.fit_transform(X)
print("Matrix:", X.shape)
print("Variance explained, PC1-PC4:", pca.explained_variance_ratio_[:4].round(3))
for t in ["T cell", "B cell", "monocyte"]:
print(f"{t:9} mean PC1 {pcs[types == t, 0].mean():6.2f} PC2 {pcs[types == t, 1].mean():6.2f}")What this does
default_rng(0)makes a random number generator with a fixed seed, so the same numbers come out every time.rng.normal(0, 1, size=(180, 200))fills a matrix with background noise: 180 cells as rows, 200 genes as columns, each value drawn from a bell curve centred on 0 with a spread of 1. Real expression data are counts, not bell-shaped, but after the usual log transformation they are closer to this.X[types == "T cell", 0:10] += 3adds 3 to the first ten genes for T cells only. These are the marker genes: ten per cell type, thirty in all, and the other 170 genes carry nothing but noise.PCA(n_components=10)finds the ten strongest directions.fit_transformreturns each cell's score on each of them.explained_variance_ratio_is the share of the total variation each component accounts for.
Reading the output
PC1 and PC2 each carry about 12.5% of the variation, and PC3 onwards drop to 1.5%. That gap is the signal: two components are enough, because three groups need only two axes to be separated, as three corners of a triangle. The mean scores show the triangle. T cells sit far to the right on PC1 (+7.44), B cells far to the left (−6.20), and monocytes high on PC2 (+7.80). Note that no component is "the T cell component": PC1 separates T cells from B cells, and PC2 lifts monocytes away from both.
The 12.5% may look small. It is not. Thirty informative genes out of 200 cannot account for much of the total variation, most of which is noise, and yet PCA found them. Judge the components by the gap between them, not by the raw percentage.
On real data the gap is rarely this sharp. A common approach is to plot the variance explained by each component in order, a scree plot, and keep the components before the curve flattens. In single-cell work, analysts usually keep somewhat more components than the elbow suggests, because keeping a few extra noisy ones costs little, while dropping one that carries a rare cell type loses it from every later step.
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
km = KMeans(n_clusters=3, n_init=10, random_state=0)
clusters = km.fit_predict(pcs[:, :5])
print(pd.crosstab(types, clusters, rownames=["true type"], colnames=["cluster"]))
print("Adjusted Rand index:", round(adjusted_rand_score(types, clusters), 3))What this does
KMeans(n_clusters=3, n_init=10, random_state=0)looks for three groups.n_init=10restarts the search ten times from different starting centres and keeps the best run, because k-means can settle on a poor answer from an unlucky start.fit_predict(pcs[:, :5])clusters the cells using their scores on the first five components, not the 200 genes. This is the usual order: reduce first, cluster second.pd.crosstabcounts how many cells of each true type landed in each cluster.- The adjusted Rand index compares two groupings of the same points. It is 1 when they agree perfectly, and about 0 for a grouping no better than chance. It ignores the cluster numbers themselves, which are arbitrary.
Reading the output
Every cell is in the right group: 60 B cells in cluster 0, 60 monocytes in cluster 1, 60 T cells in cluster 2, and an adjusted Rand index of 1.0. The cluster numbers have no meaning; cluster 2 is not "better" than cluster 0. This is the best case, with clean groups and a known k. Real data give partial agreement, and deciding whether a cluster is a cell type, a state, or an artefact is the hard part.
Batch effects
A batch effect is a systematic difference between groups of samples that comes from how they were handled, not from biology: a different day, kit, machine, operator or hospital. Omics measurements are sensitive enough that these differences are often larger than the biological ones. The next block adds a batch effect to the same matrix and clusters again.
batch = np.tile(["lab A", "lab B"], len(types) // 2) # alternate cells between two labs
X_b = X.copy()
X_b[batch == "lab B"] += rng.normal(0, 2, size=n_genes) # lab B shifts every gene a little
pcs_b = PCA(n_components=10, random_state=0).fit_transform(X_b)
clusters_b = KMeans(n_clusters=3, n_init=10, random_state=0).fit_predict(pcs_b[:, :5])
print(pd.crosstab([types, batch], clusters_b, rownames=["type", "batch"], colnames=["cluster"]))
print("ARI vs cell type:", round(adjusted_rand_score(types, clusters_b), 3))
print("ARI vs batch: ", round(adjusted_rand_score(batch, clusters_b), 3))What this does
np.tile(["lab A", "lab B"], 90)labels the cells A, B, A, B and so on, so each lab holds half of every cell type. The design is balanced on purpose.X_b[batch == "lab B"] += rng.normal(0, 2, size=200)draws one shift per gene and adds the same shift to every lab B cell. That is what a batch effect looks like: a consistent offset shared by all samples processed together.- PCA and k-means are rerun exactly as before. The last two lines score the clusters against cell type and against batch.
Reading the output
The clustering has changed completely. Cluster 1 is now every lab A cell, of all three types. Lab B splits into two clusters, one of them the lab B T cells. The adjusted Rand index against cell type has fallen from 1.00 to 0.15, while against batch it is 0.78. The method has not failed; it has faithfully found the largest source of variation, which is now the lab.
This is why single-cell and multi-omics pipelines spend so much effort on integration: making datasets from different batches, labs or technologies comparable before any biological reading. Note also that the damage was hidden from view: nothing in the clustering itself announced a batch effect. Only the batch labels revealed it, which is why they must be recorded for every sample.
The balanced design in the code is what makes integration possible here. Because each lab holds all three cell types, a method can learn the lab shift by comparing like with like, and remove it. If lab A had processed only T cells and lab B only B cells, the shift and the biology would be the same variable, and removing one would remove the other. Batch effects are best handled at the design stage; computation can only correct what the design leaves separable.
How to read the figure. Each dot is a cell, coloured by its true type; the panels show where cells sit after PCA. On the left the colours gather separately, so a clustering method finds cell types. On the right each panel-half holds all three colours mixed, so the method finds laboratories. The numbers below each panel are the adjusted Rand indices from the output.
Autoencoders and integration models
An autoencoder is a neural network trained to reproduce its own input through a narrow middle layer. The encoder squeezes a cell's thousands of genes into perhaps ten or thirty numbers; the decoder tries to rebuild the original from those numbers. Because the middle is narrow, the network must keep what matters. It is dimensionality reduction learned from the data, and unlike PCA it can be non-linear.
Single-cell variants of this idea add two things. They model counts properly, using distributions suited to count data with many zeros, and they let you tell the model which batch each cell came from, so it can learn a representation in which batch is factored out. The result is one shared space in which cells from several experiments can be compared. Related models align different kinds of measurement, such as RNA and open chromatin from the same cells, which is the multi-omics integration of Why multi-omics, and how layers are integrated, in Module 7.
Two cautions. These models have many settings, and their output changes with them; and because they can remove batch differences so effectively, they can also remove real biology that happens to line up with batch. If every patient was processed on their own day, no method can tell patient from day.
Integrating different kinds of measurement brings back the ideas of Why multi-omics, and how layers are integrated: one layer, such as RNA, may describe a cell's state while another, such as open chromatin, describes what it is poised to do. A shared representation lets you ask how the two relate, but only for features that the model has been able to link, and the links are themselves learned from data and may be wrong for rare cell types.
The count-based autoencoder approach is best known from scVI, Lopez et al. 2018, Nature Methods. A large independent comparison of integration methods is Luecken et al. 2022, Nature Methods, which scored methods on two goals at once: removing batch differences and keeping biological variation. Cite benchmarks with care. Rankings depended on the task, the dataset and how the two goals were weighted, and no method won everywhere. A method that mixes batches perfectly may have done so by erasing real differences, so check known cell types after integration as well as batch mixing.
Foundation models for cells
A foundation model is trained once, at large scale, on unlabelled data, then adapted to many tasks. The protein language model in Protein language models and embeddings is one. Several groups have now trained similar models on tens of millions of single-cell profiles, treating each cell's ranked genes rather as a language model treats a sentence. The promise is a general-purpose representation of a cell that transfers to new tasks with few labels: annotating cell types, predicting the effect of knocking out a gene, matching data across technologies.
Treat the claims carefully. Independent evaluations have found that on several standard tasks these models do not clearly beat much simpler methods, including PCA with a nearest-neighbour classifier, once the comparison is fair. The reasons are the subject of Judging AI claims in biology: training data that overlap the benchmarks, baselines that are too weak, and evaluation that rewards recognising familiar datasets. The field is moving quickly, so judge each new model on its own evidence, and ask for the simple baseline.
In practice, the fair comparison has three parts. The task should use cells, donors or studies that the model never saw in training. The simple baseline should be given the same chance: the same preprocessing, and a sensible choice of its settings. And the result should be reported for several tasks, not only the one that looks best. A foundation model that wins under these conditions is worth using; one that wins only on its developers' own benchmark is not yet known to be.
Worked example · is this cluster biology or batch?
cluster 1 = 90 cells, all from lab A, of all three types
before the batch effect: ARI vs cell type = 1.00
A collaborator sends a UMAP with three clean clusters and says the middle one is a new cell state.
Step by step. First ask what else differs between the clusters. Score the clustering against every piece of sample information you have: cell type, batch, donor, processing day, chemistry version. Here the agreement with batch is 0.78 and with cell type only 0.15, so the clusters mostly track the laboratory. Then check the composition: a cluster that contains all three cell types but only one lab is a technical group, not a cell state. Finally compare with the run before the batch effect was added, where the same method reached 1.00 against cell type.
What the answer means. The "new state" is the lab A effect. Nothing is wrong with the clustering; the data were asked a question they could not answer. The fix is in the design and the processing, not in the plot: balance cell types across batches, record the batch of every sample, and integrate before clustering. If every type had been processed in its own batch, no amount of analysis could separate them.
Key point
Unsupervised methods find the largest source of variation, whatever it is. If that is the laboratory, that is what you will see. Record every batch, check clusters against technical labels before biological ones, and treat a UMAP as a drawing rather than a measurement.
A second example: tumour and normal samples collected in different years separate beautifully on PC1. Before calling those genes cancer markers, check whether PC1 also separates the collection years, the storage method and the sequencing runs. If it does, the experiment cannot answer the question as it stands.
Further reading · Module 8scikit-learn: decomposition and clustering. · UMAP documentation: what the settings do. · Scanpy tutorials: the same steps on real single-cell data. · All reading for this course
Check your understanding
1. On a UMAP, two clusters sit at opposite edges of the plot and two others almost touch. What may you conclude about how different the cell types are?
2. Clustering splits your cells into three groups that agree strongly with the processing laboratory and barely at all with cell type. What has happened?
3. A new foundation model for cells reports strong results on a public benchmark. What is the first thing to check?
Up next
Lesson 10
Judging AI claims in biology