LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 35 lessons

Lesson 7 of 9

Module 4 · Machine learning

Unsupervised learning

Unsupervised learning

Unsupervised learning

Without labels, a model can still find structure: the main directions of variation with PCA, and groups of similar samples with clustering. This is how new disease subtypes are proposed, and also how they are over-claimed.

⏱ 12 min readModule 4 · Machine learningFree
Lesson at a glance6 steps · about 12 min
  1. PCA: how many dimensions do the data really have?
  2. What this does
  3. Reading the output
  4. k-means: how many groups?
  5. What this does
  6. Reading the output

Before you start. This lesson uses the breast cancer table from What machine learning is and X from Your first classifier. If you open a new notebook, or Colab has restarted, run this setup cell first.

Python · setup
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

data = load_breast_cancer(as_frame=True)
df = data.frame
df["diagnosis"] = df["target"].map({0: "malignant", 1: "benign"})
X = df[data.feature_names]
y = (df["diagnosis"] == "malignant").astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

PCA: how many dimensions do the data really have?

In this lesson the diagnosis is not used to train anything. The methods see only the 30 measurements and look for structure on their own; the diagnosis comes back only at the end, to check what they found.

Principal component analysis (PCA) finds new axes through the data. Each principal component (PC) is a weighted mix of the original features. The first is the direction along which the samples vary most; the second is the direction of most remaining variation, at right angles to the first; and so on. If a few components capture most of the variation, the 30 features are really telling you about fewer underlying things. For example, radius, perimeter and area all rise together with nuclear size, so one component can stand in for all three.

Python
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, adjusted_rand_score

Xs = StandardScaler().fit_transform(X)          # scale first: PCA is driven by variance
pca = PCA().fit(Xs)
print(pca.explained_variance_ratio_[:5].round(3))
print("PCs for 90%:", int(np.searchsorted(np.cumsum(pca.explained_variance_ratio_), 0.90) + 1))

What this does

  • The imports bring in PCA, the k-means clustering method, and two scores used later.
  • StandardScaler().fit_transform(X) scales all 569 masses so each feature has mean 0 and standard deviation 1. There is no train–test split here: nothing is being predicted, so there is no test score to protect.
  • PCA().fit(Xs) finds all 30 components of the scaled data.
  • explained_variance_ratio_ gives the share of the total variation captured by each component, from largest to smallest; [:5] shows the first five.
  • The last line finds how many components are needed to reach 90%. np.cumsum makes a running total of the shares. np.searchsorted(..., 0.90) finds the position of the first running total that reaches 0.90. Positions in Python start at 0, so + 1 turns the position into a count, and int prints it as a plain whole number.
[0.443 0.19 0.094 0.066 0.055] PCs for 90%: 7

Reading the output

The first component captures 0.443 of the variation (44.3%), the second 0.19 (NumPy drops the trailing zero of 0.190), the third 0.094, and so on, each smaller than the last. Added up, the first five give about 0.85. The second line says the running total first reaches 90% at the seventh component.

The first component alone captures 44% of the variation in 30 features, and 7 components capture 90%. The features are highly redundant, which matches what the correlated weights suggested in Your first classifier.

Unsupervised learning · LSMA free course Bars for the first five principal components show shares of 0.443, 0.19, 0.094, 0.066 and 0.055 of the total variation. Seven of the 30 components together capture 90%. SHARE OF VARIATION, FIRST FIVE COMPONENTS 0.5 0.25 0 0.443 PC1 0.19 PC2 0.094 PC3 0.066 PC4 0.055 PC5 7 of 30 components capture 90%
Each bar is one principal component, in order. The first carries far more than any other, and the shares shrink fast: a sign that many of the 30 features measure overlapping things. Values are from the output above.
PC1 (41% variance)PC2 (17% variance)NormalSubtype ASubtype B
Samples placed on the first two principal components. Illustrative data.

A plot like this is the most common first look at any omics dataset. Each dot is one sample, placed by its score on PC1 and PC2. Samples that sit close together have similar measurements overall. When the dots are coloured afterwards by a known label, such as diagnosis or the batch they were processed in, you can see at once whether that label lines up with the main sources of variation. A batch effect that separates samples on PC1 is a warning sign worth catching early.

To draw it for these data, ask the fitted PCA for each sample's scores and colour the points by diagnosis afterwards:

Python
import matplotlib.pyplot as plt
import seaborn as sns

scores = pca.transform(Xs)                       # one row per mass, one column per PC
sns.scatterplot(x=scores[:, 0], y=scores[:, 1], hue=df["diagnosis"], s=15)
plt.xlabel("PC1 (44.3%)"); plt.ylabel("PC2 (19.0%)")
plt.gca().set_aspect("equal")                    # one unit on PC1 = one unit on PC2
plt.show()

Two details make this an honest figure, in the spirit of Omics figures and Honest axes and scales. Each axis label says how much variation that component carries, so the reader knows PC1 matters more than twice as much as PC2. And set_aspect("equal") keeps the two axes on the same scale; stretching PC2 to fill a square plot would make the spread along it look as important as the spread along PC1. Remember that the diagnosis was not used to find the components. If the colours separate along PC1, the biggest source of variation in the measurements is linked to malignancy.

PCA is also used to shrink data before a classifier: keep, say, the first 7 components and model those instead of the 30 features. If you do, the PCA must go inside the pipeline, like the scaler in Your first classifier. PCA does not look at the labels, but fitted on all rows it still lets the test samples shape the axes the model is trained on. make_pipeline(StandardScaler(), PCA(n_components=7), LogisticRegression(max_iter=1000)) keeps every step within the training folds.

Go deeper: what the first component is made of

Each component is a recipe: one weight, called a loading, per feature. pd.Series(pca.components_[0], index=X.columns).sort_values() shows the recipe for PC1. On these data all 30 loadings have the same sign. The largest are for mean concave points, mean concavity and worst concave points, each about 0.25 to 0.26, against the 0.18 that equal weights would give. PC1 is roughly a general score for large, irregular nuclei.

One pitfall: the sign of a component is arbitrary. Multiply every loading and every score by −1 and the component describes the data equally well. Another version of the software, or another dataset, may give the flipped sign. So never read “malignant samples have positive PC1” as a finding. Read it as “malignant samples lie at one end of PC1”, and check which features sit at that end.

k-means: how many groups?

k-means places k centres and assigns each sample to the nearest one. You must choose k. The silhouette score (from −1 to 1) measures how well separated the clusters are.

A cluster is a group of samples that are more similar to each other than to the rest. k-means finds clusters by repeating two steps: assign each sample to its nearest centre, then move each centre to the average position of the samples assigned to it. It stops when the assignments no longer change. The figure shows this on a small made-up example. For each sample, the silhouette compares the average distance to the other members of its own cluster with the average distance to the members of the nearest other cluster. Values near 1 mean the sample sits well inside its cluster; near 0, on a border; below 0, probably in the wrong cluster. The silhouette score is the average over all samples.

Unsupervised learning · LSMA free course Four panels on ten points with k of 2. First, two centres are placed. Second, each point joins its nearest centre, and two points join the wrong group. Third, each centre moves to the mean of its points. Fourth, points are reassigned and the two groups are now separated correctly. 1 · PLACE k = 2 centres 2 · ASSIGN to nearest centre 3 · MOVE to the group mean 4 · REASSIGN until stable Steps 2 and 3 repeat until no point changes cluster. Diamonds are centres.
A toy example with ten points and k = 2. After the first assignment, two points from the upper-left group join the wrong centre; once the centres move to the middle of their groups (dashed lines show the move), reassignment fixes them. Schematic, not real data.
Python
for k in [2, 3, 4, 5]:
    km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(Xs)
    print(k, round(silhouette_score(Xs, km.labels_), 3))

km = KMeans(n_clusters=2, n_init=10, random_state=0).fit(Xs)
diagnosis = df["diagnosis"].rename("diagnosis")
print(pd.crosstab(pd.Series(km.labels_, name="cluster"), diagnosis))
print("ARI", round(adjusted_rand_score(diagnosis, km.labels_), 3))

What this does

  • The loop tries k = 2, 3, 4 and 5. For each, KMeans(n_clusters=k, ...) clusters the scaled data. Because the result depends on where the centres start, n_init=10 runs the method ten times from different starting centres and keeps the best run (the one whose samples sit closest to their centres). random_state=0 makes those starts repeatable.
  • km.labels_ holds the cluster number given to each of the 569 masses. silhouette_score(Xs, km.labels_) scores how well separated those clusters are.
  • After the loop, the code reruns k-means with k = 2. It takes the diagnosis column, names it, and builds a cross-tabulation with pd.crosstab: a table that counts how many masses fall in each combination of cluster and diagnosis.
  • The last line computes the adjusted Rand index between the true diagnoses and the clusters.
2 0.343 3 0.314 4 0.159 5 0.163 diagnosis benign malignant cluster 0 339 36 1 18 176 ARI 0.654

Reading the output

  • The first four lines pair each k with its silhouette score. k = 2 scores highest, 0.343; the score falls for 3, then drops sharply for 4 and 5. A silhouette of 0.343 indicates clusters that are separated but not cleanly.
  • The table has one row per cluster and one column per diagnosis. Cluster 0 holds 339 benign and 36 malignant masses; cluster 1 holds 18 benign and 176 malignant. The numbers 0 and 1 are arbitrary names: k-means does not know which cluster is "malignant", and with a different start the two numbers could swap.
  • The final line is the adjusted Rand index, 0.654.

Two clusters separate best. Without ever seeing a diagnosis, k-means puts 176 of the 212 malignant masses in one cluster and 339 of the 357 benign in the other. The adjusted Rand index (0 = chance, 1 = perfect agreement) of 0.65 confirms a strong but imperfect match.

The adjusted Rand index looks at every pair of samples and asks whether the two groupings agree on them: are the two masses together in both, or apart in both? It is then adjusted so that random cluster labels score about 0. It does not care what the clusters are called, which is why it suits a comparison like this one.

The silhouette is not the only way to choose k. Another common one is the elbow plot: draw km.inertia_, the total squared distance from each sample to its centre, against k. Inertia always falls as k grows, because more centres are always closer, so you look for the point where the fall flattens out. The elbow is often hard to see, which is why it is best used alongside the silhouette rather than alone. Neither tells you whether the clusters are real; they only compare one k with another.

k-means also makes assumptions you should know. It works best when clusters are roughly round, similar in size and similar in spread, because it simply assigns each sample to the nearest centre. A long, curved group, or one small group next to a large one, can be cut in the wrong place. Other methods relax these assumptions: hierarchical clustering (AgglomerativeClustering in scikit-learn) builds a tree of merges you can cut at any level, and in single-cell RNA-seq, cells are usually grouped with graph-based methods such as Leiden, which link each cell to its nearest neighbours.

Worked example · how well do the clusters match the diagnosis?

cluster 1 = “malignant-like”: 176 ÷ (176 + 18) = 91% malignant
cluster 0 = “benign-like”: 339 ÷ (339 + 36) = 90% benign
misplaced: (36 + 18) ÷ 569 = 9.5% of masses

Step by step. Cluster 1 holds 176 + 18 = 194 masses, of which 176 are malignant: 176 ÷ 194 = 0.907, about 91%. Cluster 0 holds 339 + 36 = 375 masses, of which 339 are benign: 339 ÷ 375 = 0.904, about 90%. The masses in the "wrong" cluster are the 36 malignant ones in cluster 0 and the 18 benign ones in cluster 1, 54 in all: 54 ÷ 569 = 0.095, or 9.5%.

What the answer means. The strongest structure in the measurements lines up well with the diagnosis, which is reassuring: malignancy changes the nuclei a great deal. But the 36 malignant masses that sit with the benign ones show why clustering cannot replace a classifier or a diagnosis. Those are exactly the cases that look ordinary on these measurements.

Clustering always returns clusters, even in random data. Before calling clusters “subtypes”, check that they are stable when you resample the data, and that they mean something clinically or biologically in an independent cohort.

A stability check reruns the clustering many times on random subsets of the samples and asks whether the same samples keep ending up together. If the groups reshuffle each time, they are not a property of the disease. An independent cohort is a separate group of patients, ideally from another centre, in which the proposed subtypes must appear again and must differ in something that matters, such as survival or response to treatment.

The standard references are Lloyd (1982), “Least squares quantization in PCM”, IEEE Transactions on Information Theory, for the k-means procedure; Rousseeuw (1987), “Silhouettes: a graphical aid to the interpretation and validation of cluster analysis”, Journal of Computational and Applied Mathematics; and Hubert and Arabie (1985), “Comparing partitions”, Journal of Classification, for the adjusted Rand index.

Both scores have limits. The silhouette favours compact, round clusters, so it can rate a real but elongated group poorly. The adjusted Rand index needs a reference grouping, which a truly unsupervised study does not have. Here the diagnosis is a check on the clusters, not something to optimise.

Key point

Scale features before PCA and k-means. Both use distances, so a feature measured in large units would otherwise dominate.

A second example from this dataset: mean area has values in the hundreds, while mean smoothness is around 0.1. Unscaled, a difference of 100 in area would swamp any difference in smoothness, and PCA's first component would be little more than "area". In RNA-seq the same thing happens with a handful of very highly expressed genes, which is why expression values are usually log-transformed and often scaled before PCA.

Further reading · Module 4Chicco 2017: ten practical tips for machine learning in biology. · All reading for this course

Check your understanding

1. Seven components explain 90% of the variation in 30 features. This means:

2. k-means was run on random data and found 4 clusters. What can you conclude?

3. Why scale before PCA?

Up next

Lesson 8

Tuning, interpreting and reporting

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.