LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 35 lessons

Lesson 2 of 9

Module 4 · Machine learning

Your first classifier

Your first classifier

Your first classifier

Split the data, lock the test set away, then train a logistic regression model inside a pipeline. This pattern is the backbone of every model in the course.

⏱ 11 min readModule 4 · Machine learningFree
Lesson at a glance6 steps · about 11 min
  1. Split, then train
  2. What this does
  3. Reading the output
  4. Probabilities and weights
  5. What this does
  6. Reading the output

Split, then train

The test set is set aside first. stratify=y keeps the same share of malignant cases in both parts. We code malignant as 1, the “positive” class, because finding cancer is the goal.

Think of the test set as a sealed envelope. You open it once, at the end, to see how the finished model does on masses it has never seen. Every decision before that, including how to scale the data and which settings to use, must be made without looking inside.

The positive class is the one you are trying to detect. It does not mean "good news"; in medical testing a positive result usually means the disease is present. Coding malignant as 1 matters because several measures in the next lesson, such as sensitivity, are calculated for the class labelled 1.

Stratifying means splitting each class separately, so both parts keep the same mix. Without it, a random split of a small dataset could by chance put too few cancers in the test set to judge the model fairly.

Why 20%? It is a trade-off. A bigger test set gives a steadier estimate of performance; a bigger training set gives the model more to learn from. With a few hundred rows, a split of 70/30 to 80/20 is common. With very small datasets no single split is reliable, which is why Cross-validation and data leakage replaces it with cross-validation.

One pitfall does not show up in this dataset but is common in real ones: several rows from the same person. If a patient gave three biopsies and two land in training and one in testing, the model can recognise the patient rather than the disease. Then the split must keep all of one person's rows on the same side. scikit-learn's GroupShuffleSplit does this when you give it a column of patient identifiers.

Your first classifier · LSMA free course All 569 masses are split into a training set of 455, with 170 malignant and 285 benign, and a locked test set of 114, with 42 malignant and 72 benign. A pipeline of StandardScaler then LogisticRegression is fitted on the training set and scored once on the test set, giving accuracy 0.965. 1 · SPLIT FIRST, STRATIFIED All 569 masses · 212 malignant (37%) Training set · 455 masses (80%) 170 malignant, 285 benign Test set · 114 42 malignant, 72 benign 2 · FIT ON TRAINING DATA, SCORE ONCE ON TEST StandardScaler LogisticRegression Score on the test set rescales each feature one weight per feature accuracy 0.965
The split comes before anything is learned. Both parts keep about 37% malignant because the split is stratified. The padlock marks the test set: it is used once, at the end. Schematic, not to scale.
Python
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X = df[data.feature_names]                          # the 30 measurements
y = (df["diagnosis"] == "malignant").astype(int)   # 1 = malignant

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
print(round(model.score(X_test, y_test), 3))       # accuracy on the test set

What this does

  • The four import lines bring in the tools: the splitting function, the pipeline builder, the scaler and the model.
  • X = df[data.feature_names] keeps only the 30 measurement columns, using the list of names that came with the dataset. The target and diagnosis columns are left out, so the answer cannot sneak into the features.
  • (df["diagnosis"] == "malignant") gives True for each malignant row and False for each benign one; .astype(int) turns those into 1 and 0.
  • train_test_split shuffles the rows and splits them. test_size=0.2 sends 20% to the test set. stratify=y keeps the share of 1s the same in both parts. random_state=42 fixes the shuffle, so you get the same split every time you run the code; the number itself has no special meaning. The function returns four pieces, in the order the code names them.
  • make_pipeline(...) chains two steps into one object, model. max_iter=1000 lets the fitting routine take up to 1,000 steps to settle on its weights; the default of 100 is sometimes too few, and scikit-learn then warns that the model has not converged.
  • model.fit(X_train, y_train) trains both steps on the training set only. model.score(X_test, y_test) predicts each test mass and returns the share predicted correctly, its accuracy.
(455, 30) (114, 30) 0.965

Reading the output

The first line shows the two feature tables: 455 rows for training and 114 for testing, each with 30 columns. They add up to 569. Twenty per cent of 569 is 113.8, and scikit-learn rounds the test set up to 114. The second line is the test accuracy: 0.965 means 110 of the 114 test masses were classified correctly, and 4 were not.

Hold that number lightly. With 114 test masses, each single mistake moves accuracy by 1 ÷ 114, a little under one percentage point. A different random_state would give a different split, and the score could easily come out a point or two higher or lower. That is not a fault in the code; it is how much one small test set can tell you. Cross-validation and data leakage shows how to measure this spread instead of hoping it is small.

The pipeline scales the features (they have very different ranges) and then fits logistic regression, which learns one weight per feature and turns their weighted sum into a probability. Because scaling is inside the pipeline, it is learned from the training data only. 96.5% of the 114 test masses were classified correctly; Evaluating a classifier shows why that number alone is not enough.

Scaling here means standardising: for each feature, StandardScaler subtracts its mean and divides by its standard deviation, so every feature ends up centred on 0 with a spread of 1. Without it, a feature such as mean area, whose values run into the hundreds, would sit on a completely different scale from mean smoothness, whose values are around 0.1. The mean and standard deviation are worked out from the training rows and then reused, unchanged, on the test rows.

Despite its name, logistic regression is a classification method. "Regression" refers to the weighted sum it fits; the logistic function then turns that sum into a probability, as the worked example below shows.

Why start with logistic regression rather than something more powerful? It is fast, it rarely fails badly, and its weights can be read. That makes it a good first model: a fair yardstick for anything more complex. Compare it with the trivial rule from What machine learning is, which scored 62.7% by calling everything benign. 96.5% is a large improvement, so the 30 measurements clearly carry information. Whether a fancier model can add more is a question for Trees and ensembles.

Go deeper: saving a trained model for later

Training takes seconds here, but on larger data you will want to keep a fitted model. The usual tool is joblib, installed with scikit-learn: import joblib, then joblib.dump(model, "breast_lr.joblib") saves it, and model = joblib.load("breast_lr.joblib") brings it back, ready to predict.

Save the whole pipeline, not just the logistic regression, so the scaler’s means and standard deviations travel with it. Two pitfalls. A saved model is not guaranteed to load correctly in a different scikit-learn version, so record the version beside the file. And loading such a file can run code stored inside it, so only load files you made yourself or got from a source you trust.

Probabilities and weights

Python
probs = model.predict_proba(X_test.iloc[:3])[:, 1]   # P(malignant), first 3 masses
print(probs.round(3))
print(y_test.iloc[:3].tolist())                        # the true answers

coefs = pd.Series(model[-1].coef_[0], index=X.columns).sort_values()
print(coefs.tail(3).round(2))                          # strongest pushes towards malignant

What this does

  • X_test.iloc[:3] takes the first three rows of the test set by position. predict_proba returns two probabilities for each mass: one column for class 0 (benign) and one for class 1 (malignant). [:, 1] keeps the second column, the probability of malignant.
  • y_test.iloc[:3].tolist() prints the true labels of the same three masses as a plain list, so you can compare.
  • model[-1] picks the last step of the pipeline, the fitted logistic regression. Its coef_ holds the weights as a table with one row; [0] takes that row. Wrapping it in a pd.Series with the column names as the index pairs each weight with its feature, and sort_values() orders them from most negative to most positive.
  • tail(3) shows the last three, which are the three largest positive weights.
[0. 1. 0.043] [0, 1, 0] worst symmetry 1.06 radius error 1.23 worst texture 1.43 dtype: float64

Reading the output

  • [0. 1. 0.043]: the predicted chance of malignancy for the three masses. NumPy prints 0. and 1. for numbers that round to 0.000 and 1.000 at three decimal places; they are very close to 0 and 1, not exactly 0 and 1. The third mass has a 4.3% chance.
  • [0, 1, 0]: the truth. The first and third masses were benign, the second malignant.
  • The three weights: worst texture (1.43), radius error (1.23) and worst symmetry (1.06). Because the features are standardised, each weight says how much the model's score rises when that feature goes up by one standard deviation, all else equal. A positive weight pushes towards malignant. dtype: float64 means the values are decimal numbers.

All three predictions match the truth. Read the weights with care: many features are correlated (radius, perimeter and area measure nearly the same thing), so credit is split between them. Weights describe what this model relies on, not what causes cancer.

Two features are correlated when they tend to rise and fall together. Perimeter is about 2π times the radius, and area about π times the radius squared, so a large nucleus scores high on all three. When the model can get the same information from three columns, it may spread the weight among them in a way that changes from one dataset to the next. That is why the weight on any one size feature cannot be read as the importance of nuclear size.

Correlation can even flip a sign. Two near-copies of the same measurement may end up with one large positive weight and one smaller negative one, which together add up to a modest positive effect. A negative weight on mean perimeter, say, would not mean that larger nuclei are safer. The weights are also only comparable because the features were standardised first; on the raw scale, a weight for area (hundreds) and one for smoothness (around 0.1) would be in quite different units.

The probabilities are often more useful than the yes-or-no answer. model.predict() calls a mass malignant when its probability is above 0.5, but nothing forces that choice. For a screening test, missing a cancer is worse than a false alarm, so you might lower the threshold:

Python
p_test = model.predict_proba(X_test)[:, 1]
preds_03 = (p_test > 0.3).astype(int)   # call malignant above 30%

A lower threshold flags more masses: it catches more cancers and raises more false alarms. Which threshold is right depends on the cost of each kind of mistake, and it must be chosen without looking at the test set. The next lesson, Evaluating a classifier, gives you the measures to make that choice.

Worked example · from a weighted sum to a probability

weighted sum of the scaled features (z) = 2.0
probability = 1 ÷ (1 + e−z) = 1 ÷ (1 + 0.135) = 0.88
z = 0 → 0.50 · z = −3 → 0.05

The logistic function squeezes any number into a probability between 0 and 1. The model predicts “malignant” when the probability passes a threshold, 0.5 by default.

Step by step. The model multiplies each scaled feature by its weight, adds them up and adds a constant; call the total z. Here z = 2.0. Then e−2, where e is the number 2.718…, equals 0.135. Add 1 to get 1.135. Divide 1 by 1.135 to get 0.881, which rounds to 0.88. For z = 0, e0 = 1, so the probability is 1 ÷ 2 = 0.50. For z = −3, e3 = 20.1, so the probability is 1 ÷ 21.1 = 0.047, about 0.05.

What the answer means. A z of 0 sits exactly on the fence. Positive sums give probabilities above 0.5 and lead to a "malignant" prediction; negative sums give probabilities below 0.5 and a "benign" one. The further z is from 0, the more confident the model is. The figure below draws the whole curve.

Your first classifier · LSMA free course An S-shaped curve turns the weighted sum z into a probability between 0 and 1. Marked points: z of minus 3 gives 0.05, z of 0 gives 0.50, and z of 2 gives 0.88. A dashed line at 0.5 marks the default threshold. FROM WEIGHTED SUM TO PROBABILITY 1 0.5 0 −6−4−20246 weighted sum of the scaled features, z P(malignant) z = −3 → 0.05 z = 0 → 0.50 z = 2 → 0.88 threshold 0.5
Whatever the value of z, the curve stays between 0 and 1. Masses to the right of z = 0 rise above the dashed threshold and are predicted malignant. The three dots are the values from the worked example.

Cite scikit-learn as Pedregosa et al. (2011), “Scikit-learn: machine learning in Python”, Journal of Machine Learning Research, and give the version you used. The documentation is at scikit-learn.org/stable.

One caveat when you compare weights with other software. scikit-learn’s LogisticRegression applies a penalty on large weights by default (L2 regularisation with C=1), so its weights are shrunk towards zero. Textbook logistic regression, such as R’s glm(), has no penalty. The two will give different coefficients on the same data, and neither is wrong. Say which you used. Tuning, interpreting and reporting explains how to choose the penalty.

Key point

Never scale, select features or tune on the whole dataset before splitting. Anything learned from the test set leaks into training and inflates the result. A pipeline prevents this by design.

A second example of the same trap: filling in missing values. If you replace each missing blood test with the average of all patients, test patients included, the training data now carry a trace of the test set. The fix is the same: put the filling-in step inside the pipeline, so its averages come from the training rows alone.

Further reading · Module 4Chicco 2017: ten practical tips for machine learning in biology. · All reading for this course

Check your understanding

1. What does model.fit(X_train, y_train) do?

2. What does stratify=y do?

3. A mass has a predicted probability of 0.043. With a threshold of 0.5, its label is:

Up next

Lesson 3

Evaluating a classifier

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.