LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 35 lessons

Lesson 6 of 9

Module 4 · Machine learning

Regression: predicting a number

Regression: predicting a number

Regression: predicting a number

Not every outcome is a category. This lesson predicts disease progression one year after baseline for 442 patients with diabetes, and judges the result in the units that matter.

⏱ 13 min readModule 4 · Machine learningFree
Lesson at a glance3 steps · about 13 min
  1. The diabetes dataset
  2. A baseline, then a linear model
  3. Coefficients, correlation and the lasso

Before you start. This lesson uses imports from Your first classifier. If you open a new notebook, run this setup cell first.

Python · setup
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

The diabetes dataset

Each patient has age, sex, body mass index (bmi), average blood pressure (bp) and six blood serum measurements (s1–s6, such as cholesterol and triglycerides). The target is a measure of disease progression a year later. It ships with scikit-learn; scaled=False keeps the original units.

This is a regression problem: the label is a number on a scale, not a category. Body mass index is weight in kilograms divided by height in metres squared. Serum is the liquid part of blood left after it clots, and it is where most routine blood chemistry is measured. According to the scikit-learn documentation, the six serum columns are total cholesterol (s1), LDL cholesterol (s2), HDL cholesterol (s3), the ratio of total to HDL cholesterol (s4), a triglyceride measure (s5) and blood sugar (s6). The progression score has no everyday unit; what matters is that larger numbers mean more progression, and that errors are measured on the same scale.

Without scaled=False, scikit-learn returns a version of these data in which every feature has already been centred and rescaled, which makes the values hard to read. Keeping the original units lets you check that ages and blood pressures look sensible.

Python
from sklearn.datasets import load_diabetes

d = load_diabetes(as_frame=True, scaled=False)
print(d.frame.shape)
print(d.frame[["age", "bmi", "bp", "s5", "target"]].describe().loc[["mean", "min", "max"]].round(1))

What this does

  • load_diabetes(as_frame=True, scaled=False) loads the data as pandas tables, in their original units.
  • d.frame.shape gives rows and columns of the full table, which holds the ten features and the target.
  • The second print line chains four steps. It picks five columns; describe() computes summary statistics for each (count, mean, standard deviation, minimum, quartiles and maximum); .loc[["mean", "min", "max"]] keeps only three of those rows; and round(1) rounds to one decimal place.
(442, 11) age bmi bp s5 target mean 48.5 26.4 94.6 4.6 152.1 min 19.0 18.0 62.0 3.3 25.0 max 79.0 42.2 133.0 6.1 346.0

Reading the output

There are 442 patients and 11 columns: 10 features and the target. Each column of the summary is one variable and each row one statistic. The patients are aged 19 to 79, with an average of 48.5. Their BMI runs from 18.0 to 42.2, and average blood pressure from 62 to 133. The target ranges from 25 to 346 with a mean of 152.1. That range is the scale against which you will judge the errors below: a miss of 40 is large for a patient near 25, but modest across a range of more than 300.

This quick look is the same habit as in Cleaning and combining data: before modelling, check that every column holds plausible values. A BMI of 4 or a blood pressure of 900 would mean a unit or typing error, and a model would treat it as real. Here the ranges look sensible. Notice, too, that sex is stored as a number with two values. With two values that is harmless for a linear model, but a column coded 1, 2, 3 for three unordered groups would wrongly tell the model that group 3 is "more" than group 1. Such columns need one-hot encoding, one 0/1 column per group, before modelling.

Is 442 patients enough? A rough rule for linear models is at least 10 to 20 samples per feature. With 10 features and about 350 training patients, this dataset clears that easily, which is part of why a simple linear model works reasonably here.

Cite the dataset as Efron, Hastie, Johnstone and Tibshirani (2004), “Least angle regression”, Annals of Statistics, where it was introduced. The lasso comes from Tibshirani (1996), “Regression shrinkage and selection via the lasso”, Journal of the Royal Statistical Society, Series B.

Little is published about how the patients were recruited, and the documentation describes the target only as a measure of disease progression one year after baseline. The scikit-learn documentation also says s5 is “possibly” the log of triglycerides. In the default scaled version, each feature is centred and divided by its standard deviation times the square root of the number of samples. Treat these as teaching data, not clinical evidence.

A baseline, then a linear model

A baseline is the simplest prediction you could make without a model. For regression, the usual baseline is to predict the training average for everyone. Linear regression instead learns one weight per feature and predicts a weighted sum plus a constant, the straight-line idea from school maths extended to ten features.

Python
from sklearn.linear_model import LinearRegression, LassoCV
from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score

Xd, yd = d.data, d.target
Xd_train, Xd_test, yd_train, yd_test = train_test_split(Xd, yd, test_size=0.2, random_state=42)

base = np.full(len(yd_test), yd_train.mean())          # always predict the average
print("baseline MAE", round(mean_absolute_error(yd_test, base), 1))

lin = make_pipeline(StandardScaler(), LinearRegression()).fit(Xd_train, yd_train)
pred = lin.predict(Xd_test)
print("linear MAE", round(mean_absolute_error(yd_test, pred), 1),
      "RMSE", round(root_mean_squared_error(yd_test, pred), 1),
      "R2", round(r2_score(yd_test, pred), 3))

What this does

  • The imports bring in two models (plain linear regression and the lasso, used later) and three error measures.
  • d.data holds the 10 features and d.target the progression scores; the line stores them as Xd and yd.
  • train_test_split holds back 20% of the patients, 89 of 442. There is no stratify here: the target is a number, not a class, so there are no class shares to keep equal.
  • np.full(len(yd_test), yd_train.mean()) builds an array as long as the test set, in which every entry is the training mean. That is the baseline prediction. Note that it uses the training mean, because the test set is not supposed to be known.
  • lin is a pipeline of a scaler and a linear regression, fitted on the training set. lin.predict(Xd_test) predicts a score for each test patient.
  • The final print compares the predictions with the truth using three measures, rounded to one or three decimal places.
baseline MAE 64.0 linear MAE 42.8 RMSE 53.9 R2 0.453

Reading the output

Always predicting the average misses by 64.0 units on average. The linear model misses by 42.8 on average, with an RMSE of 53.9 and an R² of 0.453. The worked example below unpacks each of these.

Numbers alone hide where the errors fall. A plot of predicted against true values, like the scatter plots in Comparisons and uncertainty, shows it at a glance:

Python
import matplotlib.pyplot as plt

plt.scatter(yd_test, pred, s=14, color="#0F1B33")
plt.plot([25, 346], [25, 346], "--", color="grey")   # perfect predictions
plt.xlabel("True progression"); plt.ylabel("Predicted progression")
plt.show()

Points on the dashed line are perfect predictions. With an R² well below 1, expect a broad cloud around it, and look for a common pattern: the predictions are squeezed towards the middle, so patients with low progression tend to be over-predicted and those with high progression under-predicted. That is what a model does when it can explain only part of the variation; it hedges towards the average. If the errors instead grow steadily with the true value, or curve away from the line, the model is missing something systematic.

One pitfall in the code above: with no classes to stratify, a single random split of 89 patients can be unusually easy or hard. Cross-validation works for regression as well. cross_val_score(lin, Xd, yd, cv=5, scoring="neg_mean_absolute_error") gives five MAEs, but as negative numbers, because scikit-learn always treats a higher score as better. Put a minus sign in front before you report them.

Regression: predicting a number · LSMA free course Six patients' true progression scores are shown as dots. On the left, the baseline predicts the same average for all, leaving long error lines. On the right, a model's predictions sit closer to each true value, leaving shorter error lines. MAE is the average length of these lines. 1 · BASELINE: THE MEAN 2 · LINEAR MODEL progression progression true value prediction error MAE = average length of the grey lines
Each column is one patient. The baseline (left) gives everyone the same prediction, the dashed line; the model (right) adjusts each prediction to the patient. Shorter grey lines mean a smaller MAE. Schematic: six made-up patients, not the real data.

Worked example · reading regression errors

MAE 42.8 → on average, predictions are 42.8 units from the truth (the baseline is 64.0 off)
RMSE 53.9 → like MAE, but large misses count more
R² 0.453 → the model explains about 45% of the variation between patients

A useful improvement on the baseline, but more than half the variation is unexplained. Say that plainly when you report it.

Step by step. The error for one patient is the true value minus the prediction. MAE (mean absolute error) drops the minus signs and averages: if three patients are missed by 10, 20 and 60, the MAE is (10 + 20 + 60) ÷ 3 = 30. RMSE (root mean squared error) squares each error first, averages, and takes the square root: (100 + 400 + 3,600) ÷ 3 = 1,366.7, and its square root is 37.0. RMSE is above the MAE of 30 because squaring gives the miss of 60 extra weight. R² compares the model's squared errors with those of predicting the average: R² = 1 − (model's squared errors ÷ average-only squared errors). An R² of 0.453 means the model removes about 45% of the squared error that predicting the average would leave.

What the answer means. The model cuts the typical miss from 64.0 to 42.8, a drop of 21.2 units, about a third. That is real, but a typical miss of 42.8 on a scale that runs from 25 to 346 is still large for any one patient. The model could help sort patients into broad groups; it could not give a precise forecast for an individual.

Go deeper: why R² can be negative

R² compares a model with predicting the average. Which average, though? r2_score uses the mean of the test targets, which a real model could never know in advance. So the honest baseline from this lesson, the training mean, scores slightly below zero: on this split its test R² is about −0.01.

R² has no lower limit on new data. A model that predicts worse than the average gets a negative R², and an overfitted model can easily do that. The 0 to 1 range you may have met applies only to a linear model with a constant, scored on the same data it was fitted to. So a negative test R² is not a bug in your code. It is a clear message that the model does worse than guessing the average.

Coefficients, correlation and the lasso

A coefficient is the weight a linear model gives a feature. Looking at them shows what the model relies on. The lasso is a version of linear regression with a penalty on the size of the coefficients: it prefers smaller weights, and it will set a weight to exactly zero if the feature adds too little. How strong the penalty should be is a setting, and LassoCV chooses it by cross-validation.

Python
print(pd.Series(lin[-1].coef_, index=Xd.columns).round(1).sort_values())

lasso = make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=0)).fit(Xd_train, yd_train)
lc = pd.Series(lasso[-1].coef_, index=Xd.columns).round(1)
print("kept", (lc != 0).sum(), "of", len(lc), "· dropped", lc[lc == 0].index.tolist())
print("lasso R2", round(r2_score(yd_test, lasso.predict(Xd_test)), 3))

What this does

  • The first line takes the linear regression from the end of the pipeline (lin[-1]), pairs each coefficient with its feature name, rounds to one decimal place and sorts from most negative to most positive.
  • The next line builds and fits a second pipeline: the same scaler, then LassoCV. cv=5 means it tries a range of penalty strengths and picks the best by 5-fold cross-validation within the training set. random_state=0 keeps the result repeatable.
  • lc holds the lasso's coefficients, rounded to one decimal place. (lc != 0).sum() counts the non-zero ones; lc[lc == 0].index.tolist() lists the names of those that are zero. Because the values were rounded first, a weight smaller than 0.05 in size would also show as zero.
  • The last line scores the lasso on the test set with R².
s1 -44.4 sex -11.5 age 1.8 s6 2.4 s3 7.7 s4 13.1 bp 16.8 s2 24.6 bmi 25.6 s5 35.2 dtype: float64 kept 7 of 10 · dropped ['age', 's2', 's4'] lasso R2 0.471

Reading the output

The first block lists the ten linear regression coefficients in order. s1 has the largest negative weight (−44.4) and s5 the largest positive (35.2). The next line says the lasso kept 7 features and set three to zero: age, s2 and s4. The last line gives its test R², 0.471, a little above the plain linear model's 0.453.

Regression: predicting a number · LSMA free course Horizontal bars show the ten linear regression coefficients from the output, from s1 at minus 44.4 to s5 at 35.2. The bars for age, s2 and s4, the three features the lasso dropped, are drawn faded. CHANGE IN PROGRESSION PER STANDARD DEVIATION s1 −44.4 sex −11.5 age 1.8 dropped by lasso s6 2.4 s3 7.7 s4 13.1 dropped by lasso bp 16.8 s2 24.6 dropped by lasso bmi 25.6 s5 35.2 0 ← lowers predicted progression raises predicted progression →
The bars are the plain linear regression weights from the output above; faded bars mark the three features the lasso set to zero. Note the pair s1 and s2: large weights pulling in opposite directions.

Because features are scaled, each coefficient is the change in progression for one standard deviation of that feature. BMI, blood pressure and s5 (a triglyceride measure) push progression up. The large opposite weights on s1 and s2 are a warning sign: they are total and LDL cholesterol, which are strongly correlated, so the model balances one against the other. The lasso shrinks weights and sets some to zero; here it drops three features and does slightly better on the test set.

Why are opposite weights a warning sign? LDL cholesterol makes up a large part of total cholesterol, so the two columns rise and fall together. The model can then raise one weight and lower the other with almost no change to its predictions, and the exact pair it settles on depends on small quirks of the training data. Neither weight can be read on its own as "the effect of total cholesterol". This problem is called collinearity. The lasso eases it by keeping one of a correlated pair and setting the other to zero; here it kept s1 and dropped s2. A different training set could make the opposite choice, so the lasso's choice is not a biological finding either.

Linear models are not the only option for a number. Every classifier from Trees and ensembles has a regression twin, such as RandomForestRegressor and HistGradientBoostingRegressor, used in exactly the same way with .fit and .predict. They can capture curved effects and interactions. Remember their limit, though: a tree-based model cannot predict beyond the range of targets it saw in training, whereas a linear model will extend its straight line, sensibly or not. For a clinical score that has a natural ceiling, either behaviour can be the wrong one, so check predictions at the extremes.

The standardised coefficients answer "which feature moves the prediction most per standard deviation?". A clinician will often want a different question answered: "how much does progression change per extra unit of BMI?". To get that, divide each standardised coefficient by the feature's standard deviation in the training data. The ranking of features can change when you do, which is one more reason to say in any report which scale your weights are on.

Key point

Report regression errors in the outcome's own units (MAE) as well as R², and always next to a baseline. “R² = 0.45” means little on its own.

A second example: the same R² can hide very different errors. Here, R² 0.453 goes with a typical miss of 42.8 units. On an outcome whose values spread far more widely, the same R² would mean a much larger typical miss, and on a narrow one a smaller miss. Only the MAE, next to the baseline's 64.0, tells a clinician how far off a prediction for one patient is likely to be.

Further reading · Module 4Chicco 2017: ten practical tips for machine learning in biology. · All reading for this course

Check your understanding

1. What does an MAE of 42.8 mean here?

2. Why might two correlated features get large weights of opposite sign?

3. What does the lasso do that plain linear regression does not?

Up next

Lesson 7

Unsupervised learning

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.