LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 35 lessons

Lesson 8 of 9

Module 4 · Machine learning

Tuning, interpreting and reporting

Tuning, interpreting and reporting

Tuning, interpreting and reporting

Settings such as regularisation strength must be chosen without touching the test set. Then you need to say what drives the model, whether its probabilities can be trusted, and what it cannot do.

⏱ 14 min readModule 4 · Machine learningFree
Lesson at a glance3 steps · about 14 min
  1. Tuning with GridSearchCV
  2. What drives the model: permutation importance
  3. Can the probabilities be trusted? Calibration

Before you start. This lesson carries on in the notebook from Trees and ensembles. If you open a new notebook, or Colab has restarted, run this setup cell first. It repeats the loading and the split from Your first classifier, and the imports used since.

Python · setup
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

data = load_breast_cancer(as_frame=True)
df = data.frame
df["diagnosis"] = df["target"].map({0: "malignant", 1: "benign"})
X = df[data.feature_names]
y = (df["diagnosis"] == "malignant").astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier

Tuning with GridSearchCV

C controls how strongly logistic regression is regularised: small values give simpler models. GridSearchCV tries each value with cross-validation on the training set only.

A hyperparameter is a setting you choose before training, as opposed to the weights the model learns during training. C is one. Regularisation is a penalty on large weights, which stops the model leaning too hard on any one feature and so guards against overfitting. In scikit-learn's logistic regression, C is the inverse of the penalty's strength: C = 0.01 means a heavy penalty and small, cautious weights, while C = 100 means almost no penalty. The default is 1. Tuning means trying several values and keeping the one that does best, and a grid search tries every value on a list you supply.

Python
from sklearn.model_selection import GridSearchCV

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
grid = {"logisticregression__C": [0.01, 0.1, 1, 10, 100]}
search = GridSearchCV(pipe, grid, cv=StratifiedKFold(5, shuffle=True, random_state=1),
                      scoring="roc_auc")
search.fit(X_train, y_train)
print(search.best_params_, round(search.best_score_, 3))

best = search.best_estimator_
print("test AUC", round(roc_auc_score(y_test, best.predict_proba(X_test)[:, 1]), 3))

# nested cross-validation: an honest estimate of the whole tuning procedure
outer = cross_val_score(search, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=2),
                        scoring="roc_auc")
print("nested", round(outer.mean(), 3))

What this does

  • pipe is the familiar scaler-plus-logistic-regression pipeline. max_iter=5000 allows more fitting steps than before, because some values of C need longer to settle.
  • grid is a dictionary of settings to try. The key names the setting by step and parameter: make_pipeline names each step after its class in lower case, so the logistic regression step is logisticregression, and two underscores join the step name to the parameter C. The list holds the five values to try.
  • GridSearchCV(...) sets up the search: for each value of C, run 5-fold stratified cross-validation with shuffling and score by AUC. That is 5 values × 5 folds = 25 fits.
  • search.fit(X_train, y_train) runs the search on the training set only. When it finishes, it refits the pipeline with the best value on the whole training set.
  • best_params_ is the winning setting, and best_score_ its mean cross-validated AUC. best_estimator_ is the refitted pipeline, which is then scored once on the test set.
  • The last block passes the whole search to cross_val_score, with a different set of five outer folds over all 569 masses. Inside each outer training part, a complete grid search runs again, picks its own best C, and the result is scored on the outer test part.
{'logisticregression__C': 0.1} 0.994 test AUC 0.998 nested 0.996

Reading the output

The first line shows that C = 0.1 won, with a mean cross-validated AUC of 0.994 on the training set. That is a fairly strong penalty: simpler weights did slightly better than the default. The second line is the test AUC of the tuned model, 0.998. The third is the nested estimate, 0.996: the mean AUC over five outer folds, each produced by a full, separate tuning run.

The best setting is chosen inside the training data, then checked once on the test set. Nested cross-validation wraps the whole search in an outer loop, so the estimate includes the effect of tuning; use it when you have no separate test set.

Why is best_score_ not enough on its own? It is the best of five tries, and picking the best of several scores flatters the winner a little, because some of its advantage is luck in those particular folds. The nested estimate avoids this: each outer test part played no role in choosing C. With only five candidates and a strong signal, as here, the difference is small; with hundreds of combinations of settings it can be large.

Notice how the grid is built: each value is ten times the last. Hyperparameters such as C often matter on a ratio scale, so the step from 0.01 to 0.1 matters about as much as the step from 10 to 100, and evenly spaced values such as 1, 2, 3, 4, 5 would waste most of the search. A pitfall to watch for is a winner at the edge of the grid. If C = 0.01 had won, the true best might be smaller still, and you should extend the grid in that direction before trusting the result. Here 0.1 sits inside the range, so the search has bracketed it.

Do not read too much into the test AUC of 0.998 being higher than the cross-validated 0.994. Tuning cannot make a model better on unseen data than on its own folds; the difference is the luck of one test set of 114 masses, as Your first classifier warned. For models with several hyperparameters, a full grid grows fast: three settings with five values each is 125 combinations, each fitted five times. RandomizedSearchCV tries a fixed number of random combinations instead, which usually finds a good region far more cheaply.

Tuning, interpreting and reporting · LSMA free course An outer loop splits the data five times. Inside each outer training part, an inner 5-fold cross-validation tries the five values of C and keeps the best. The tuned model is scored on the outer test part, and the five outer scores average to 0.996. 1 · OUTER SPLIT 2 · INNER SEARCH 3 · SCORE 5 rounds · vermilion = outer test In the 4 training parts, try each value of C: 0.010.1110100 each by inner 5-fold CV keep the best C, refit Score the tuned model on the outer test part. Repeat for each of the 5 outer rounds. nested AUC 0.996
The inner search (middle) runs again inside every outer round, using only that round's training parts, so each outer test part stays unseen until it is scored. The best C can differ between rounds; the nested score judges the whole procedure, not one value of C. Schematic.

Two papers back the advice on nested cross-validation. Varma and Simon (2006), “Bias in error estimation when using cross-validation for model selection”, BMC Bioinformatics, showed that reporting the best tuned cross-validation score can give an optimistic estimate even on data with no signal. Cawley and Talbot (2010), Journal of Machine Learning Research, showed that overfitting during model selection can be as serious as overfitting the model itself.

For reporting a clinical prediction model, follow TRIPOD+AI (Collins et al. 2024, BMJ), which updates the 2015 TRIPOD statement. The template in this lesson covers only part of its checklist. A full report also describes the participants, how missing data were handled and the intended use.

What drives the model: permutation importance

Shuffle one feature in the test set and see how much the score drops. A big drop means the model relies on that feature. It works for any model.

Shuffling one column breaks the link between that feature and the label, while keeping its values realistic and leaving every other column untouched. If the model's AUC hardly changes, the model was not using that feature, or could get the same information elsewhere. Because the method only needs predictions, it works the same way for a forest, a boosted model or logistic regression; that is what "model-agnostic" means.

Tuning, interpreting and reporting · LSMA free course A small test table with three feature columns. The radius column is shuffled between rows while the other columns stay the same. The drop in AUC after shuffling is the feature's importance, averaged over repeated shuffles. 1 · TEST DATA 2 · SHUFFLE ONE 3 · COMPARE radiusareatexture radiusareatexture AUC before − AUC after = importance repeat 20 times, take the mean shading stands for each value radius values swapped between rows
Only the outlined column changes: the same radius values are dealt out to different masses, so radius no longer matches the rest of each row. The other columns, and the model, stay exactly as they were. Schematic.
Python
from sklearn.inspection import permutation_importance

rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(X_train, y_train)
pi = permutation_importance(rf, X_test, y_test, scoring="roc_auc", n_repeats=20, random_state=0)
print(pd.Series(pi.importances_mean, index=X.columns).sort_values(ascending=False).head(5).round(4))

What this does

  • The first line imports permutation_importance.
  • rf is a random forest of 300 trees, as in Trees and ensembles, trained on the training set. The method is shown on a forest to make the point that it works for any model.
  • permutation_importance(...) takes the fitted model and the test data. For each of the 30 features it shuffles that column, re-scores the model by AUC, and records the drop. n_repeats=20 repeats the shuffle 20 times per feature, because one shuffle can be lucky or unlucky; random_state=0 makes the shuffles repeatable.
  • pi.importances_mean holds the average drop for each feature. The last line pairs these with the feature names, sorts from largest to smallest, keeps the top five and rounds to four decimal places.
worst radius 0.0090 worst area 0.0090 worst concave points 0.0066 worst perimeter 0.0061 mean concave points 0.0027 dtype: float64

Reading the output

Each line is a feature and the average drop in test AUC when it is shuffled. Shuffling worst radius or worst area costs about 0.009 of AUC each; worst concave points and worst perimeter a little less; mean concave points less again. Four of the top five are "worst" features, and three of them (radius, area, perimeter) describe nuclear size.

The drops are small because correlated features cover for each other: shuffle radius and the model still has area. Importance describes the model, not biology, and correlated features share it.

Put the two ideas together. A drop of 0.009 out of an AUC near 1 looks tiny, yet nuclear size clearly matters for this diagnosis. Shuffle radius, area and perimeter together and the drop would usually be larger, because the model would lose its backup copies too. When features are correlated, consider shuffling them as a group, and read single-feature importances as "what the model would miss if this one column alone were lost".

Two practical points. First, compute importance on the test set, as here, not on the training set. A forest can fit its training data almost perfectly, so on training rows it may seem to depend on features that only helped it memorise. Second, each importance is an average of 20 shuffles, and pi.importances_std holds how much those shuffles varied. Two features whose averages differ by less than their spread cannot be ranked against each other, the same reasoning as reading error bars in Comparisons and uncertainty. Here worst radius and worst area tie at 0.0090, and nothing in the output says one matters more.

Go deeper: one patient at a time with SHAP

Permutation importance describes the model as a whole. Sometimes you want to know why it gave one patient a high risk. SHAP values (Lundberg and Lee 2017, NeurIPS) do this. For a single prediction, each feature gets a number saying how far it pushed the prediction up or down from the model’s average output. The numbers add up exactly to the difference between that patient’s prediction and the average.

The idea comes from Shapley values in game theory, a fair way to share a team’s winnings among its players. SHAP needs the separate shap package. The same caution applies as for permutation importance. Correlated features share credit in ways that can change from model to model, and a large SHAP value says what the model used, not what causes disease.

Can the probabilities be trusted? Calibration

A model is calibrated when its probabilities mean what they say: among all masses given a 70% chance of malignancy, about 70% should turn out malignant. A model can rank cases very well (a high AUC) and still be poorly calibrated, for example if all its probabilities are pushed too close to 0 and 1. For a clinician who will act on a stated risk, calibration matters as much as ranking.

Python
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss

p = best.predict_proba(X_test)[:, 1]
observed, predicted = calibration_curve(y_test, p, n_bins=5, strategy="quantile")
print(pd.DataFrame({"predicted": predicted.round(2), "observed": observed.round(2)}))
print("Brier", round(brier_score_loss(y_test, p), 3))

What this does

  • p is the tuned model's predicted probability of malignancy for each of the 114 test masses.
  • calibration_curve sorts the masses into 5 bins by predicted probability. strategy="quantile" makes bins with equal numbers of masses, rather than equal widths of probability. For each bin it returns the share that were truly malignant (observed) and the average predicted probability (predicted); it returns them in that order, which is why the code unpacks observed first.
  • The print line puts the two side by side in a small table, rounded to two decimal places.
  • brier_score_loss computes the Brier score, explained below.
predicted observed 0 0.00 0.00 1 0.01 0.00 2 0.08 0.05 3 0.68 0.78 4 0.99 1.00 Brier 0.023

Reading the output

Each row is one bin, from the lowest predicted risks (row 0) to the highest (row 4). In the lowest two bins the model predicts almost no risk and none of the masses were malignant. In row 2 it predicts 8% and 5% were malignant. In row 3 it predicts 68% and 78% were malignant, so in that middle band it slightly underestimates risk. In the top bin it predicts 99% and all were malignant. The last line is the Brier score, 0.023.

In each fifth of the test set, the average predicted risk is close to the share that were truly malignant, so the probabilities are reasonably calibrated. The Brier score (0.023; lower is better) summarises this. With only 114 test cases, each bin holds about 23 masses, so treat it as a rough check.

The Brier score is the average, over all test masses, of (predicted probability − outcome)², where the outcome is 1 for malignant and 0 for benign. A malignant mass given 0.9 adds (0.9 − 1)² = 0.01; a benign mass given 0.9 adds (0.9 − 0)² = 0.81. Confident mistakes are punished hard, and a perfect model scores 0. The score reflects both ranking and calibration, so read it next to the calibration table rather than instead of it.

What if a model is poorly calibrated? It can often be repaired without retraining from scratch. CalibratedClassifierCV, from sklearn.calibration, wraps a model and learns, by cross-validation on the training data, a mapping from its raw probabilities to better ones. method="sigmoid" fits a smooth S-shaped correction and suits small datasets; method="isotonic" fits a more flexible step-shaped one and needs more data. Logistic regression is often reasonably calibrated out of the box, as here. Tree ensembles and heavily regularised models are more often off, so check before quoting their probabilities as risks.

A calibration check also depends on where the data came from. A model calibrated on a test set that is 37% malignant will give probabilities that are too high in a screening population where cancer is far rarer, for the same reason that PPV fell in Evaluating a classifier. Calibration has to be checked again in each new setting.

0.00.51.00.00.51.0Predicted riskObserved rateWell calibratedMatches reality OverconfidentExtremes overstatedDashed line:perfect calibration
A calibration curve: predicted risk against what actually happened. Illustrative data.

The template below gathers everything from this module into one short paragraph. Each sentence answers a question a reviewer will ask: what model and software, how settings were chosen, what data it was trained and tested on, how well it ranks cases, whether its probabilities can be trusted, and what has not yet been shown.

Template

A logistic regression classifier (scikit-learn version __, regularisation chosen by 5-fold cross-validation, C = 0.1) was trained on 30 nuclear features from 455 fine-needle aspirates and tested on a held-out, stratified set of 114. Test AUC was 0.998 and nested cross-validated AUC 0.996; probabilities were well calibrated (Brier score 0.023). The data come from a single centre and the model has not been validated externally or prospectively.

How to use the template

  • Fill the blank. The software version goes where the underscores are, because results can change between versions.
  • Keep the numbers traceable. Every figure in it, 455, 114, 0.998, 0.996 and 0.023, comes from code you ran in this module.
  • Keep the last sentence. External validation means testing the model on data from other centres; prospective validation means testing it on new patients as they arrive, rather than on records collected in the past. Until both have been done, the honest claim is that the model works on this one dataset.
  • Add the errors when a threshold is used. If the model will make yes-or-no calls, add a sentence giving the threshold and the sensitivity and specificity at it, with the counts behind them, as in Evaluating a classifier. A reader deciding whether to use the model needs to know how many cancers it would miss.

Further reading · Module 4Chicco 2017: ten practical tips for machine learning in biology. · All reading for this course

Check your understanding

1. Where does GridSearchCV choose the best setting?

2. Permutation importance for a feature is near zero. Why might the feature still matter?

3. A model predicts 20% risk for a group in which 50% have the disease. The model is:

Up next

Project

Predicting disease progression

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.