LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 35 lessons

Guided project

Module 4 · Machine learning · Self-checked

Predicting disease progression

Predicting disease progression

Predicting disease progression

Build, compare and report a regression model on the diabetes dataset from Module 4, lesson 6, in one notebook of your own. Work through the six tasks, check each against its checkpoint, and tick it off. Nothing is submitted.

⏱ 10 min readModule 4 · Machine learning · Self-checkedFree
Lesson at a glance7 steps · about 10 min
  1. How to approach this
  2. Explore
  3. Split and set a baseline
  4. Compare four models
  5. Test once
  6. Interpret
  7. Report

Research question

How well can baseline measurements predict diabetes progression one year later, and which measurements matter most?

How to approach this

The question has two halves, and they need different tools. "How well" is answered by error measures on data the model has not seen: MAE next to a baseline, and R². "Which measurements matter most" is answered by interpretation: permutation importance and the lasso's coefficients, read with the cautions about correlated features from Regression: predicting a number and Tuning, interpreting and reporting.

This project draws on the whole course, not only this module. Loading the table, counting missing values and checking that ranges are plausible are the pandas habits from Python basics. The histogram of the target and the predicted-against-true plot should follow Figures that tell the truth: labelled axes in the target's units, the same scale on both axes of the scatter, and no cropping that flatters the fit. And if you use an AI assistant to draft any of the code, the rules from Coding with AI assistants apply in full: read each line before you run it, and test that it does what you meant.

That last point matters most in task 3. A plausible-looking comparison loop can contain exactly the mistakes this module warned about: cross-validating on all 442 patients instead of the training set, creating new folds for each model so that they face different splits, or scaling the data before the loop instead of inside a pipeline. Each produces numbers that look reasonable and are subtly wrong. Check for all three before you trust the table.

Know in advance what a good result looks like here, so that a surprising one makes you suspicious rather than pleased. These ten measurements explain somewhat under half the variation in progression, as Regression: predicting a number found. A model that reaches an R² of 0.8 on this dataset is far more likely to contain a leak than a breakthrough.

Work in one notebook, top to bottom, so that anyone can rerun it and get your numbers. A few habits make that possible:

  • Fix every random seed the tasks give you (random_state=42 for the split, random_state=1 for the folds). Your numbers should then match the checkpoints closely.
  • Do not look at the test set until task 4. Exploring, comparing and choosing all happen on the training set. The test set is scored once, for the model you have already chosen.
  • Write numbers down as you go, in a text cell under each result. Task 6 is much quicker when every figure you need is already recorded next to the code that produced it.

Three terms in the tasks are new. KFold is the plain version of the fold splitter from Trees and ensembles; there is no "stratified" version here, because the target is a number, not a class. min_samples_leaf=5 tells a random forest that every leaf must hold at least five patients, which stops the trees splitting down to single patients and overfitting. max_depth=3 limits each boosting tree to three questions in a row. The figure shows how the six tasks connect.

Predicting disease progression · LSMA free course Six boxes in two rows: explore the data, split and set a baseline, compare four models by cross-validation, test the chosen model once, interpret it, and report. The test step is highlighted because the test set is used only there. 1 · EXPLORE 2 · BASELINE 3 · COMPARE 4 · TEST ONCE 5 · INTERPRET 6 · REPORT 442 patients, 10 features no missing values hold out 20% baseline MAE about 64 4 models, same folds R² about 0.36–0.49 fit the chosen model score the test set permutation importance vs lasso weights MAE next to baseline state the limits
Read the top row left to right, then the bottom row. Everything in the top row uses the training set only; the highlighted step is the one place the test set is used. Numbers are the checkpoint values. Schematic.
1

Explore

What machine learning is · Regression: predicting a number

  • Load the data with scaled=False; report rows, columns and missing values.
  • Plot the distribution of the target and its correlation with each feature.

Hint. Start from the loading code in Regression: predicting a number. pandas can count the missing values in every column at once, and it can compute the correlation of every column with every other; you only need the row or column for the target. A correlation runs from −1 to 1: positive means the feature tends to be higher when progression is higher. A histogram is the usual way to show the distribution of one number. Remember that a correlation looks at one feature at a time, so it can disagree with the weights of a model that uses all ten.

Checkpoint: 442 patients, 10 features, no missing values; BMI and s5 have the strongest positive correlation with the target.

2

Split and set a baseline

Your first classifier · Regression: predicting a number

  • Hold out 20% as a test set with random_state=42.
  • Compute the MAE of always predicting the training mean.

Hint. Regression: predicting a number already did exactly this; reuse its split so that your test set matches. The baseline must use the mean of the training targets, not of the test targets, and it is compared with the test targets. If your baseline is far from the checkpoint, check which mean you used and whether the split was stratified (it should not be, for a numeric target).

Checkpoint: baseline test MAE of about 64.

3

Compare four models

Cross-validation and data leakage · Trees and ensembles · Regression: predicting a number

  • Linear regression and lasso (in pipelines with scaling), a random forest (min_samples_leaf=5) and gradient boosting (max_depth=3).
  • Use the same 5-fold KFold(shuffle=True, random_state=1) on the training set; report mean and spread of R².

Hint. Follow the comparison loop from Trees and ensembles: a dictionary of models, one folds object made once and passed to every model, and cross_val_score on the training data. Two things change. The models are the regression versions (the classes whose names end in "Regressor" for the tree models), and the score is R² instead of AUC. Print the mean and standard deviation for each model, as Trees and ensembles did. Trees do not need the scaling step.

Checkpoint: linear regression about 0.49 and the lasso about 0.48; the random forest about 0.45–0.46; gradient boosting lower, about 0.36 with GradientBoostingRegressor or 0.42 with HistGradientBoostingRegressor. The fold spreads are wide, about ± 0.07 to ± 0.10. The linear models come out ahead, and the tree models do not win here.

4

Test once

Regression: predicting a number · Tuning, interpreting and reporting

  • Choose one model from the cross-validation results, fit it on the full training set and score it on the test set.
  • Plot predicted against true values with a diagonal line.

Hint. Decide which model you will test, and write down why, before you score anything on the test set. When the top cross-validated scores are this close, a simpler model is a reasonable choice, as Trees and ensembles argued. On the plot, put the true values on one axis and the predictions on the other; the diagonal is where a perfect prediction would sit, so the vertical distance from it is each patient's error. Report MAE and R² on the test set, next to the baseline from task 2.

Checkpoint: for the lasso, test R² about 0.47 and MAE about 43, clearly better than the baseline.

5

Interpret

Tuning, interpreting and reporting

  • Compute permutation importance on the test set.
  • Compare it with the lasso coefficients and explain any difference.

Hint. Tuning, interpreting and reporting used permutation importance on a classifier with AUC; here choose a regression score such as R². Then put the importances and the lasso coefficients side by side. Where they disagree, think about correlated features: Regression: predicting a number showed that total and LDL cholesterol (s1 and s2) carry overlapping information, and that the lasso dropped one of them. Neither ranking is a statement about what causes progression.

Checkpoint: BMI and s5 are the two most important features by a wide margin.

6

Report

Regression: predicting a number · Tuning, interpreting and reporting

  • Write a short report in text cells, using the template from Tuning, interpreting and reporting as a guide.
  • State the limits: 442 patients from one study, a moderate R², no external validation.

Hint. Answer the research question directly in the first two sentences: how well, in the target's units, and which measurements. Then give the methods in one sentence each, in the order you did them. Use the numbers you recorded, not numbers recalled from memory. The key point of Regression: predicting a number applies: an R² on its own means little, so put the MAE and the baseline beside it.

Checkpoint: your report gives MAE in the target's units next to the baseline, and says the model explains under half the variation.

A few pitfalls catch many people on this project. Each has a quick check.

  • Scores that are negative or oddly scaled. For a regression model, cross_val_score uses R² by default, so scoring="r2" changes nothing. If you ask for MAE instead, use scoring="neg_mean_absolute_error" and remember the minus sign, as in Regression: predicting a number.
  • A test score that does not match the checkpoint. First check the split: random_state=42, 20% held out, no stratify. Then check that you fitted the chosen model on Xd_train only, and scored it on Xd_test only.
  • Permutation importance with the wrong score. Pass scoring="r2" (or an error measure) to permutation_importance. AUC, used in Tuning, interpreting and reporting, only makes sense for a classifier.
  • Reporting the cross-validation score as the final result. Cross-validation chose the model; the test set judges it. Report both, and say which is which.

When your numbers differ slightly from the checkpoints, for example in the third decimal place, that is usually a difference in library version, not an error. When they differ in the first decimal place, look for a mistake in the steps above.

Go deeper: why the predicted-versus-true plot looks flattened

In task 4, the points will not lie along the diagonal. Patients with low true scores tend to be over-predicted, and patients with high scores under-predicted. This is expected, not a bug. When a model explains only part of the variation, its best guess for each patient is pulled towards the average, a version of regression to the mean.

There is a worked number behind it. For a least-squares linear model on the data it was fitted to, the variance of the predictions equals R² times the variance of the true values. With R² near 0.47, the predictions spread only about √0.47 ≈ 0.69 as widely as the truth. Note this in your report. It means the model will rarely predict the most extreme patients, which is exactly where a clinician might most want a warning.

This project closes the course, and it is worth seeing how far the pieces have come together. In the first module you learned to load and clean a table and write code you could rerun. In the second you learned that a figure can mislead as easily as a number, and how to make one that does not. In the third you learned to treat code, your own or an assistant's, as something to read and test rather than trust. In this module you used all three to build a model, judge it on data it had not seen, and say plainly what it can and cannot do. The habits matter more than any one library: check the data, keep the test set sealed, compare with a baseline, show the uncertainty, and state the limits.

Take it further

Share the notebook on GitHub with a short README. Then continue with the AI, Machine Learning & Data Science diploma, which goes on to deep learning, language models and deployment.

GitHub displays Jupyter notebooks with their outputs, so a reader can see your results without running anything. Before you share, restart the notebook and run every cell from the top: this proves the notebook works in order and that every number in your report comes from the code as it stands.

Further reading · Module 4Chicco 2017: ten practical tips for machine learning in biology. · All reading for this course

Up next

Finish

Course complete

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.