Guided project
Module 4 · Machine learning · Self-checked
Build, compare and report a regression model on the diabetes dataset from Module 4, lesson 6, in one notebook of your own. Work through the six tasks, check each against its checkpoint, and tick it off. Nothing is submitted.
Research question
How well can baseline measurements predict diabetes progression one year later, and which measurements matter most?
How to approach this
The question has two halves, and they need different tools. "How well" is answered by error measures on data the model has not seen: MAE next to a baseline, and R². "Which measurements matter most" is answered by interpretation: permutation importance and the lasso's coefficients, read with the cautions about correlated features from Regression: predicting a number and Tuning, interpreting and reporting.
This project draws on the whole course, not only this module. Loading the table, counting missing values and checking that ranges are plausible are the pandas habits from Python basics. The histogram of the target and the predicted-against-true plot should follow Figures that tell the truth: labelled axes in the target's units, the same scale on both axes of the scatter, and no cropping that flatters the fit. And if you use an AI assistant to draft any of the code, the rules from Coding with AI assistants apply in full: read each line before you run it, and test that it does what you meant.
That last point matters most in task 3. A plausible-looking comparison loop can contain exactly the mistakes this module warned about: cross-validating on all 442 patients instead of the training set, creating new folds for each model so that they face different splits, or scaling the data before the loop instead of inside a pipeline. Each produces numbers that look reasonable and are subtly wrong. Check for all three before you trust the table.
Know in advance what a good result looks like here, so that a surprising one makes you suspicious rather than pleased. These ten measurements explain somewhat under half the variation in progression, as Regression: predicting a number found. A model that reaches an R² of 0.8 on this dataset is far more likely to contain a leak than a breakthrough.
Work in one notebook, top to bottom, so that anyone can rerun it and get your numbers. A few habits make that possible:
- Fix every random seed the tasks give you (
random_state=42for the split,random_state=1for the folds). Your numbers should then match the checkpoints closely. - Do not look at the test set until task 4. Exploring, comparing and choosing all happen on the training set. The test set is scored once, for the model you have already chosen.
- Write numbers down as you go, in a text cell under each result. Task 6 is much quicker when every figure you need is already recorded next to the code that produced it.
Three terms in the tasks are new. KFold is the plain version of the fold splitter from Trees and ensembles; there is no "stratified" version here, because the target is a number, not a class. min_samples_leaf=5 tells a random forest that every leaf must hold at least five patients, which stops the trees splitting down to single patients and overfitting. max_depth=3 limits each boosting tree to three questions in a row. The figure shows how the six tasks connect.
Explore
What machine learning is · Regression: predicting a number
- Load the data with
scaled=False; report rows, columns and missing values. - Plot the distribution of the target and its correlation with each feature.
Hint. Start from the loading code in Regression: predicting a number. pandas can count the missing values in every column at once, and it can compute the correlation of every column with every other; you only need the row or column for the target. A correlation runs from −1 to 1: positive means the feature tends to be higher when progression is higher. A histogram is the usual way to show the distribution of one number. Remember that a correlation looks at one feature at a time, so it can disagree with the weights of a model that uses all ten.
Checkpoint: 442 patients, 10 features, no missing values; BMI and s5 have the strongest positive correlation with the target.
Split and set a baseline
Your first classifier · Regression: predicting a number
- Hold out 20% as a test set with
random_state=42. - Compute the MAE of always predicting the training mean.
Hint. Regression: predicting a number already did exactly this; reuse its split so that your test set matches. The baseline must use the mean of the training targets, not of the test targets, and it is compared with the test targets. If your baseline is far from the checkpoint, check which mean you used and whether the split was stratified (it should not be, for a numeric target).
Checkpoint: baseline test MAE of about 64.
Compare four models
Cross-validation and data leakage · Trees and ensembles · Regression: predicting a number
- Linear regression and lasso (in pipelines with scaling), a random forest (
min_samples_leaf=5) and gradient boosting (max_depth=3). - Use the same 5-fold
KFold(shuffle=True, random_state=1)on the training set; report mean and spread of R².
Hint. Follow the comparison loop from Trees and ensembles: a dictionary of models, one folds object made once and passed to every model, and cross_val_score on the training data. Two things change. The models are the regression versions (the classes whose names end in "Regressor" for the tree models), and the score is R² instead of AUC. Print the mean and standard deviation for each model, as Trees and ensembles did. Trees do not need the scaling step.
Checkpoint: linear regression about 0.49 and the lasso about 0.48; the random forest about 0.45–0.46; gradient boosting lower, about 0.36 with GradientBoostingRegressor or 0.42 with HistGradientBoostingRegressor. The fold spreads are wide, about ± 0.07 to ± 0.10. The linear models come out ahead, and the tree models do not win here.
Test once
Regression: predicting a number · Tuning, interpreting and reporting
- Choose one model from the cross-validation results, fit it on the full training set and score it on the test set.
- Plot predicted against true values with a diagonal line.
Hint. Decide which model you will test, and write down why, before you score anything on the test set. When the top cross-validated scores are this close, a simpler model is a reasonable choice, as Trees and ensembles argued. On the plot, put the true values on one axis and the predictions on the other; the diagonal is where a perfect prediction would sit, so the vertical distance from it is each patient's error. Report MAE and R² on the test set, next to the baseline from task 2.
Checkpoint: for the lasso, test R² about 0.47 and MAE about 43, clearly better than the baseline.
Interpret
Tuning, interpreting and reporting
- Compute permutation importance on the test set.
- Compare it with the lasso coefficients and explain any difference.
Hint. Tuning, interpreting and reporting used permutation importance on a classifier with AUC; here choose a regression score such as R². Then put the importances and the lasso coefficients side by side. Where they disagree, think about correlated features: Regression: predicting a number showed that total and LDL cholesterol (s1 and s2) carry overlapping information, and that the lasso dropped one of them. Neither ranking is a statement about what causes progression.
Checkpoint: BMI and s5 are the two most important features by a wide margin.
Report
Regression: predicting a number · Tuning, interpreting and reporting
- Write a short report in text cells, using the template from Tuning, interpreting and reporting as a guide.
- State the limits: 442 patients from one study, a moderate R², no external validation.
Hint. Answer the research question directly in the first two sentences: how well, in the target's units, and which measurements. Then give the methods in one sentence each, in the order you did them. Use the numbers you recorded, not numbers recalled from memory. The key point of Regression: predicting a number applies: an R² on its own means little, so put the MAE and the baseline beside it.
Checkpoint: your report gives MAE in the target's units next to the baseline, and says the model explains under half the variation.
A few pitfalls catch many people on this project. Each has a quick check.
- Scores that are negative or oddly scaled. For a regression model,
cross_val_scoreuses R² by default, soscoring="r2"changes nothing. If you ask for MAE instead, usescoring="neg_mean_absolute_error"and remember the minus sign, as in Regression: predicting a number. - A test score that does not match the checkpoint. First check the split:
random_state=42, 20% held out, nostratify. Then check that you fitted the chosen model onXd_trainonly, and scored it onXd_testonly. - Permutation importance with the wrong score. Pass
scoring="r2"(or an error measure) topermutation_importance. AUC, used in Tuning, interpreting and reporting, only makes sense for a classifier. - Reporting the cross-validation score as the final result. Cross-validation chose the model; the test set judges it. Report both, and say which is which.
When your numbers differ slightly from the checkpoints, for example in the third decimal place, that is usually a difference in library version, not an error. When they differ in the first decimal place, look for a mistake in the steps above.
Go deeper: why the predicted-versus-true plot looks flattened
In task 4, the points will not lie along the diagonal. Patients with low true scores tend to be over-predicted, and patients with high scores under-predicted. This is expected, not a bug. When a model explains only part of the variation, its best guess for each patient is pulled towards the average, a version of regression to the mean.
There is a worked number behind it. For a least-squares linear model on the data it was fitted to, the variance of the predictions equals R² times the variance of the true values. With R² near 0.47, the predictions spread only about √0.47 ≈ 0.69 as widely as the truth. Note this in your report. It means the model will rarely predict the most extreme patients, which is exactly where a clinician might most want a warning.
This project closes the course, and it is worth seeing how far the pieces have come together. In the first module you learned to load and clean a table and write code you could rerun. In the second you learned that a figure can mislead as easily as a number, and how to make one that does not. In the third you learned to treat code, your own or an assistant's, as something to read and test rather than trust. In this module you used all three to build a model, judge it on data it had not seen, and say plainly what it can and cannot do. The habits matter more than any one library: check the data, keep the test set sealed, compare with a baseline, show the uncertainty, and state the limits.
Take it further
Share the notebook on GitHub with a short README. Then continue with the AI, Machine Learning & Data Science diploma, which goes on to deep learning, language models and deployment.
GitHub displays Jupyter notebooks with their outputs, so a reader can see your results without running anything. Before you share, restart the notebook and run every cell from the top: this proves the notebook works in order and that every number in your report comes from the code as it stands.
Further reading · Module 4Chicco 2017: ten practical tips for machine learning in biology. · All reading for this course
Up next
Finish
Course complete