Lesson 1 of 7
Module 2 · Statistics
Before any test, describe what you have: the centre, the spread and the shape. This lesson uses a real clinical dataset of 189 births to do exactly that.
The birth weight dataset
Module 1 gave you the working tools of R: vectors and functions, data frames read from files, cleaning with dplyr and tidyr, and figures with ggplot2. Module 2 puts those tools to work on questions. Before you can say whether two groups really differ, you need to describe each one honestly, so the module starts with the numbers that summarise a variable and builds towards tests, regression and a complete analysis of one dataset.
birthwt comes with R in the MASS package. It records 189 births at a US medical centre in 1986: the baby's weight in grams (bwt), whether it was under 2,500 g (low), and the mother's age, weight (lwt, in pounds), smoking during pregnancy and recorded race. The numeric codes become labelled factors first.
In the dataset each row is one birth and each column one variable, a measured characteristic. Some variables are continuous, such as birth weight in grams, where any value in a range is possible; others are categorical, such as smoking, where each baby falls into one group. The two kinds are described differently: a continuous variable by its centre and spread, a categorical one by counts and percentages. In the raw data the categories are stored as numbers, 0 and 1 for smoking, 1 to 3 for race, which R would happily average as if they were measurements. Turning them into labelled factors prevents that mistake and makes every table and plot readable.
The recoding uses two tools from Module 1. In Cleaning data with dplyr and tidyr you changed columns with mutate(); in Data frames and files you set the order of groups with factor() and its levels argument. Here factor() also takes labels, which replace the stored codes with words. The first level becomes the reference group in later tables and models, so listing non-smokers first means smokers will be compared with them.
library(MASS) # comes with R
library(dplyr)
library(ggplot2)
bw <- birthwt |>
mutate(smoke = factor(smoke, levels = c(0, 1), labels = c("non-smoker", "smoker")),
race = factor(race, levels = 1:3, labels = c("white", "black", "other")),
low = factor(low, levels = c(0, 1), labels = c("normal", "low")))
nrow(bw)
summary(bw$bwt)
c(sd = sd(bw$bwt), iqr = IQR(bw$bwt))What this does, line by line
library(MASS),library(dplyr)andlibrary(ggplot2)load the three packages. MASS is installed with R, so it needs noinstall.packages().bw <- birthwt |> mutate(…)makes a copy of the data, calledbw, with three columns changed. The originalbirthwtis left as it was.factor(smoke, levels = c(0, 1), labels = c("non-smoker", "smoker"))says: the codes 0 and 1 are the levels, and they should be shown as "non-smoker" and "smoker". The order oflevelsandlabelsmust match. The same pattern turns race codes 1, 2 and 3, and the low-weight codes 0 and 1, into labels.nrow(bw)checks the number of births.summary(bw$bwt)gives six numbers that describe birth weight.c(sd = sd(bw$bwt), iqr = IQR(bw$bwt))calculates the standard deviation and interquartile range and puts them side by side, named, in one vector.
Reading the output. There are 189 births. The six numbers from summary() are the minimum (709 g), the first quartile (2,414 g: a quarter of babies weighed less), the median (2,977 g: half weighed less), the mean (2,945 g), the third quartile (3,487 g: three quarters weighed less) and the maximum (4,990 g). The last two numbers are the SD, 729.2 g, and the IQR, 1,073 g, which is the third quartile minus the first: 3,487 − 2,414 = 1,073.
Checking the size of the data before summarising it is a habit worth keeping. If nrow() does not match the number of births you expect, something went wrong when the data were read or filtered. summary() would also add a count of NA's if any values were missing. birthwt has none, which is unusual: real clinical data rarely arrive so complete, and mean() or sd() return NA if even one value is missing, unless you add na.rm = TRUE. When you do drop missing values, report how many, because the babies with missing data may differ from the rest.
One message to expect: MASS also has a select() function, and whichever package is loaded last wins. If select() misbehaves later, write dplyr::select().
When two loaded packages contain functions with the same name, R uses the one from the package loaded most recently and prints a note that the other has been "masked". Writing the package name and two colons before the function, as in dplyr::select(), tells R exactly which one you mean, whatever the loading order.
The mean (2,945 g) and median (2,977 g) are close, so the distribution is roughly symmetric. The standard deviation (729 g) is the typical distance from the mean; the interquartile range (1,073 g) is the width of the middle half. The median and IQR are not pulled by extreme values, so prefer them for skewed data such as hospital stays or read counts.
A few definitions, to be exact. The mean is the sum of the values divided by how many there are. The median is the middle value once they are sorted. The standard deviation is, roughly, the typical distance of a value from the mean: R squares each distance from the mean, adds them up, divides by one less than the number of values, and takes the square root. Squaring is why a single extreme value can inflate the SD a lot.
A distribution is skewed when it has a long tail on one side. Take five genes with read counts of 10, 12, 15, 18 and 945. Their mean is (10 + 12 + 15 + 18 + 945) ÷ 5 = 1,000 ÷ 5 = 200, larger than four of the five values; their median is 15. One very high count moved the mean far from anything typical, while the median stayed put. That is why read counts are usually log-transformed or summarised by medians before they are described.
A practical rule follows. Report the mean and SD when a distribution is roughly symmetric, as birth weight is here, and the median with the first and third quartiles when it is skewed. Writing "mean 200, SD 416" for the five read counts above would describe none of the genes. An SD larger than the mean is a common warning sign for a variable that cannot be negative: it usually means a long right tail, and that the median and quartiles are the better summary.
Go deeper: why software disagrees about quartiles
With 189 values there is no single value exactly a quarter of the way along the sorted list; that point falls between two births, and a rule is needed to fill the gap. Statisticians have proposed several rules, and R's quantile() offers nine, chosen with its type argument. summary() and quantile() use type 7 by default, which interpolates between neighbouring sorted values.
The choice changes little in the middle but more near the ends. For these birth weights, quantile(bw$bwt, type = 6), a rule used by some other statistics programs, gives a first quartile of 2,412 g instead of 2,414 g and a third quartile of 3,515.5 g instead of 3,487 g. Neither is wrong. Hyndman and Fan compared the rules in The American Statistician (1996). If a colleague's quartiles differ slightly from yours, check which rule each program uses before hunting for a mistake.
By group
bw |>
group_by(smoke) |>
summarise(n = n(), mean = mean(bwt), sd = sd(bwt), median = median(bwt), .groups = "drop") |>
mutate(across(mean:median, round))
ggplot(bw, aes(x = bwt, fill = smoke)) +
geom_histogram(binwidth = 250, colour = "white", alpha = 0.8, position = "identity") +
scale_fill_manual(values = c("non-smoker" = "#0F1B33", smoker = "#D9481F")) +
labs(x = "Birth weight (g)", y = "Number of babies", fill = NULL) +
theme_minimal()What the code does
group_by(smoke)splits the rows into smokers and non-smokers. The next step is then done once per group.summarise(n = n(), mean = …, sd = …, median = …, .groups = "drop")calculates one row of summaries per group: the number of babies, the mean, SD and median of birth weight..groups = "drop"removes the grouping afterwards, so later steps see an ordinary table.mutate(across(mean:median, round))rounds every column frommeantomedianto whole grams.- The
ggplot()call draws two histograms on one set of axes. A histogram divides the weights into bins, here 250 g wide (binwidth = 250), and draws a bar for how many babies fall in each.position = "identity"draws the two groups over each other rather than stacked, andalpha = 0.8makes them slightly see-through so both remain visible.
Reading the output. The table has one row per group: 115 non-smokers and 74 smokers, which add up to 189. Non-smokers' babies averaged 3,056 g (SD 753 g, median 3,100 g); smokers' babies averaged 2,772 g (SD 660 g, median 2,776 g). The difference in means is 3,056 − 2,772 = 284 g. Notice how much the groups overlap: the SDs, about 700 g, are more than twice the size of the difference, so many smokers' babies were heavier than many non-smokers' babies.
Babies of mothers who smoked were about 280 g lighter on average. Comparing two groups asks whether a difference that size could be chance.
Describing the groups side by side is the step before any test. The table shows the size of the difference and how much each group varies, and those are the two things a test weighs against each other in Comparing two groups. The same 284 g gap would stand out clearly if birth weights within each group varied by only 50 g, and would be lost in the noise if they varied by 2,000 g. The number of babies matters too, as the standard error below shows.
One pitfall with the histogram: the groups are different sizes, 115 against 74, so the non-smokers' bars are taller partly because there are more of them. To compare shapes rather than counts, put y = after_stat(density) inside aes(), which scales each group's bars so that their total area is 1, or give each group its own panel with facet_wrap(~ smoke), as in Figures with ggplot2.
Remember too that this is a description, not an explanation. Mothers who smoked may differ from those who did not in other ways, such as age, weight or previous premature labour, and any of these could contribute to the gap. Later lessons in this module show how to compare like with like by adjusting for such factors.
The birthwt data come from Hosmer and Lemeshow's textbook Applied Logistic Regression (Wiley; first edition 1989) and were collected at Baystate Medical Center, Springfield, Massachusetts, in 1986. R distributes them in MASS, the package that accompanies Venables and Ripley, Modern Applied Statistics with S (4th edition, Springer, 2002); cite it with citation("MASS"). Treat the data as a teaching set. They are 189 births from a single hospital in one year, so a figure such as the 284 g smoking gap describes these mothers. It is not an estimate for all births today, and should not be quoted as one.
Spread, and the standard error
m <- mean(bw$bwt); s <- sd(bw$bwt) mean(abs(bw$bwt - m) < s) # share within 1 SD of the mean mean(abs(bw$bwt - m) < 2 * s) # share within 2 SD s / sqrt(nrow(bw)) # standard error of the mean
What this does, line by line
m <- mean(bw$bwt); s <- sd(bw$bwt)stores the mean and SD. The semicolon lets two commands share one line.abs(bw$bwt - m)is each baby's distance from the mean, ignoring direction. Comparing it withsgives TRUE for babies within 1 SD. Taking themean()of TRUE/FALSE values gives the share that are TRUE, because TRUE counts as 1 and FALSE as 0.- The second line does the same within 2 SD.
s / sqrt(nrow(bw))is the SD divided by the square root of the number of babies: the standard error of the mean.
Reading the output. 0.677 means 67.7% of babies, 128 of the 189, lie within one SD of the mean, and 0.963 means 96.3%, 182 of 189, lie within two SDs. The standard error is 53.0 g.
The two shares check the 68–95 rule: for data that follow a normal, bell-shaped distribution, about 68% of values lie within 1 SD of the mean and about 95% within 2 SD. Birth weight comes close, so its mean and SD summarise it well. For skewed data the rule fails, and an interval of mean ± 2 SD can even run below zero for a quantity that cannot be negative, another sign to switch to the median and quartiles.
The standard error answers a different question. The SD is a property of the babies and stays roughly the same however many you measure; the SE is a property of the estimate and shrinks as the sample grows. The same calculation works for each group: for smokers, 660 ÷ √74 = 660 ÷ 8.60 ≈ 77 g, and for non-smokers, 753 ÷ √115 = 753 ÷ 10.72 ≈ 70 g. The smokers' mean is less precise mainly because there are fewer of them. The t-test in the next lesson combines these two SEs.
One condition hides inside the formula: the SE assumes every value is an independent observation. If some mothers contributed two babies, or one sample was measured three times, the rows are not independent, and dividing by the square root of the number of rows overstates the precision. Count the independent units, such as patients or animals, not the rows.
Worked example · SD versus standard error
SE = 729 ÷ √189 = 729 ÷ 13.75 = 53 g: how precisely we know the mean
68% within 1 SD and 96% within 2 SD, close to the 68% and 95% expected for a normal distribution
Step by step. The square root of 189 is 13.75, because 13.75 × 13.75 ≈ 189. Dividing the SD by it gives 729.2 ÷ 13.75 = 53.0 g. For the shares, 0.677 × 189 = 128 babies within 1 SD, and 0.963 × 189 = 182 within 2 SD.
What the answer means. The SD describes babies: about a third of them (100% − 67.7% = 32.3%) weighed more than 729 g above or below the average. The SE describes the average itself: if the study were repeated with another 189 births from the same population, the new mean would typically differ from this one by around 53 g. An approximate 95% confidence interval for the mean is the mean plus or minus about 2 SE, roughly 2,840 to 3,050 g. Because the SE divides by the square root of the sample size, collecting four times as many babies would only halve it.
Use the SD to describe your sample and the SE (or, better, a confidence interval) to describe how certain an estimate is. Error bars must say which they show.
Key point
Always plot the data before you summarise it. Two groups can share a mean and SD and still look completely different.
For example, one group of values might form a single hump around 3,000 g, while another forms two separate humps, one near 2,300 g and one near 3,700 g, perhaps because it mixes two kinds of patient. Both can have the same mean and SD, but the second group contains almost no one near its own average. Only a histogram, density plot or strip of points would show it.
Further reading · Module 2Greenland et al. 2016: 25 common misreadings of p-values and confidence intervals. · All reading for this course
Check your understanding
1. Hospital stays are 2, 3, 3, 4 and 30 days. Which describes a typical stay best?
2. A sample of 100 has SD 20. What is the standard error of the mean?
3. What does the IQR measure?
Up next
Lesson 2
Comparing two groups