Lesson 2 of 7
Module 2 · Statistics
Is the 280 g difference between smokers' and non-smokers' babies real, or could it be chance? This lesson explains p-values and confidence intervals, and when the samples are paired.
What a p-value is
Assume there is no real difference (the null hypothesis). The p-value is the chance of seeing a difference at least this large anyway. A small p-value means the data would be surprising if there were no effect. It is not the chance that the null hypothesis is true, and it says nothing about how big or important the effect is.
Here is the reasoning in plain steps. First, imagine a world in which smoking has no effect on birth weight. Even in that world, two random groups of babies would differ a little by chance. Second, work out how often chance alone would produce a gap as large as the one you observed. That share is the p-value. If it is very small, either something unusual happened or the no-effect world is the wrong description, and you take the second as evidence against the null hypothesis.
By convention, a p-value below 0.05 is called statistically significant. The cut-off is a habit, not a law of nature: p = 0.049 and p = 0.051 carry almost the same evidence. A tiny effect measured in a huge sample can give a very small p-value, and an important effect in a small sample can miss 0.05. That is why the size of the effect, with its confidence interval, should always be reported alongside the p-value.
Two kinds of error are possible. Calling a difference real when it is only chance is a false positive, also called a type I error; with a cut-off of 0.05 it happens in about 5% of tests where the null hypothesis is true. Missing a difference that is real is a false negative, or type II error. How often that happens depends on the size of the effect and the number of observations, a question this module returns to when it covers power.
Two misreadings are common enough to name. A p-value of 0.007 does not mean a 0.7% chance that smoking has no effect, or a 99.3% chance that the finding is real: the p-value is calculated by assuming the null hypothesis is true, so it cannot also give the probability of that hypothesis. And a p-value above 0.05 does not show that two groups are the same. It means the data are compatible with no difference, and often with sizeable differences as well; the confidence interval shows which. Every p-value also rests on the test's assumptions, such as independent observations. If those fail, the p-value can be badly wrong however precise it looks.
The American Statistical Association's statement on p-values (Wasserstein & Lazar 2016, The American Statistician) sets out the points in this section: a p-value measures how incompatible the data are with a model, not the probability that a hypothesis is true, and not the size of an effect. The sleep data were analysed by "Student" (William Sealy Gosset) in "The probable error of a mean" (Biometrika, 1908), the paper that introduced the t distribution; Welch's unequal-variance version comes from Welch 1947, also in Biometrika. When you report a test, give the exact p-value (p = 0.007) rather than only p < 0.05, and name the test.
Two independent groups: the t-test
t.test(bwt ~ smoke, data = bw)
What this does. The ~ sign, read "by" or "explained by", is R's formula notation: bwt ~ smoke means "birth weight, split by smoking group". data = bw tells R where to find both columns. t.test() then compares the mean birth weight of the two groups. A t-test divides the difference between the two means by its standard error; the result, t, says how many standard errors apart the means are.
This builds directly on Describing data, where you calculated each group's mean, SD and standard error. The t-test combines the two groups' standard errors into one for the difference, then asks how unusual the observed ratio would be if the true difference were zero. The order of the result follows the factor levels: non-smoker is the first level of smoke, so R reports non-smoker minus smoker. Reverse the levels and the signs of t and the interval flip, while the p-value stays the same.
The t-test assumes that the observations are independent, here one baby per mother, and that each group's mean is roughly normally distributed. The second condition is weaker than it sounds. With 74 and 115 babies, means behave close to normally even when the individual values are somewhat skewed, a result known as the central limit theorem. With five or ten values per group, though, the shape of the data matters, and a histogram or strip of points should come before the test.
Reading the output, line by line
- Welch Two Sample t-test names the test.
- t = 2.7299: the difference in means is about 2.7 standard errors. df = 170.1 is the degrees of freedom, a number linked to the sample size that sets the exact shape of the t distribution used to get the p-value. Welch's version adjusts it for unequal spreads, which is why it is not a whole number.
- p-value = 0.007003: if smoking made no difference, a gap this large or larger, in either direction, would turn up about 7 times in 1,000.
- alternative hypothesis: the test was two-sided, looking for a difference in either direction.
- 95 percent confidence interval: 78.6 to 489.0 g, for the non-smoker mean minus the smoker mean.
- sample estimates: the two group means, 3,055.7 g and 2,771.9 g.
Worked example · reading the result
95% confidence interval: 79 to 489 g
p = 0.007: a difference this large would be rare if smoking made no difference
Report the effect and its interval first, then the p-value: “babies of smokers were 284 g lighter (95% CI 79 to 489 g; Welch t-test, p = 0.007)”. The wide interval shows the size of the effect is uncertain.
Step by step. The difference is 3,055.696 − 2,771.919 = 283.777 g, which rounds to 284 g. Its standard error is the difference divided by t: 283.777 ÷ 2.7299 ≈ 104 g. With 170 degrees of freedom, a 95% interval reaches about 1.97 standard errors either side: 1.97 × 104 ≈ 205 g. So the interval is 284 − 205 = 79 g to 284 + 205 = 489 g, which matches R's output.
What the answer means. A 95% confidence interval is a range of values for the true difference that are reasonably compatible with the data. Strictly, if the study were repeated many times and an interval calculated each time, about 95% of those intervals would contain the true difference. Here the data are compatible with a modest effect of about 80 g and with a large one of nearly 500 g. Because the interval does not include 0, the p-value is below 0.05; the two always agree in this way for the same test.
R uses Welch's t-test by default, which does not assume equal variances: a safe choice. If the data are skewed or the groups small, a rank-based test makes fewer assumptions:
The classic version, Student's t-test, assumes both groups have the same spread. Here the SDs were 753 g and 660 g (Describing data), not identical, and Welch's version allows for that by adjusting the standard error and degrees of freedom. When the spreads really are equal it gives almost the same answer, so there is little to lose by using it.
A rank-based test first replaces every value with its rank: the lightest of the 189 babies gets rank 1, the heaviest rank 189. It then asks whether one group's ranks are systematically higher. Because only the order matters, one extreme value cannot dominate the result. This test has two names, the Wilcoxon rank sum test and the Mann–Whitney U test.
The price of using ranks is small. When the data really are normal, a rank-based test has slightly less power than a t-test, so it misses a real difference a little more often; when the data have outliers or long tails, it can have more. Be careful with how you describe the result: the Wilcoxon test asks whether values in one group tend to be larger than in the other. It is a test of medians only if both groups have the same shape and spread, and it gives no confidence interval for a difference in means.
wilcox.test(bwt ~ smoke, data = bw)
Reading the output. W = 5249.5 is the test statistic. In R it counts, across every possible pair of one non-smoker's baby and one smoker's baby, how many pairs have the non-smoker's baby heavier, with ties counted as a half. There are 115 × 74 = 8,510 such pairs, and 5,249.5 ÷ 8,510 = 0.62, so in about 62% of pairs the non-smoker's baby was heavier; with no difference you would expect about 50%. The continuity correction is a small adjustment R makes when it calculates the p-value from an approximation instead of exactly. The "location shift" is the hypothesis that one group's whole distribution sits higher or lower than the other's. The p-value, 0.0068, agrees closely with the t-test, so the conclusion does not depend on the normality assumption.
Agreement between the two tests is reassuring, and is the usual result when groups are reasonably large and the data roughly symmetric. If they disagreed sharply, that would be a sign that the shape of the data, such as a few extreme values, is driving one of them. The answer then is to look at the data and decide which question matters, not to report whichever test gave the smaller p-value.
Go deeper: more than two groups: ANOVA and Tukey's test
A t-test compares two groups. With three or more, running every pairwise t-test raises the chance of a false positive, the multiple testing problem of Multiple testing and power. One-way ANOVA (analysis of variance) first asks a single question: could all the group means be equal? For birth weight by recorded race, fit <- aov(bwt ~ race, data = bw) and summary(fit) give F = 4.91 on 2 and 186 degrees of freedom, p = 0.008.
That says the means differ somewhere, not where. TukeyHSD(fit), Tukey's post-hoc test, then compares every pair while keeping the overall false-positive rate at 5%. Babies of mothers recorded as black (mean 2,720 g) or other (2,805 g) were lighter than those of mothers recorded as white (3,103 g), with adjusted p = 0.043 and 0.026; black and other did not differ (p = 0.86). Classic ANOVA assumes equal spreads, like Student's t-test; oneway.test() is the Welch-style version.
Paired samples
R's built-in sleep data are from a classic 1908 study: 10 patients each took two sleeping drugs, and extra is the extra hours of sleep. Because every patient took both, each patient is their own control.
Two measurements are paired when they come from the same unit, such as the same person before and after treatment, the same donor's treated and untreated cells, or a tumour and normal tissue from the same patient. Paired values are not independent: a person who sleeps well on one drug tends to sleep well on the other. A paired test uses that link instead of ignoring it.
Pairing changes the question. Instead of comparing two groups of people, you compute one difference per person and ask whether the average difference is zero. Everything that makes one patient sleep more than another, such as age or habits, appears in both of their measurements and cancels out in the difference. What is left is the effect of switching drugs, plus a smaller amount of noise.
sleep_wide <- data.frame(drug1 = sleep$extra[sleep$group == 1],
drug2 = sleep$extra[sleep$group == 2])
t.test(sleep_wide$drug2, sleep_wide$drug1, paired = TRUE)
t.test(sleep_wide$drug2, sleep_wide$drug1)$p.value # wrongly unpairedWhat this does, line by line
- The
sleepdata have 20 rows: 10 patients under drug 1, then the same 10 under drug 2, in the same patient order.sleep$extra[sleep$group == 1]takes the 10 drug 1 values; the same with== 2takes the drug 2 values.data.frame()puts them side by side, one row per patient. t.test(sleep_wide$drug2, sleep_wide$drug1, paired = TRUE)works out each patient's difference, drug 2 minus drug 1, and tests whether the average difference is zero. This is the same as a one-sample t-test on the ten differences.- The last line runs the same comparison without
paired = TRUE, treating the two columns as unrelated groups, and$p.valuetakes only the p-value from the result.
Reading the output. t = 4.0621 on df = 9: with 10 pairs there are 10 − 1 = 9 degrees of freedom. The p-value is 0.0028. The 95% confidence interval for the mean difference runs from 0.70 to 2.46 hours, and the estimated mean difference is 1.58 hours. The final line, 0.079, is the p-value from the wrong, unpaired test.
The paired t-test makes one assumption about shape: that the differences, not the raw values, are roughly normal. With only ten pairs that is worth checking with a quick plot of the ten differences. If they are skewed, or include an extreme value, wilcox.test() with paired = TRUE runs the rank-based alternative, the Wilcoxon signed-rank test.
A practical pitfall: the code above pairs values by their position, which works because the sleep rows list the same ten patients in the same order for both drugs. The data also have an ID column, and in your own data you should always pair by an identifier, for example by reshaping with pivot_wider() from Cleaning data with dplyr and tidyr. If the rows were sorted differently, pairing by position would compare one patient's drug 1 value with another patient's drug 2 value, and R would give no warning.
sleep data.Drug 2 gave 1.58 more hours on average (95% CI 0.70 to 2.46; p = 0.003). Ignoring the pairing gives p = 0.08 from the same numbers: patients differ so much from each other that the drug effect is hidden unless each is compared with themselves. The same logic is behind the paired design in RNA-seq.
The arithmetic shows why. The ten differences vary much less than the patients do: their standard deviation is about 1.2 hours, while the SD of the patients on each drug is about 1.8 to 2.0 hours. The standard error of the mean difference is 1.58 ÷ 4.0621 ≈ 0.39 hours when the pairing is used, and roughly twice as large, when it is ignored. The same difference of 1.58 hours is about 4 standard errors from zero in the paired test and fewer than 2 in the unpaired one.
In the airway experiment from Module 1, each of the four donors gave one treated and one untreated sample. Comparing within donors removes the large differences between people, as here, so the drug's effect on each gene is easier to detect.
Key point
Choose the test from the design, before you look at the results: independent or paired, and how many groups. Trying tests until one gives p < 0.05 makes every p-value meaningless.
For example, suppose you run a t-test, get p = 0.08, then try a Wilcoxon test, then remove an "outlier", then test only the women, and stop when p falls below 0.05. Each attempt is another chance for noise to look like an effect, so the final p-value no longer means what it claims. Writing down the planned test, and why, before seeing the outcome prevents this.
Further reading · Module 2Greenland et al. 2016: 25 common misreadings of p-values and confidence intervals. · All reading for this course
Check your understanding
1. p = 0.007 means:
2. Blood pressure is measured in the same 20 patients before and after a drug. Which test?
3. A 95% CI for a difference runs from −10 to 40. What does that tell you?
Up next
Lesson 3
Counts and proportions