LSMA

LONDON SCHOOL OF MULTIOMICS & AI

Free course

Free course

0 of 25 lessons

Lesson 5 of 7

Module 2 · Statistics

Multiple testing and power

Multiple testing and power

Multiple testing and power

Genomics tests thousands of things at once. This lesson simulates exactly that, so you can see how many false discoveries a 0.05 threshold produces, how to control them, and how many samples a study needs.

⏱ 13 min readModule 2 · StatisticsFree
Lesson at a glance4 steps · about 13 min
  1. 10,000 tests with nothing to find
  2. What this does, line by line
  3. Power and sample size
  4. What this does

10,000 tests with nothing to find

Simulate 10,000 “genes” measured in two groups of 10, with no real difference, and test each one. Then add 500 genes with a real change of 2 standard deviations.

Recall what p < 0.05 promises: when there is truly no difference, a test gives p below 0.05 only 5% of the time. A false positive is exactly that case, a "significant" result where nothing real is going on. One test carries a 5% risk. Ten thousand tests each carry it, so false positives pile up by the hundred. This is the multiple testing problem, and RNA-seq meets it every time, because it tests about 20,000 genes at once.

A simulation is ideal for seeing this, because you decide the truth in advance. You know which "genes" changed and which did not, so you can count exactly how many hits are real, which is never possible with real data.

The problem is not confined to genomics. Each test in the earlier lessons of this module carried its own 5% risk, and a small study that compares 20 outcomes, or 20 subgroups, faces the same arithmetic. If the tests are independent and none of the null hypotheses is false, the chance of at least one false positive is 1 − 0.95 multiplied by itself once per test: 1 − 0.9510 ≈ 0.40 for 10 tests, and 1 − 0.9520 ≈ 0.64 for 20. This chance of at least one false positive across a set of tests is called the family-wise error rate. It is why the ANOVA box in Comparing two groups used Tukey's test rather than every pairwise t-test, and why trying test after test, as its key point warned, undermines the final p-value.

R
set.seed(1)
p_null <- replicate(10000, t.test(rnorm(10), rnorm(10))$p.value)   # no real change
sum(p_null < 0.05)

set.seed(2)
p_real <- replicate(500, t.test(rnorm(10), rnorm(10, mean = 2))$p.value)   # real change
p_all <- c(p_null, p_real)
truth <- factor(rep(c("no change", "real change"), c(10000, 500)))

table(truth, raw = p_all < 0.05)
table(truth, BH = p.adjust(p_all, method = "BH") < 0.05)
table(truth, Bonferroni = p.adjust(p_all, method = "bonferroni") < 0.05)

What this does, line by line

  1. set.seed(1) fixes the random numbers so the results can be repeated exactly.
  2. replicate(10000, …) runs the expression in brackets 10,000 times and collects the answers. Each time, rnorm(10) draws two groups of 10 values from the same normal distribution (mean 0, SD 1), t.test() compares them, and $p.value keeps only the p-value. The 10,000 p-values are stored as p_null.
  3. sum(p_null < 0.05) counts how many of these no-difference tests came out "significant".
  4. After set.seed(2), the second replicate() does the same 500 times, but the second group has mean = 2: shifted by 2, which is 2 standard deviations, since the SD is 1.
  5. c(p_null, p_real) joins all 10,500 p-values, and truth labels the first 10,000 "no change" and the last 500 "real change".
  6. Each table() crosses the truth with a decision: hit (TRUE) or not (FALSE). The first uses raw p-values. The second and third first pass the p-values through p.adjust(), which applies a correction for the number of tests.
[1] 506 raw truth FALSE TRUE no change 9494 506 real change 3 497 BH truth FALSE TRUE no change 9987 13 real change 113 387 Bonferroni truth FALSE TRUE no change 10000 0 real change 467 33

Reading the output. The first line, 506, is the number of false positives among the 10,000 null tests: 5.06%, almost exactly the 5% promised. In each table the rows are the truth and the columns the decision; the TRUE column holds the hits. With raw p-values, 506 no-change genes and 497 real-change genes are hits. With Benjamini–Hochberg, 13 and 387. With Bonferroni, 0 and 33. The FALSE entries in the "real change" row are real changes the method missed: 3, 113 and 467.

Why 5% exactly? When there is no real difference, p-values are spread evenly between 0 and 1: any value is as likely as any other. So about 5% of them land below 0.05, about 1% below 0.01, and so on, however many tests you run.

The corrections answer two different questions. Bonferroni controls the family-wise error rate: the chance of even one false positive among all the tests stays at or below 5%. Benjamini–Hochberg controls the false discovery rate (FDR): on average, no more than 5% of the hits are false. The first suits a few tests where any false claim would be costly, such as the main outcomes of a clinical trial. The second suits screening thousands of genes, where you expect to follow up a list and can accept that a few entries are wrong.

An adjusted p-value from p.adjust() should be read against the same 0.05 cut-off, but its meaning changes. A BH-adjusted value of 0.03 says that if you call this gene and every gene with a smaller p-value a hit, the expected share of false hits in that list is at most 3%. Note too that p.adjust() uses method = "holm" unless you say otherwise. Holm's method controls the same family-wise error rate as Bonferroni but never finds fewer hits, so it is the better choice when you want that kind of control.

A common pitfall is to correct only part of the tests. The number that matters is every test you ran, including those that are not reported; correcting only the "interesting" ones, or choosing which genes to test after seeing the results, brings the false positives back.

Multiple testing and power · LSMA free course Bars of hits, split into real changes and false discoveries. Raw p below 0.05: 1,003 hits, 497 real and 506 false. Benjamini–Hochberg: 400 hits, 387 real and 13 false. Bonferroni: 33 hits, all real. HITS AMONG 10,500 TESTS real change false discovery Raw p < 0.05 Benjamini–Hochberg Bonferroni 1,003 hits 400 hits 33 hits 497 real · 506 false · 3 real changes missed 387 real · 13 false · 113 missed 33 real · 0 false · 467 missed
Bar length is the number of hits, drawn to scale. Without correction, half the hits are false; Benjamini–Hochberg removes almost all the false ones while keeping most real changes; Bonferroni removes every false hit but also most of the real ones.

Worked example · three thresholds, one dataset

raw p < 0.05: 1,003 hits, of which 506 are false (50%)
Benjamini–Hochberg: 400 hits, of which 13 are false (3.3%, under the 5% target)
Bonferroni: 33 hits, none false, but 467 real changes missed

With no correction, half the “discoveries” are noise. Bonferroni controls the chance of any false positive and is very strict. Benjamini–Hochberg controls the false discovery rate, the share of hits that are false, which is why genomics uses it.

Step by step. Raw: 506 + 497 = 1,003 hits, and 506 ÷ 1,003 = 0.504, so 50% are false. Benjamini–Hochberg: 13 + 387 = 400 hits, and 13 ÷ 400 = 0.0325, or 3.3%. Bonferroni: 0 + 33 = 33 hits, none false, but 500 − 33 = 467 of the 500 real changes are missed.

How the two corrections work. Bonferroni multiplies each p-value by the number of tests, here 10,500 (capping the result at 1). That is the same as demanding raw p < 0.05 ÷ 10,500, about 0.0000048. Benjamini–Hochberg sorts the p-values from smallest to largest and compares each with a threshold that grows with its rank: the smallest must beat 0.05 × 1/10,500, the second 0.05 × 2/10,500, and so on. Every p-value up to the last one that passes is a hit. Its adjusted p-values are what many genomics tools report as padj.

What the answer means. A false discovery rate of 5% is a promise about the average share of false hits over many experiments, not a guarantee for each one; this run happened to give 3.3%. If you follow up 400 genes from a BH list at 0.05, expect a handful, not none, to be false.

Go deeper: read the p-value histogram first

Before trusting any correction, draw a histogram of all the raw p-values: hist(p_all, breaks = 20). Null p-values are spread evenly between 0 and 1, so they form a flat floor. Real effects pile up near 0 and make a spike in the first bar. In the simulation each bar of the floor holds about 500 tests (10,000 ÷ 20), and the first bar rises to about 1,000.

The shape is a diagnostic. A flat floor plus a spike is the healthy pattern. No spike suggests few or no real changes. A bump in the middle, or a pile near 1, usually means the test's assumptions do not fit the data, for example counts analysed with a test built for normal values, and then neither raw nor adjusted p-values can be trusted. The height of the floor also gives a rough estimate of the share of tests that are truly null, an idea used by the q-value method.

Power and sample size

Power is the chance a study detects a real effect of a given size. It depends on the effect, the variability and the number of samples.

Two kinds of error frame this. A type I error is a false positive, finding an effect that is not there; the 0.05 threshold sets its rate. A type II error is a false negative, missing an effect that is there. Power is the chance of avoiding a type II error, so a study with 80% power misses a real effect of the planned size one time in five. Larger effects, less variable measurements and more samples all raise power.

Effect and variability usually enter together, as the standardised effect size: the difference divided by the SD. A difference of 2 with an SD of 1, as in the simulation, is an effect size of 2. In Describing data the smokers' gap of 284 g against an SD of about 700 g was an effect size of about 0.4. By convention studies aim for 80% or 90% power at the 0.05 level, but these are habits, like the 0.05 cut-off itself.

R
power.t.test(n = 10, delta = 2, sd = 1)$power      # the simulation above

# how many babies per group to detect a 300 g difference, SD 700 g, with 80% power?
power.t.test(delta = 300, sd = 700, power = 0.8)

What this does

  • power.t.test() links five quantities: the number per group (n), the true difference (delta), the standard deviation (sd), the significance level (0.05 unless you change it) and the power. Give it all but one, and it calculates the missing one.
  • The first call gives n, delta and sd, so it returns the power. It matches the simulation: 10 per group, a difference of 2, SD 1. $power keeps only that number.
  • The second call gives delta, sd and power, so it solves for the sample size: how many babies per group to have an 80% chance of detecting a true difference of 300 g when the SD is 700 g.
[1] 0.988179 Two-sample t test power calculation n = 86.43651 delta = 300 sd = 700 sig.level = 0.05 power = 0.8 alternative = two.sided NOTE: n is number in *each* group

Reading the output. 0.988 means a 98.8% chance of p < 0.05 for each real change in the simulation. Of 500 real changes you would therefore expect about 0.988 × 500 = 494 raw hits; the simulation gave 497, close to that. In the second result, n = 86.43651 is the sample size per group; delta, sd, sig.level and power repeat the inputs; two.sided means the test looks for a difference in either direction. You cannot recruit part of a baby, so always round up: 87 per group.

About 87 per group, so 174 in total: close to the size of the birth weight study. Small studies have low power, and the effects they do find tend to be exaggerated, because only the lucky large estimates cross the threshold.

Why is the simulation's power so high and the birth weight study's need so large? In the simulation the effect was 2 standard deviations; here it is 300 ÷ 700 = 0.43 standard deviations, almost five times smaller. Needed sample size grows roughly with 1 ÷ (effect size)², so a small effect needs many more samples: halving the effect you want to detect roughly quadruples the sample size.

The exaggeration of significant effects in small studies is sometimes called the winner's curse. If a study has only 20% power, the estimates that happen to reach p < 0.05 are mostly the ones that chance pushed upwards, so the published effect overstates the truth, and a larger follow-up study usually finds a smaller one.

A power calculation is only as good as its inputs, and both the effect and the SD are guesses before the study starts. The SD of 700 g here came from the birth weight data themselves; in practice it would come from a pilot study or published work. Because the answer depends strongly on the guess, try a range. With an SD of 600 g the same 300 g difference needs about 64 babies per group; with 800 g, about 113.

Multiple testing and power interact. The simulation's 98.8% power applied at the raw 0.05 threshold, and with it 497 of the 500 real changes were found. Under Bonferroni's threshold of about 0.0000048, only 33 were: the effective power fell to 33 ÷ 500 = 6.6%. A study that will test thousands of genes needs its power calculated at the stricter threshold it will actually use, which is one reason RNA-seq experiments need more replicates than intuition suggests.

One more pitfall: "observed power", calculated after a study from its own estimated effect, adds nothing. It is fixed by the p-value, so a non-significant result always has low observed power. To judge what a finished study could detect, look at its confidence interval instead.

The false discovery rate and the step-up procedure in this lesson come from Benjamini & Hochberg 1995, "Controlling the false discovery rate" (Journal of the Royal Statistical Society, Series B). The method assumes the tests are independent or positively related; for arbitrary dependence, Benjamini & Yekutieli 2001 (Annals of Statistics) give a stricter version, available as p.adjust(method = "BY"). The link between low power and exaggerated effects is reviewed in Button et al. 2013, "Power failure" (Nature Reviews Neuroscience), which estimated the median power of neuroscience studies at about 21%. That figure is specific to the field and period studied, so cite it as an example, not as a general rate.

Key point

Decide the sample size before the study, from the smallest effect that would matter. A non-significant result from an underpowered study is “we could not tell”, not “there is no effect”.

For example, suppose a pilot of 10 babies per group finds smokers' babies 300 g lighter, p = 0.35. With an SD of 700 g, that study had far too little power to detect 300 g reliably; the calculation above says it needed about 87 per group. The honest conclusion is that the pilot could not tell, and its confidence interval will show it: wide enough to include both no effect and a large one.

Further reading · Module 2Greenland et al. 2016: 25 common misreadings of p-values and confidence intervals. · All reading for this course

Check your understanding

1. You test 20,000 genes with no real differences at p < 0.05. About how many will look significant?

2. What does Benjamini–Hochberg at 0.05 control?

3. A small pilot study finds no significant effect. What should you conclude?

Up next

Lesson 6

A reproducible workflow

Ask BioTutor

Create a free website with Framer, the website builder loved by startups, designers and agencies.