MISSION · Data and probability
Hypothesis Testing
What a p-value says, and the four things it doesn't
LEVEL 1 · EXPLORE
Make the idea move
Change the controls and watch what happens. Follow your curiosity and see what you notice.
Loading the activity…
LEVEL 2 · LEARN
Unlock the explanation
Open the chapters in any order. Try the examples as you go.
The idea in one sentence
Assume nothing interesting is happening, work out how surprising your data would be under that assumption, and reject it only if the answer is “very”.
The two hypotheses
\(H_0\), the null hypothesis — the boring explanation. The coin is fair. The drug does nothing. The two groups are the same.
\(H_1\), the alternative — what you suspect instead.
The test is deliberately lopsided. You assume \(H_0\) and look for evidence strong enough to abandon it, in the same way a court assumes innocence. This asymmetry means that failing to reject \(H_0\) is not proof that \(H_0\) is true — it means the evidence wasn’t strong enough, which can happen because there’s no effect or because the sample was too small to see one.
The p-value
\[p = P(\text{data this extreme or more} \mid H_0 \text{ true})\]Flip a coin 100 times and get 60 heads. If the coin is fair, the mean is 50 with \(\sigma = 5\), so 60 sits two standard deviations out. The chance of landing that far from the middle, in either direction, is about 0.046.
That’s the p-value: not the probability that the coin is fair, but the probability of data like this if it were.
The threshold, and what it costs
Convention sets \(\alpha = 0.05\). Below that, reject \(H_0\) and call the result significant.
The 0.05 is arbitrary — a convenient line Fisher suggested, not a law of nature. And it has a direct cost: if \(H_0\) is true, you will wrongly reject it 5% of the time. That’s a Type I error, a false alarm. Its opposite, missing a real effect, is a Type II error, and tightening the threshold to avoid one increases the other. Every test picks a balance between crying wolf and sleeping through the wolf.
Our coin lands at \(p = 0.046\) — significant, by a whisker. And it is worth knowing how thin that whisker is. The 0.046 comes from the normal approximation; counting the coin outcomes exactly gives 0.057, on the other side of the line. Same data, same question, two standard methods, opposite verdicts. A result sitting that close to the threshold is telling you the threshold is doing the deciding, not the evidence.
What a p-value is not
Four claims it cannot support, all common:
- Not the probability that \(H_0\) is true.
- Not the probability the result was luck.
- Not a measure of effect size. With 100,000 participants a trivial difference will be highly significant. “Significant” means detectable, not important.
- Not a licence to stop thinking. A p-value from a biased sample is worthless however small it is.
p-hacking
If twenty independent tests are run on data where nothing is going on, one will come out significant at 0.05 on average. That’s not misconduct, it’s arithmetic.
It becomes misconduct when only the significant one is published. Testing many outcomes, trying subgroups until one works, or stopping data collection once the p-value dips — all produce significant-looking results from noise. The defence is pre-registering what you will test before you look, and reporting everything you tried.
Common mix-ups
- Reading \(p\) as “chance the null is true”. It’s conditional the other way.
- Treating “not significant” as “no effect”. Absence of evidence isn’t evidence of absence.
- Confusing statistical and practical significance.
- Running tests until one works. That manufactures the result.
Check yourself
A drug trial gives \(p = 0.03\). What may you conclude, and what may you not?
You may say the data would be unlikely if the drug had no effect, so at the 5% level the null is rejected. You may not say there’s a 97% chance the drug works, nor that the effect is large enough to matter.
LEVEL 3 · REFLECT
Add this quest to your trail
When you feel ready, mark this idea as explored. You can always come back and try it again.
EXPLORER STRATEGIES