MISSION · Data and probability
Samples, Populations, and Bias
How a survey goes wrong before a single number is calculated
LEVEL 1 · EXPLORE
Make the idea move
Change the controls and watch what happens. Follow your curiosity and see what you notice.
Loading the activity…
LEVEL 2 · LEARN
Unlock the explanation
Open the chapters in any order. Try the examples as you go.
The idea in one sentence
You can learn about a whole population from a small sample, but only if every member had a fair chance of being in it.
Why sampling works at all
Asking everyone is usually impossible — too slow, too expensive, and for destructive tests (how many drops before this phone breaks?) self-defeating. So you ask a few and generalise.
The surprising part is how few. A well-drawn sample of 1,000 people estimates a national opinion to within about 3 percentage points, whether the country has 5 million people or 300 million. Population size barely matters; sample quality is everything. A random 1,000 beats a careless 100,000, and that single fact is the most counterintuitive thing in the subject.
Bias is a property of the method
A sample is biased when the way it was collected systematically favours some part of the population. Bias isn’t bad luck — it’s a fault in the procedure, and collecting more data the same way makes it worse rather than better, because you get a more precise estimate of the wrong number.
The classic failures:
- Convenience sampling — asking whoever is nearby. A survey about public transport conducted at a bus stop misses everyone who drives.
- Self-selection — online polls and phone-ins. People with strong opinions answer; the indifferent majority doesn’t.
- Non-response — those who decline may differ systematically from those who reply. A 10% response rate makes a sample of 10,000 less trustworthy than a complete one of 500.
- Leading questions — “Do you support the government’s reckless plan?” measures the wording, not the opinion.
- Undercoverage — the sampling frame misses part of the population. Landline-only polling now skips most people under forty.
The 1936 Literary Digest poll collected 2.4 million responses and confidently predicted the wrong US president. Its list came from car registrations and telephone directories, which in 1936 meant wealthier households. Two million biased responses were worth less than a few thousand fair ones.
Sampling methods
- Simple random — everyone equally likely. The standard against which others are judged.
- Systematic — every \(k\)th item on a list. Easy, and fine unless the list has a repeating pattern.
- Stratified — split the population into groups and sample each in proportion. Better precision when the groups genuinely differ.
- Cluster — sample whole groups, like a few entire schools. Cheaper, less precise.
All of them rely on randomness somewhere. The moment a human chooses who to include, unconscious preferences enter.
Questions to ask about any statistic
Who was asked? How were they chosen? How many declined? What exactly was the question? Who paid for the study?
Most misleading statistics fail at one of those five, and none of them are about the arithmetic.
Common mix-ups
- Believing a big sample fixes bias. It doesn’t; it entrenches it.
- Calling a convenience sample random. Random means a defined chance for everyone.
- Ignoring non-response. Who didn’t answer is part of the result.
- Generalising past the population sampled. A survey of one university’s students says nothing about the country.
Check yourself
A radio station asks listeners to phone in about a local policy, and 70% of callers oppose it. What’s wrong with concluding that most residents oppose it?
Callers are self-selected, and people who feel strongly are far likelier to ring in. The sample also only covers listeners of that station. It measures the opinion of motivated callers, not of residents.
LEVEL 3 · REFLECT
Add this quest to your trail
When you feel ready, mark this idea as explored. You can always come back and try it again.
EXPLORER STRATEGIES