Sampling Distributions and Standard Error

The idea in one sentence

Take a different sample and you get a different mean, so the sample mean is itself a random quantity with its own distribution — and knowing how spread out that distribution is turns one sample into a statement about the whole population.

Three distributions, easily confused

Keeping these apart is most of the topic.

The population distribution is how the individual values are spread. Adult heights, say — possibly skewed, possibly not.

The sample distribution is the histogram of the 100 heights you actually measured. It looks roughly like the population, and it looks more like it the bigger the sample.

The sampling distribution is different in kind. Imagine taking a sample of 100, recording just its mean, then doing that again, and again, thousands of times. The histogram of all those means is the sampling distribution of \(\bar{x}\).

The first two describe individuals. The third describes a statistic. You never actually build it — you take one sample — but its shape is what licenses every conclusion you draw.

What the sampling distribution looks like

Three facts, and they are the reason inference works at all.

Its centre is the population mean. \(\mu_{\bar x} = \mu\). The sample mean is neither too high nor too low on average, which makes it an unbiased estimator.

Its spread is smaller, by a factor of \(\sqrt n\):

This is the standard error. With \(\sigma = 20\) and \(n = 100\) it is \(\frac{20}{10} = 2\) — individual heights vary by 20, but averages of 100 vary by only 2. Averaging cancels the highs against the lows.

Its shape is approximately normal once \(n\) is reasonably large, whatever the population looked like. That is the central limit theorem, and it is the surprising part: skewed populations, bimodal populations, even a coin flip’s two-spike distribution all produce bell-shaped sampling distributions for the mean.

The square root is the whole economics of sampling

\(\sqrt n\) in the denominator means precision improves, but slowly.

To halve the standard error you need four times the data. To get ten times the precision, a hundred times the data. There is no sample size at which the next hundred responses are as valuable as the first hundred were.

This is why national polls stop at around 1000 people. Going to 4000 would only halve the margin of error, at four times the cost, and other sources of error — bias in who responds — would dominate long before the arithmetic did.

It is also why the population size is almost absent from the formula. Sampling 1000 people works about as well for a city of 100,000 as for a country of 300 million. What matters is how many you asked, not what fraction of the whole they are — which is deeply counterintuitive and completely correct.

Where inference comes from

Everything downstream is this one idea applied.

If \(\bar x\) is normal with centre \(\mu\) and spread \(\frac{\sigma}{\sqrt n}\), then about 95% of samples give a mean within two standard errors of \(\mu\). Turn that around and you have a confidence interval: \(\bar x \pm 2\,\text{SE}\) captures \(\mu\) about 95% of the time.

Or assume a value for \(\mu\), see how many standard errors your \(\bar x\) sits away from it, and you have a p-value.

The margin of error is not the standard error — it is the standard error multiplied by a critical value for whatever confidence level you chose.

Common mix-ups

• Confusing standard error with standard deviation. SD describes individuals; SE describes a statistic, and SE is always the smaller.
• Dividing by \(n\) instead of \(\sqrt n\). That would make large samples implausibly precise.
• Thinking the sample must be large for the population to be normal. The CLT is about the sampling distribution, and it applies precisely because the population may not be normal.
• Worrying about the population size. It barely enters, provided the sample is a small fraction of it.
• Believing a larger sample fixes bias. It does not. A badly drawn sample of a million is still badly drawn, just more precisely wrong.

Check yourself

What is the standard deviation of the sampling distribution of \(\bar x\) called, what is it when \(\sigma = 20\) and \(n = 100\), and what must happen to \(n\) to halve it?

It is the standard error. \(\frac{20}{\sqrt{100}} = 2\). And since the \(\sqrt n\) is on the bottom, halving it needs \(n\) multiplied by 4.