Bias and Variability Are Different Problems

You’ll be able to

Distinguish bias from sampling variability
Explain why increasing sample size reduces variability but not bias
Identify which design flaws produce which problem

Two ways an estimate can be wrong

A sampling method has bias if it systematically produces estimates that miss the true parameter in the same direction. It has variability if repeated samples give noticeably different answers. The two are independent: a method can be unbiased and imprecise, biased and precise, both, or neither. The target metaphor is standard and worth keeping — bias is shots consistently off-center, variability is shots scattered widely, and only a method with neither is trustworthy.

Sample size fixes one and not the other

This is the single most examinable point in the unit. Increasing n reduces variability — the standard deviation of a sampling distribution carries √n in the denominator, so estimates from larger samples cluster more tightly. Increasing n does nothing to bias. A voluntary-response poll of a million people is a precisely measured wrong answer; the systematic tilt in who chooses to respond is present in every one of those responses and does not average out. The famous 1936 Literary Digest poll surveyed over two million people and got the election wrong.

Which flaw causes which problem

Bias comes from the design: voluntary response, convenience sampling, undercoverage of part of the population, nonresponse that differs systematically from response, and question wording that leads. Variability comes from the size of the sample relative to the variation in the population. Note what does not control variability: the size of the population. A sample of 1,000 is about as precise for a country of 300 million as for a town of 30,000, which surprises people and is regularly tested.

On the exam

When a question asks whether a larger sample would fix a described problem, identify the problem first. If it is bias, the answer is no, and you must say why — the systematic tilt is present in every observation, so collecting more of them collects more of the same tilt.

Worked example

A radio station invites listeners to call in and vote on a proposal. After 40,000 calls, 78% support it. Identify the problem, state whether it is bias or variability, and explain whether a larger response would help.

1.The sample is self-selected: listeners chose to call rather than being chosen by the researcher.
2.People with strong opinions, particularly strong objections or strong enthusiasm, are far more likely to make the effort to call, and the station's listeners are not representative of the population to begin with.
3.This is voluntary-response bias — a systematic tilt, not random noise.
4.A larger number of calls reduces variability but leaves the tilt untouched, so it would produce a more precise estimate of the wrong quantity.

Answer: This is voluntary-response bias. The 78% estimates the opinion of people motivated enough to call a particular radio station, not the population's opinion. Increasing the number of calls narrows the variability but cannot remove the systematic tilt, so a larger response would not help — only a randomly selected sample would.

Checkpoint

A survey suffers from serious undercoverage. Quadrupling the sample size using the same method would —

Checkpoint

The precision of a sample proportion depends primarily on —

Answer the 2 checkpoints as you read.