Bias in Data Collection
- Identify undercoverage, nonresponse, and response bias
- Explain how question wording can bias survey results
- Distinguish bias from sampling variability
Bias tilts every estimate the same way
Bias is a systematic tendency to over- or under-estimate the truth — the error does not average out as the sample grows. This is different from ordinary sampling variability, the random wiggle from one sample to the next that does shrink with larger samples. A biased method with a huge sample is still wrong; more data just gives a more precise wrong answer.
Three common sampling biases
Undercoverage occurs when some groups in the population are left out of the sampling process (a phone survey misses people without phones). Nonresponse bias occurs when selected individuals cannot be reached or refuse to answer, and they differ from those who do respond. Response bias occurs when people give inaccurate answers — because of a sensitive topic, a lie, or an interviewer’s influence.
Wording effects
The way a question is phrased can push respondents toward an answer. Leading or emotionally loaded wording ("Don’t you agree that hardworking families deserve tax relief?") inflates agreement. Confusing double negatives and questions that suggest a socially "correct" answer also distort results. Neutral, clear wording is essential.
Increasing the sample size reduces variability, never bias. If the method systematically misses or distorts, a larger sample just locks in the same error more confidently. Fixing bias requires fixing the method — random selection, high response rates, neutral wording.
A city mails 5,000 surveys about a proposed tax; only 400 are returned, and returners are overwhelmingly homeowners angry about the tax. Identify the type of bias and its likely direction.
- 1.Note that most selected people did not respond — only 400 of 5,000 (8%) returned the survey.
- 2.The people who did respond differ systematically from those who did not (angry homeowners, motivated to reply).
- 3.Selected individuals failing to respond, in a way that skews results, is nonresponse bias.
- 4.Because responders oppose the tax more than the general population, the survey will overstate opposition.
A telephone survey uses only landline numbers, so households with only cell phones have no chance of being selected. This is an example of:
When asked to identify bias, name the type (undercoverage, nonresponse, response, or wording) and state its likely direction — will the estimate be too high or too low? AP rubrics reward explaining how the flaw pushes the result.
A survey asks, "Do you support wasteful government spending on failed programs?" Most say no. What is the main problem?
Answer the 2 checkpoints as you read.
Sign in to save your progress