Errors, Significance & Power

You’ll be able to

Distinguish Type I and Type II errors and their consequences
Relate the significance level α to the probability of a Type I error
Identify the factors that increase the power of a test

Two ways to be wrong

A test can err in two ways. A Type I error rejects a true H₀ — a "false alarm," concluding there is an effect when there is none. A Type II error fails to reject a false H₀ — a "miss," failing to detect a real effect. Which is worse depends on context: a false-positive medical test versus a missed diagnosis carry very different costs.

α, β, and power

The significance level α is the probability of a Type I error — reject a true null. β is the probability of a Type II error. The power of a test is 1 − β, the probability of correctly rejecting a false H₀ (detecting a real effect). There is a tradeoff: lowering α (fewer false alarms) raises β (more misses) if nothing else changes.

What increases power

Power rises when the effect is easier to detect. It increases with a larger sample size, a larger true effect (further from H₀), less variability, and a larger α. The single lever you most control is sample size: more data sharpens the sampling distribution and boosts your chance of catching a real difference.

Errors and power at a glance

P(Type I) = α · P(Type II) = β · power = 1 − β

Type I = reject a TRUE null (false positive). Type II = fail to reject a FALSE null (false negative). Increasing n raises power without raising α.

Worked example

A drug is tested with H₀: the drug is no better than placebo. Describe the Type I and Type II errors and a real-world consequence of each.

1.Type I error: reject H₀ when it is true — conclude the drug works when it actually does not.
2.Consequence of Type I: an ineffective drug is approved and marketed, exposing patients to cost and side effects with no benefit.
3.Type II error: fail to reject H₀ when it is false — conclude the drug does not work when it actually does.
4.Consequence of Type II: a genuinely helpful drug is abandoned, so patients miss out on an effective treatment.

Answer: Type I error: approving a drug that does not actually work (false positive), exposing patients to a useless treatment. Type II error: rejecting a drug that actually works (false negative), denying patients an effective treatment.

Checkpoint

A researcher fails to reject H₀, but in reality H₀ is false. What type of error is this?

On the exam

Always describe errors in context. On free response, spell out what rejecting or failing to reject means for the real situation, and state a consequence. "Type I error: concluding the water is unsafe when it is actually safe, causing a needless costly shutdown" earns full credit.

Checkpoint

Which change would increase the power of a significance test?

Answer the 2 checkpoints as you read.