P-Values: What They Are and What They Are Not

You’ll be able to

State the definition of a p-value precisely, including the conditional structure
Identify and correct the standard misinterpretations
Distinguish statistical significance from practical importance

The definition, with the conditional intact

A p-value is the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. Every word of that conditional matters. It is P(data this extreme | H₀ true), and it is emphatically not P(H₀ true | data). Reversing the conditional is the same error as confusing P(positive test | disease) with P(disease | positive test), and it is just as consequential: the p-value cannot tell you the probability that the null hypothesis is correct, because it was computed by assuming that it is.

What a small p-value licenses

A small p-value says the observed data would be surprising if the null were true, which is evidence against the null. It does not measure the size of the effect, and it does not prove the alternative. A large p-value is weaker still: it means the data are consistent with the null, which is not the same as the null being true. This is why the correct language is "we fail to reject H₀" rather than "we accept H₀" — an absence of evidence against a claim is not evidence for it.

Significance is not importance

With a large enough sample, an effect of no practical consequence whatever will be statistically significant, because the standard error shrinks toward zero and any nonzero difference eventually clears the threshold. A drug that lowers blood pressure by 0.4 mmHg in a trial of 100,000 people may have p < 0.001 and be medically irrelevant. Conversely, an important effect can fail to reach significance in a small study. Statistical significance answers "is this distinguishable from chance?"; practical importance answers "is it big enough to matter?" — and only the second requires judgment about the subject.

On the exam

The template: "Assuming [H₀ in context] is true, there is a p-value probability of observing a sample result at least as extreme as the one obtained purely by chance." Then the conclusion: "Because p < α, we reject H₀ and conclude there is convincing evidence that [H_a in context]." Contextual wording is required for the point in both sentences.

Worked example

A test yields p = 0.03 at α = 0.05. State the conclusion, interpret the p-value, and evaluate this claim: "There is a 3% chance the null hypothesis is true."

1.Compare p with α: 0.03 < 0.05, so reject H₀.
2.State the conclusion in context: there is convincing evidence for the alternative.
3.Interpret the p-value with the conditional in the right order.
4.Evaluate the claim: it reverses the conditional, and the p-value was computed under the assumption that H₀ is true, so it cannot report the probability of that assumption.

Answer: Since p = 0.03 is less than α = 0.05, we reject H₀ and conclude there is convincing evidence for the alternative in context. The p-value means that IF the null hypothesis were true, there would be a 3% probability of obtaining a sample result at least as extreme as the one observed. The claim is incorrect: it states P(H₀ true | data) when the p-value is P(data this extreme | H₀ true). The p-value assumes the null is true as its starting point, so it can never measure how likely that assumption is.

Checkpoint

A study reports p = 0.42. The correct conclusion is that —

Checkpoint

A very large study finds a statistically significant difference of 0.2 points on a 100-point scale. The best assessment is that the result is —

Answer the 2 checkpoints as you read.