P values and confidence intervals, explained

Almost every clinical paper reports P values and confidence intervals, yet both are often misread, even by experienced authors. In 2016 the American Statistical Association (ASA) published a formal statement on P values because misuse was so common. This guide explains, without formulas, what the two numbers mean, why "significant" is not the same as "important", and how to report them in your manuscript.

What a p value is (and is not)

A P value is calculated from your data under a statistical model, usually one that assumes there is no real difference or effect (the "null hypothesis") together with all the other assumptions of the test. Roughly, it tells you how unusual your data would be if that model were true. A small P value means the data fit poorly with the model; it does not tell you which part of the model is wrong.

The ASA statement sets out six principles. In plain words:

1. P values can indicate how incompatible the data are with a specified statistical model.
2. They do not measure the probability that the hypothesis is true, or the probability that the data were produced by chance alone.
3. Conclusions should not rest only on whether a P value passes a threshold such as 0.05.
4. Proper inference needs full reporting and transparency, not only the analyses that "worked".
5. A P value does not measure the size of an effect or the importance of a result.
6. On its own, a P value is not a good measure of evidence about a model or hypothesis.

Two common mistakes follow from this. First, "P = 0.01, so there is a 1% chance the null hypothesis is true" is wrong: a guide to misinterpretations by Greenland and colleagues explains that the P value assumes the test hypothesis is true and is not a probability for it. Second, "P = 0.20, so there is no difference" is also wrong: a non-significant result may simply mean the study was too small to detect a difference.

Reading a confidence interval

A 95% confidence interval (CI) gives a range of values for the effect that are reasonably compatible with your data, given the model. It shows two things a P value hides: the likely size of the effect and how precisely it was estimated.

Example: "Mortality was lower in the treated group (risk difference −4%, 95% CI −9% to 1%)." The best estimate is a 4-point reduction; the data are compatible with anything from a 9-point reduction to a 1-point increase. A wide interval means an imprecise estimate, usually because the sample was small.

Be careful with the wording. Greenland and colleagues point out that it is a fallacy to say a particular computed interval has a 95% chance of containing the true value; the 95% describes how often such intervals would contain the true value over many repeated studies, if all the assumptions were correct. They also warn that overlapping confidence intervals for two groups do not by themselves show that the groups do not differ; compare groups with a test or an interval for the difference itself.

Statistical and clinical significance

"Statistically significant" only means that the P value fell below a chosen cut-off. It does not mean the result matters to patients. Greenland and colleagues note that in a large study even very small effects can produce a small P value. The reverse is also true: a clinically important effect may not reach significance in a small study.

So, before you analyse, decide what difference would matter clinically. The SAMPL guidelines for statistical reporting suggest stating, where possible, the smallest difference considered clinically important. Then judge your result by the confidence interval: does it include values that would change practice, values that would not, or both?

• Narrow CI, all clinically important values: strong support for a meaningful effect.
• Narrow CI, all trivial values: the effect, if any, is probably too small to matter, even if P is below 0.05.
• Wide CI spanning both: the study is inconclusive; say so honestly.

How to report them

This journal's Instructions for Authors ask for exact P values (for example P=0.048) together with effect sizes and 95% confidence intervals where appropriate, and for the significance level and software (with version) to be stated in the Statistical Analysis subsection. ICMJE gives the same advice: present findings with indicators of uncertainty such as confidence intervals, and avoid relying solely on P values, which do not convey effect size or precision. SAMPL adds practical details. A short checklist:

• Report the effect estimate first, then its 95% CI, then the exact P value: "OR 1.8 (95% CI 1.1 to 2.9); P=0.02".
• Do not write only "P<0.05" or "NS".
• State the alpha level used to define significance and whether tests were one- or two-tailed.
• Report the number of patients analysed in each group.
• In the discussion, interpret the size and precision of the effect, not just whether P crossed 0.05.

Sources

• Wasserstein RL, Lazar NA. The ASA's statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129-33. doi:10.1080/00031305.2016.1154108. Summary of the six principles (ASA press release): https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf
• Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337-50. doi:10.1007/s10654-016-0149-3. https://doi.org/10.1007/s10654-016-0149-3
• Lang TA, Altman DG. Basic statistical reporting for articles published in biomedical journals: the SAMPL Guidelines. Int J Nurs Stud. 2015;52(1):5-9. EQUATOR record: https://www.equator-network.org/reporting-guidelines/sampl/
• ICMJE. Preparing a manuscript for submission to a medical journal (Statistics). https://www.icmje.org/recommendations/browse/manuscript-preparation/preparing-for-submission.html
• This journal: Instructions for Authors (Statistics).