What is statistical power?

The four quantities that move together

UCLA's statistical consulting seminar on power analysis defines power as the probability of detecting an effect, given that the effect is really there, which is the probability of rejecting the null hypothesis when it is false. It also sets out the relationship that drives planning: effect size, sample size, significance level and power are linked, so if three are fixed the fourth is determined. Larger effects are easier to detect, larger samples give more power, and a stricter significance level needs more participants for the same power. Most recommendations for power fall between 0.8 and 0.9.

Reading the table

The table shows power for a two group comparison of means with a standardised effect of 0.5, calculated with the noncentral t distribution. With 20 people per group, a real effect of that size would be missed about two times in three. Reaching 80 percent power takes 64 per group, and 90 percent takes 86. The gains shrink as the sample grows: moving from 64 to 100 per group adds only 14 points of power. A smaller expected effect changes everything; halving the effect to 0.25 roughly quadruples the sample needed for the same power. For yes or no outcomes the same logic applies with proportions: detecting a drop from 20 to 15 percent needs far more participants than a drop from 20 to 10 percent, because the absolute difference is half as large.

Why underpowered studies mislead

Low power does not only cause missed effects. Button and colleagues, in a 2013 analysis in Nature Reviews Neuroscience titled Power failure, estimated that the median power of neuroscience studies they examined was low, and showed two further consequences. A significant result from a low powered study is less likely to reflect a true effect, and when an underpowered study does find a true effect, it tends to overestimate its size, because only the lucky large estimates cross the significance line. That is one reason small, striking studies so often fail to replicate at the same effect size.

Planning and reporting a sample size

The CONSORT statement for randomised trials asks authors to report how the sample size was determined, and similar items appear in other reporting guidelines. A complete statement names the primary outcome, the difference the study was designed to detect and why it matters, the assumed standard deviation or control group rate, the significance level, the power, and any allowance for dropouts. On a poster this fits in one methods entry. Choosing the target difference is the hard part: it should be the smallest effect worth knowing about, not the effect researchers hope to see. If recruitment fell short, say so and give both the planned and the achieved numbers, because readers judge a result differently when a study designed for 90 percent power finished with half its intended sample.

Common mistakes

Calculating power after the study from the observed effect is the most common error. The UCLA seminar notes that power computed this way is directly related to the p value and adds no new information; a non significant result will always show low observed power. Other mistakes include assuming an optimistic effect size to justify a small sample, ignoring clustering in designs that randomise clinics or schools, forgetting dropouts, and describing a pilot or feasibility study as if it were powered to test effectiveness.

Where it shows up in the poster builder

Power belongs in the methods, and the protocol block, present on 38 of the 40 layouts, holds up to five labelled entries such as design, participants, intervention, outcome and analysis, so the sample size calculation fits as one line under analysis or participants. The generator writes methods from the brief, so give it the target difference, the power and the planned and achieved sample sizes. Where a board has a participant flow, on ten layouts, the analysed counts should match the numbers the power calculation assumed. The political science example reports a large randomised sample in its methods entries.

Questions people ask