8 Statistical Power: How to Find Things That May Not Be There

Simply put, statistical power is the probability of success - where “success” means detecting an effect that is really there. In the language of the last chapter, power is the one cell of the error table you can actually plan for before you collect a single observation. Everything else about a study - your hypotheses, your \(\alpha\), the vagaries of sampling - is set once the data arrive. Power is where you take back control.

Power is a SWAG - and that’s the honest part

Here is the framing (PEM) I use with my own students. A power analysis is really an estimate of

\[P(\text{Party at the Pub!} \mid \text{your research plans}),\]

the probability you’ll be celebrating a detected effect given the study you’re about to run - which is to say, a Scientific Wild Ass Guess. That is not an insult to the method; it is honesty about it, because power depends on things you must guess before any data exist (chiefly the true effect size). Why bother guessing at all? Because power protects the things you care about (funding, your time, scarce participants), and it is the one part of a study you can shape in advance. The goal is not false precision; it is self-reliance. Power estimation is daunting, important, and easy to fool yourself with, and it is far more within your reach than most people realize, because when the formulas intimidate you, you can always simulate it.

8.1 Learning Objectives

Define power as \(1 - \beta\) and locate it in the error table.
Identify the four levers that raise or lower power.
Use R to compute power and to plan a sample size.
Understand the honest trade-off between power and Type I error.

8.2 Power Is One Cell of a Table You Already Know

Recall the decision table from Hypothesis Testing:

	\(H_0\) is really true	\(H_0\) is really false
We reject \(H_0\)	Type I error (\(\alpha\))	Correct ✓ (power, \(1-\beta\))
We fail to reject \(H_0\)	Correct ✓ (\(1-\alpha\))	Type II error (\(\beta\))

Power is the highlighted cell: the probability we correctly reject the null given that the null is false - that is, given there is a real effect to find. Its complement is \(\beta\), the Type II error rate (the miss). By long convention we aim for power of 0.80: an 80% chance of catching a true effect, and thus a 20% chance of missing it. That 20% should bother you; it is why underpowered research is such a serious problem. When power is low, you not only miss real effects, but the significant results you do find are more likely to be flukes.

8.3 The Four Levers

Power is a function of four things, and only four. Three you can influence; one you mostly cannot.

Effect size - how big the true effect is. Bigger effects are easier to detect. We usually express it in standardized units, e.g., Cohen’s \(d = \frac{\text{mean difference}}{\text{standard deviation}}\). You do not control the true effect, but you can sometimes measure it more cleanly to keep it from being buried in noise.
Sample size (\(n\)) - more data, more power. This is the lever you actually turn. Nearly every underpowered study is really a too-small study.
Significance level (\(\alpha\)) - a more lenient \(\alpha\) (say .10) makes it easier to reject, so power goes up; a stricter \(\alpha\) (say .01) makes it harder to reject, so power goes down. (More on this trade-off below; it is exactly backwards in a lot of lecture notes, including an old version of mine.)
Directionality - a one-tailed test concentrates all of \(\alpha\) in the predicted direction, so it has more power there - at the cost of being blind to an effect in the other direction.

The \(\alpha\)-Power Trade-off (Getting It Right)

It is tempting, and wrong, to say “lower \(\alpha\) gives higher power.” Lowering \(\alpha\) shrinks the rejection region, which makes it harder to reject the null, which lowers power. You cannot buy fewer false alarms (small \(\alpha\)) and more hits (high power) for free by tightening \(\alpha\) alone; those two pull in opposite directions. The only way to improve both at once is to change something outside this trade-off - almost always, collect a larger sample.

Power is a story about variance

Every lever above is really about the signal-to-noise ratio we learned to decompose as \(S^2_{obs} = S^2_{true} + S^2_{error}\). Detecting an effect gets easier as the between-group variance (the signal - your effect) grows relative to the within-group variance (the noise). That is exactly why Cohen’s \(d\) puts the mean difference over the standard deviation: power lives in the ratio, not the numerator alone. So two moves you can sometimes control - a cleaner measure (less \(S^2_{error}\)) and a stronger manipulation (more signal) - buy you power just as surely as more subjects do.

8.4 Rules of Thumb (A Rough First Pass)

Before any software, a few back-of-the-envelope numbers get you into the right ballpark for the conventional 80% power:

Comparing groups (mean differences): about \(n = 30\) per group is the classic target - and never go below ~7 per cell if you can possibly avoid it.
Associations (correlations, \(r\)): you want \(N > 50\) or so.
Measurement models (EFA/CFA): \(N > 300\) is usually safe.

These are deliberately crude (they hide the effect size you’re implicitly assuming), but they are enough to sanity-check a plan, or to notice a design is hopeless before you spend anything. Then you refine with a real calculation, which comes in two flavors: a point estimate from a formula (next), and a simulation you can watch (further down). The right habit is to reach for an existing tool first, make sure you understand the model you’re planning (a \(t\)-test, a regression, a mixed model), and then simulate to check your guesses.

8.5 Seeing Power in R

Base R ships with power.t.test(), which ties together effect size, \(n\), \(\alpha\), and power - give it any three and it solves for the fourth. First, the trade-offs, made concrete:

# Hold the effect (d = 0.5) and n = 30 per group; vary alpha
sapply(c(0.01, 0.05, 0.10), function(a)
  power.t.test(n = 30, delta = 0.5, sd = 1, sig.level = a)$power)

[1] 0.2437156 0.4778410 0.6060253

Read left to right: as \(\alpha\) goes from strict (.01) to lenient (.10), power rises. Stricter \(\alpha\) costs you power - the trade-off, in numbers.

Now the lever you actually control - sample size:

# Hold effect (d = 0.5) and alpha (.05); vary n per group
sapply(c(20, 50, 100), function(nn)
  power.t.test(n = nn, delta = 0.5, sd = 1, sig.level = 0.05)$power)

[1] 0.3377084 0.6968888 0.9404272

More data, more power - and unlike the \(\alpha\) trade-off, this buys power without raising your false-alarm rate.

8.6 Planning a Study: Solve for \(n\)

The real payoff is planning. If you want 80% power to detect a medium effect (\(d = 0.5\)) at \(\alpha = .05\), how many people do you need? Leave n out and let R solve for it:

power.t.test(delta = 0.5, sd = 1, sig.level = 0.05, power = 0.80)


     Two-sample t test power calculation 

              n = 63.76576
          delta = 0.5
             sd = 1
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

The answer (about 64 per group) is a number you want before you run the study, not after. Running first and computing power later, so-called “post-hoc power,” tells you almost nothing useful; power is a design tool, not a consolation prize.

8.7 Power Is Really Just Simulation

If the formulas ever feel like a black box, remember they are only shortcuts for something you can watch directly. The recipe is four steps: generate data from a world where the effect is real, run your model, collect whether it hit (\(p < \alpha\)), and repeat a few thousand times. The fraction of hits is the power, and unlike a formula, this recipe works for any design you can simulate (a mixed model, a mediation, a custom test), long after the closed-form equations give up:

set.seed(1988)
d <- 0.5; n <- 64
hits <- replicate(2000, {
  group1 <- rnorm(n, mean = 0,  sd = 1)
  group2 <- rnorm(n, mean = d,  sd = 1)   # a real difference of d
  t.test(group1, group2)$p.value < 0.05   # did we detect it?
})
mean(hits)   # simulated power ~ 0.80

[1] 0.812

The simulation lands right where power.t.test() said it would. It is simply a count of how often a true effect trips the wire.

8.8 Challenge

Do One Yourself

Use power.t.test() to find the sample size needed for 80% power to detect a small effect (\(d = 0.2\)) at \(\alpha = .05\). Compare it to the \(n\) for a large effect (\(d = 0.8\)). Why the enormous difference?
Fix \(n = 40\) per group and \(d = 0.4\). Compute power at \(\alpha = .01\), \(.05\), and \(.10\). State in one sentence what you are giving up as you raise \(\alpha\) to gain power.
Write the simulation from the last section yourself, but for a one-sample test against a null mean. Confirm your simulated power matches power.t.test(..., type = "one.sample").

If part 2 makes you a little uncomfortable, good: that discomfort is the trade-off, and ignoring it is how underpowered studies get funded.

8.9 Where We Go Next

Power closes out our tour of inference: we now know how to test, how we can be wrong, and how to design a study that can actually find what it is looking for. Next we turn from testing our conclusions to trusting our measurements - the twin questions of reliability and validity. After all, the most powerful test in the world is worthless if the ruler is rubber.