We closed the last chapter with a slogan: ANOVA is regression with categorical predictors. That is true, and it is the deep view. But ANOVA also has a classical view, one built entirely out of the variance ideas from early in the book, and seeing that view makes the machinery click. Analysis of variance earns its name honestly: it works by partitioning variance into the part explained by group membership and the part left over. Let’s take it apart.
One-way ANOVA asks: do three or more groups differ on a continuous outcome? Say we test a plant fertilizer at three doses:
set.seed(17)
dose <- factor(rep(c("none", "low", "high"), each = 30),
levels = c("none", "low", "high"))
growth <- rnorm(90, mean = rep(c(10, 12, 15), each = 30), sd = 3)
tapply(growth, dose, mean) none low high
11.04343 11.85207 14.23987 The group means differ in our sample - but is that real, or just sampling noise? ANOVA answers by comparing two kinds of variation.
Here is the whole idea in one line: the total spread of every observation around the grand mean can be split cleanly into spread between groups and spread within groups.
\[SS_{total} = SS_{between} + SS_{within}\]
Let’s build all three by hand and confirm they add up:
grand <- mean(growth)
group_means <- tapply(growth, dose, mean)
SS_total <- sum((growth - grand)^2)
SS_between <- sum(table(dose) * (group_means - grand)^2)
SS_within <- sum((growth - group_means[dose])^2)
c(SS_total = SS_total,
between_plus_within = SS_between + SS_within) # they match SS_total between_plus_within
1060.502 1060.502 The total variance really does split exactly into “explained by group” plus “unexplained.” That partition is analysis of variance.
We cannot compare sums of squares directly - they depend on how many things went into each. We convert each to a mean square by dividing by its degrees of freedom, then take their ratio:
\[F = \frac{MS_{between}}{MS_{within}} = \frac{SS_{between}/(k-1)}{SS_{within}/(N-k)}\]
with \(k\) groups and \(N\) observations. \(F\) asks a simple question: is the variation between groups bigger than the variation within them? If groups matter, the between-group spread is large relative to the within-group noise and \(F\) climbs well above 1.
k <- 3; N <- 90
MS_between <- SS_between / (k - 1)
MS_within <- SS_within / (N - k)
F_stat <- MS_between / MS_within
p_val <- pf(F_stat, df1 = k - 1, df2 = N - k, lower.tail = FALSE)
round(c(F = F_stat, p = p_val), 5) F p
8.05691 0.00062 And of course, R will do the bookkeeping for us with aov() - confirm it matches our by-hand work:
summary(aov(growth ~ dose)) Df Sum Sq Mean Sq F value Pr(>F)
dose 2 165.7 82.86 8.057 0.000616 ***
Residuals 87 894.8 10.28
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1ONEWAY growth BY dose /STATISTICS DESCRIPTIVES /POSTHOC=TUKEY.using GLM
ftest(lm(@formula(growth ~ dose), df).model)Same \(F\), same p, exactly as we computed by hand.
To honor the slogan from the last chapter, let’s prove the classical ANOVA and the regression GLM are one and the same by pulling the identical \(F\) out of lm():
fit <- lm(growth ~ dose)
anova(fit) # the regression's F test == the ANOVA| Df | Sum Sq | Mean Sq | F value | Pr(>F) | |
|---|---|---|---|---|---|
| dose | 2 | 165.7271 | 82.86353 | 8.056913 | 0.0006163 |
| Residuals | 87 | 894.7753 | 10.28477 | NA | NA |
c(regression_F = summary(fit)$fstatistic[1],
aov_F = summary(aov(growth ~ dose))[[1]][["F value"]][1])regression_F.value aov_F
8.056913 8.056913 REGRESSION /STATISTICS COEFF R ANOVA /DEPENDENT growth /METHOD=ENTER dose.using GLM
coeftable(lm(@formula(growth ~ dose), df))Identical. The “two techniques” were always one. ANOVA is just the variance-partitioning story told about a regression on categorical predictors.
The omnibus \(F\) only says “somewhere among these groups there is a difference.” It does not say which groups differ. For that we run post-hoc comparisons - and because we are now making several comparisons at once, we must control the inflated false-alarm rate (recall Type I error from the hypothesis-testing chapter). Tukey’s Honest Significant Difference does exactly that:
TukeyHSD(aov(growth ~ dose)) Tukey multiple comparisons of means
95% family-wise confidence level
Fit: aov(formula = growth ~ dose)
$dose
diff lwr upr p adj
low-none 0.8086412 -1.1658052 2.783088 0.5935699
high-none 3.1964393 1.2219930 5.170886 0.0006306
high-low 2.3877981 0.4133518 4.362245 0.0135958Each row is a pairwise comparison with a confidence interval already adjusted for the fact that we peeked at all of them. Any interval that excludes zero is a real difference; any that includes zero is not.
The two-step procedure, omnibus \(F\) first and post-hoc comparisons only if it is significant, is not just a formality. Running every pairwise t-test without the omnibus gate (and without correction) is how you manufacture false positives, one comparison at a time. The \(F\) is the gatekeeper, and Tukey (or a similar correction) keeps the gate honest.
summary(aov(...)).anova(lm(y ~ group)).You now hold the thread that runs through all of it: quantify your uncertainty, and remember that variance is the currency you quantify it in. From the first histogram to this \(F\) ratio, it was one idea the whole way down. But there is a loose end worth pulling. Every ANOVA you just ran handed R a factor and trusted it to build the right columns of numbers, and how those columns are built quietly decides what each coefficient means. That is the subject of the next chapter, where we stop trusting the defaults and take the coding into our own hands.