17  ANOVA and the General Linear Model

The way statistics is usually taught tends to hide something important: ANOVA is not a different technique from regression. It is regression, the very same least-squares machinery from the last two chapters, applied to categorical predictors. Analysis of variance, t-tests, regression: they are all special cases of one thing, the General Linear Model (GLM). Once you see the unification, a large part of your statistics education collapses into a single idea.

The only new skill you need is coding: how to turn a category (“cat,” “dog”) into numbers a regression can chew on. That is the whole chapter.

17.1 Learning Objectives

  1. See that a categorical predictor is just a set of coded numbers in a regression.
  2. Interpret dummy-coded coefficients as differences from a reference group.
  3. Interpret effects-coded coefficients as differences from the grand mean.
  4. Recognize the ANOVA \(F\) as the same omnibus test regression already gave you.

17.2 A Categorical Predictor Is a Sliced-Up Continuous One

Let us start with a continuous relationship. Suppose some trait \(x\) makes people happy:

set.seed(1925)  # Fisher's "Statistical Methods for Research Workers"
x <- sort(rnorm(100))
happy <- x + rnorm(100)

Now imagine \(x\) measures something like “dogness,” where higher values mean more dog-like. We rarely measure dogness on a continuum; instead we lump animals into categories. Let us chop the continuum into four pet types:

pet <- cut(x, 4, labels = c("cat", "fish", "pig", "dog"))
table(pet)
pet
 cat fish  pig  dog 
  17   47   29    7

Does pet type relate to happiness? A regression will happily tell us, even though pet is a factor:

lm_pet <- lm(happy ~ pet)
round(coef(lm_pet), 3)
(Intercept)     petfish      petpig      petdog 
     -1.323       1.185       2.052       2.820 
ONEWAY happy BY pet /STATISTICS DESCRIPTIVES.
using GLM; ftest(lm(@formula(happy ~ pet), df).model)

R fit it without complaint. But look at those coefficient names - petfish, petpig, petdog. Where did they come from, and what do they mean? That is the coding question, and there is more than one answer.

17.3 Dummy Coding: Differences From a Reference

By default, R uses dummy coding (contr.treatment): it picks the first level (“cat”) as the reference, and each coefficient is the difference between that group’s mean and the reference group’s mean. Let us prove it by pulling the group means directly:

means <- aggregate(happy, by = list(pet = pet), FUN = mean)
means
pet x
cat -1.3234082
fish -0.1385552
pig 0.7284205
dog 1.4965846

Now rebuild every coefficient from those means:

# intercept should be the reference (cat) mean:
c(intercept = coef(lm_pet)[[1]], cat_mean = means$x[1])
intercept  cat_mean 
-1.323408 -1.323408 
# petfish should be (fish mean - cat mean):
c(petfish_coef = coef(lm_pet)[["petfish"]], fish_minus_cat = means$x[2] - means$x[1])
  petfish_coef fish_minus_cat 
      1.184853       1.184853 
# petdog should be (dog mean - cat mean):
c(petdog_coef = coef(lm_pet)[["petdog"]], dog_minus_cat = means$x[4] - means$x[1])
  petdog_coef dog_minus_cat 
     2.819993      2.819993

There is no magic in the box. A dummy-coded coefficient is a difference between two group means, and the intercept is the reference group’s mean. That is exactly what a t-test between two groups reports, which is your first hint that t-tests, too, are just this same model.

17.4 Effects Coding: Differences From the Grand Mean

Dummy coding answers “how does each group differ from the reference?” Often we would rather ask “how does each group differ from the overall average?” For that we switch to effects coding (contr.sum):

pet_e <- pet
contrasts(pet_e) <- contr.sum(4)
lm_pet_e <- lm(happy ~ pet_e)
round(coef(lm_pet_e), 3)
(Intercept)      pet_e1      pet_e2      pet_e3 
      0.191      -1.514      -0.329       0.538

The coefficients changed, because the question changed. Now the intercept is the grand mean - and here is a subtlety worth stopping for: the grand mean of effects coding is the mean of the group means (each group weighted equally), not the mean of the raw scores. With unequal group sizes those two differ, and confusing them is a classic error.

GM <- mean(means$x)                       # mean of the group means (unweighted)
c(intercept = coef(lm_pet_e)[[1]], grand_mean = GM)
 intercept grand_mean 
 0.1907604  0.1907604

Each effects coefficient is then that group’s mean minus the grand mean:

# first coefficient = cat mean - grand mean:
c(coef1 = coef(lm_pet_e)[[2]], cat_minus_GM = means$x[1] - GM)
       coef1 cat_minus_GM 
   -1.514169    -1.514169
ImportantWhere Did the Last Group Go?

Count the coefficients: with four groups, effects coding gives you an intercept plus only three effects. The fourth group (“dog”) seems to have vanished. It has not - it is implied. Because the effects are deviations from the grand mean, they must sum to zero, so the last group’s effect is minus the sum of the others:

dog_effect_implied <- -(coef(lm_pet_e)[[2]] + coef(lm_pet_e)[[3]] + coef(lm_pet_e)[[4]])
c(implied_dog_effect = dog_effect_implied, dog_minus_GM = means$x[4] - GM)
implied_dog_effect       dog_minus_GM 
          1.305824           1.305824

They match. The “missing” group is never missing; it is the one the constraint solves for. (Work the algebra once and reference levels stop being confusing.)

NoteFrom the Mailbag (a real question from PSYC 754)

“I don’t see the difference between dummy coding and contrast coding. Why would labeling a 0 vs. -1 be a big change? … How is effects coding different from regular contrast coding?”

A perennial knot, so let’s cut it. Dummy coding uses 0s and 1s and compares each group to a single reference group (the one coded 0). Effects coding swaps that reference 0 for a \(-1\), and that small change quietly rewrites the question: each coefficient now compares a group to the grand mean rather than to a reference group. Contrast coding is the general family both belong to - any set of codes chosen to test a specific comparison - and effects coding is simply the member that yields deviations from the grand mean. So the “big change” from \(0\) to \(-1\) is really a change in what you compare against: one baseline group, or the average of them all.

17.5 The Punchline: This Is ANOVA

Whether you dummy-code or effects-code, the overall test - “do the group means differ at all?” - is identical, and it is identical to what analysis of variance calls the omnibus \(F\). Watch the same number fall out three ways:

# 1) the regression's own F test
summary(lm_pet)$fstatistic[1]
   value 
20.99332 
# 2) run it as an ANOVA on the regression object
anova(lm_pet)["pet", "F value"]
[1] 20.99332
# 3) the classic aov() function
summary(aov(happy ~ pet))[[1]]["pet", "F value"]
[1] 20.99332

One model, one \(F\), three names. ANOVA is regression with categorical predictors: the coding scheme only changes how we slice up and label the group differences, never the overall test. This is the General Linear Model: pick your predictors (continuous, categorical, or both), pick your coding, and it is all least squares underneath.

17.6 Two Categorical Predictors and Their Interaction

Add a second factor and you get a factorial design - still just regression. Suppose messiness also matters:

mess  <- factor(ifelse(rnorm(100) > 0, "messy", "tidy"))
sad   <- -x + as.numeric(mess) + rnorm(100)
lm_fac <- lm(sad ~ pet * mess)
round(coef(lm_fac), 3)
     (Intercept)          petfish           petpig           petdog 
           2.079           -1.139           -1.609           -2.789 
        messtidy petfish:messtidy  petpig:messtidy  petdog:messtidy 
           0.849            0.614            0.117           -0.403 
UNIANOVA sad BY pet mess /DESIGN=pet mess pet*mess.
using GLM; lm(@formula(sad ~ pet * mess), df)

The pet:messmessy terms are the interaction, and the same logic applies here. An interaction coefficient is a difference of differences: how the effect of pet changes across levels of mess. It is the cross-product of the two predictors’ codes. A “significant interaction” is nothing to worry about; it just means the group-mean pattern is not purely additive.

17.7 Beyond the Straight Line: The Generalized Linear Model

Everything so far assumes a continuous outcome with roughly normal residuals. But plenty of outcomes are not like that - a yes/no, a count, a rare event. The generalized linear model extends the same framework by letting the outcome follow a different distribution through a link function. The most common case is a binary outcome via logistic regression:

passed <- rbinom(100, size = 1, prob = plogis(x))   # 0/1 outcome driven by x
glm_fit <- glm(passed ~ x, family = binomial)
round(coef(glm_fit), 3)
(Intercept)           x 
     -0.181       0.728 
LOGISTIC REGRESSION VARIABLES passed WITH x.
using GLM; glm(@formula(passed ~ x), df, Binomial(), LogitLink())

Same formula syntax, same partialling logic, same goal of quantifying the uncertainty. We have just swapped lm() for glm() and told it the outcome is binomial. Linear model, general linear model, generalized linear model: it is the same core idea extended step by step.

17.8 Challenge

TipDo One Yourself

Using the pet factor and happy from this chapter:

  1. Fit the model with dummy coding and, without reading the coefficients off the summary, reproduce all four coefficients (intercept + three) from the group means.
  2. Refit with effects coding and reproduce the intercept (grand mean) and all four group effects - including the implied “dog” effect - from the group means.
  3. Confirm the omnibus \(F\) is identical under both coding schemes, and identical to summary(aov(happy ~ pet)).
  4. In one sentence each, state what the intercept means under dummy coding versus effects coding. If you can do that cleanly, you understand why “ANOVA vs. regression” was always a false choice.

17.9 Where We Go Next

You now hold the unifying idea of the whole enterprise: the General Linear Model. Correlation, t-tests, regression, ANOVA, ANCOVA - all one model wearing different outfits. We build these coding schemes by hand in the dedicated coding chapter; for the definitive treatment, see Cohen, Cohen, West, and Aiken (Aiken 2002) and the accessible comparison of dummy and effects coding in Alkharusi (Alkharusi 2012); for analysis of variance specifically, Iversen and Norpoth (Iversen and Norpoth 1987) remains a compact classic. From here, the road leads into the specific ANOVA designs (one-way, factorial, repeated measures), each of which is just another coded regression.

Aiken, Stephen G. West, Patricia Cohen. 2002. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. 3rd ed. Routledge. https://doi.org/10.4324/9780203774441.
Alkharusi, Hussain. 2012. “Categorical Variables in Regression Analysis: A Comparison of Dummy and Effect Coding.” International Journal of Education 4 (April): 202–10. https://doi.org/10.5296/ije.v4i2.1962.
Iversen, Gudmund R., and Helmut Norpoth. 1987. Analysis of Variance. 2nd ed. Sage University Papers. Quantitative Applications in the Social Sciences, no. 07-001. Sage Publications.