The General Linear Model (GLM) is a statistical model that is used to model the relationship between a single dependent variable and one or more independent variables. Multiple regression and ANOVA are specific applications of the GLM. In fact, the t-test, correlation, regression, ANOVA, and ANCOVA are really one technique seen from different angles. Learn the one, and the rest come with it.
Strip away the vocabulary and every model in this section is:
\[Y = b_0 + b_1 X_1 + b_2 X_2 + \cdots + b_k X_k + e\]
An outcome \(Y\) is modeled as a weighted sum of predictors plus error \(e\). That is it. Whether the \(X\)’s are continuous (regression), categorical (ANOVA), or a mix (ANCOVA) changes only how we code them - never the underlying machine. And, as we saw in the covariance chapter, the weights themselves are built from covariances and variances. Covariance is the fuel; the GLM is the engine.
Let’s take one simple question, do two groups differ on an outcome, and answer it with three “different” methods. Watch the p-value:
set.seed(12)
group <- factor(rep(c("control", "treatment"), each = 40))
y <- rnorm(80) + ifelse(group == "treatment", 0.6, 0)
p_ttest <- t.test(y ~ group, var.equal = TRUE)$p.value
p_anova <- summary(aov(y ~ group))[[1]][["Pr(>F)"]][1]
p_regr <- summary(lm(y ~ group))$coefficients[2, "Pr(>|t|)"]
round(c(t_test = p_ttest, ANOVA = p_anova, regression = p_regr), 6) t_test ANOVA regression
0.00026 0.00026 0.00026 * Same question, three ways: t-test, ANOVA, regression.
T-TEST GROUPS=group('control' 'treatment') /VARIABLES=y.
ONEWAY y BY group /STATISTICS DESCRIPTIVES.
REGRESSION /STATISTICS COEFF R ANOVA /DEPENDENT y /METHOD=ENTER group.using HypothesisTests, GLM
EqualVarianceTTest(df.y[df.group .== "control"], df.y[df.group .== "treatment"])
ftest(lm(@formula(y ~ group), df).model)
coeftable(lm(@formula(y ~ group), df))Three identical p-values. The t-test, the ANOVA, and the regression are literally the same model here - the t-test is a two-group ANOVA, the two-group ANOVA is a regression with one dummy-coded predictor, and (as a bonus) \(F = t^2\):
t_stat <- t.test(y ~ group, var.equal = TRUE)$statistic
F_stat <- summary(aov(y ~ group))[[1]][["F value"]][1]
round(c(t_squared = t_stat^2, F = F_stat), 4) # F = t^2t_squared.t F
14.6466 14.6466 Once you see this, a huge amount of statistics stops being a pile of disconnected recipes and becomes a single, learnable idea.
The GLM is general because it swallows so many named procedures. But it does carry assumptions - chiefly that the errors are roughly normal and that the relationship is linear in the parameters. When the outcome is not continuous-and-normal (a yes/no, a count), we need the generalized linear model, logistic and Poisson regression, which we met briefly at the end of the ANOVA & GLM chapter. Do not confuse the two: the general linear model is the unifying frame for this whole section; the generalized linear model extends it to other kinds of outcomes.
Everything remaining in this book is a special case of the equation above:
Same model, different predictors. Keep the one equation in your head and you will not get lost.
t.test(..., var.equal = TRUE), aov(), and lm() give the identical p-value, and that \(F = t^2\).lm(y ~ group + covariate). You have just run an ANCOVA without learning a new technique - explain to a friend why that is not cheating.Let’s build the bridge that makes the “categorical predictors are just regression” claim concrete, using the simplest possible categorical predictor - a single yes/no. That is the point-biserial correlation.