Let’s introduce an idea that reframes the whole enterprise: your measures are sensors. A questionnaire, a stopwatch, a bathroom scale - each is a device that reads out a number about something, and like any sensor it is imperfect. The number is not the truth; it is the truth plus some noise. Before we trust a measurement to carry an argument, we must ask two separate questions: is it consistent (reliability, this chapter), and is it on target (validity, the next)? These are not the same question, and confusing them wrecks research.
Classical test theory rests on one deceptively simple equation:
\[O = T + E\]
Every observed score (\(O\)) is a person’s true score (\(T\)) - what we actually want - plus error (\(E\)), the sensor’s noise. Because true score and random error are independent, the same decomposition holds for their variances:
\[\text{var}(O) = \text{var}(T) + \text{var}(E)\]
Observed-score variance is always larger than true-score variance, because error only ever adds spread. Our whole goal in measurement is to make the error term small.
Reliability (\(\rho\)) is the fraction of the observed variance that is real - the signal share:
\[\rho = \frac{\text{var}(T)}{\text{var}(O)} = \frac{\text{var}(T)}{\text{var}(T) + \text{var}(E)}\]
It runs from 0 (all noise) to 1 (all signal).
Reliability is necessary but not sufficient: a consistent measure can still be meaningless until it is anchored to something real. Sechrest, McKnight, and McKnight (Sechrest et al. 1996) made exactly this argument for psychotherapy-outcome measures - calibration, not just consistency, is what lets a score be interpreted. It is the same warning the dartboard makes later in this chapter, made in print.
Let’s build a world where we know the true scores and confirm the definition:
set.seed(8)
truth <- rnorm(1000, 100, 15) # the true scores we wish we could see
error <- rnorm(1000, 0, 8) # sensor noise
observed <- truth + error # what we actually record
reliability <- var(truth) / var(observed)
c(reliability = round(reliability, 3))reliability
0.803 About 0.78 of the variance in our observed scores is real signal; the rest is sensor noise. In practice we can never see truth, so we estimate reliability from clever comparisons.
Alpha asks: relative to the total score’s variance, how small are the individual item variances? If items measure the same thing, the whole is much more than the sum of its jittery parts.
\[\alpha = \frac{k}{k-1}\left(1 - \frac{\sum \text{var(item)}}{\text{var(total)}}\right)\]
set.seed(9)
# simulate 6 items that share a common "true" factor plus item-specific noise
common <- rnorm(500)
items <- sapply(1:6, function(i) common + rnorm(500)) # 500 people x 6 items
items <- as.data.frame(items)
k <- ncol(items)
var_items <- sum(apply(items, 2, var))
var_total <- var(rowSums(items))
alpha <- (k / (k - 1)) * (1 - var_items / var_total)
c(cronbach_alpha = round(alpha, 3))cronbach_alpha
0.836 An alpha around 0.85 says the six items hang together well; they are largely reading the same underlying construct. By rough convention, 0.70 is “acceptable,” 0.80+ “good,” but do not lean on the threshold too hard: alpha climbs with more items whether or not they are any good, so a high alpha is necessary, not sufficient.
Cronbach’s alpha, which we just computed, quietly assumes something about the items, and whether that assumption holds decides whether alpha is right or too low. There is a small hierarchy of measurement models, each making a stronger claim about how the items relate to the true score.
Why it matters: when items are congeneric (unequal loadings), alpha underestimates reliability, and McDonald’s omega (\(\omega\)) is the right coefficient. Omega is computed from a factor model, so it can weight items by how well they actually measure the construct. Watch alpha fall short of omega when the loadings differ:
set.seed(3); N <- 500
Tval <- rnorm(N)
lambda <- c(0.9, 0.8, 0.7, 0.5, 0.4) # congeneric: unequal item quality
items <- sapply(lambda, function(l) l * Tval + rnorm(N, sd = sqrt(1 - l^2)))
k <- ncol(items)
alpha <- (k / (k - 1)) * (1 - sum(apply(items, 2, var)) / var(rowSums(items))) # assumes tau-equiv.
fa <- factanal(items, 1) # a one-factor model gives the loadings
lam <- as.numeric(fa$loadings); psi <- fa$uniquenesses
omega <- sum(lam)^2 / (sum(lam)^2 + sum(psi)) # McDonald's omega
c(cronbach_alpha = round(alpha, 3), mcdonald_omega = round(omega, 3))cronbach_alpha mcdonald_omega
0.798 0.815 Omega is the larger, and more honest, number. When the items are truly tau-equivalent (equal loadings), the two coincide; the more the loadings differ, the more alpha understates the reliability. The rule the authors follow: report omega, because it does not assume tau-equivalence. Alpha is a fine quick check, but it is a lower bound, not the last word.
Now the practical consequence. Unreliable measures attenuate (shrink) the correlations you are trying to detect. A real relationship between two constructs looks weaker than it is, purely because your sensors are noisy:
set.seed(10)
Tx <- rnorm(1000); Ty <- 0.6 * Tx + rnorm(1000, 0, 0.8) # true correlation ~ .6
# now observe each through a noisy sensor:
Ox <- Tx + rnorm(1000, 0, 1)
Oy <- Ty + rnorm(1000, 0, 1)
c(true_r = round(cor(Tx, Ty), 2), observed_r = round(cor(Ox, Oy), 2)) true_r observed_r
0.60 0.32 The observed correlation is noticeably smaller than the true one. A real effect can hide behind a bad ruler, which means a null result might be a measurement failure rather than the absence of an effect. Reliability is more than bookkeeping; it can be the difference between finding your effect and missing it.
Enough simulation. Let’s run the whole pipeline on real data: a 10-item scale (x1–x10) from the graduate GLM course (data/moddat2.csv, \(n = 30\)). Three moves: key the items, estimate \(\alpha\), and check which items earn their place.
Some items are worded in reverse, so a high true standing produces a low score - flip them first or they fight the rest of the scale and sink \(\alpha\). We spot a reverse item by its negative item-rest correlation (it correlates the wrong way with the sum of the others):
d <- read.csv("data/moddat2.csv")
X <- as.matrix(d[, paste0("x", 1:10)])
alpha_keyed <- function(X) {
r <- sapply(seq_len(ncol(X)), # item-rest correlations
function(j) cor(X[, j], rowSums(X[, -j, drop = FALSE])))
keys <- ifelse(r < 0, -1, 1) # flip the reverse-worded items
Xk <- sweep(X, 2, keys, `*`)
k <- ncol(Xk)
alpha <- (k / (k - 1)) * (1 - sum(apply(Xk, 2, var)) / var(rowSums(Xk)))
list(alpha = alpha, keys = keys, scored = Xk)
}
a <- alpha_keyed(X)
a$keys # -1 marks a reverse-scored item [1] 1 -1 1 1 -1 1 1 -1 1 1round(a$alpha, 3) # Cronbach's alpha for the keyed scale[1] 0.92A healthy \(\alpha\). Now the classic “should I cut this item?” diagnostic - recompute \(\alpha\) with each item deleted:
drop <- sapply(1:10, function(j) alpha_keyed(X[, -j, drop = FALSE])$alpha)
names(drop) <- colnames(X)
round(drop, 3) x1 x2 x3 x4 x5 x6 x7 x8 x9 x10
0.911 0.910 0.908 0.914 0.911 0.912 0.907 0.912 0.914 0.919 Every “alpha-if-deleted” value sits below the full-scale \(\alpha\), so no item is dead weight - each one, even the weakest, adds more signal than noise. (Had any value come out above the full \(\alpha\), that item would be a candidate for the chopping block: removing it would make the scale more reliable.)
Now the payoff, and the link back to statistical power. Detection is the between-group signal over the within-group noise, and unreliability is that noise (\(S^2_{error}\)). Build the keyed scale score, then ask a real question of it - do women and men differ? - sizing the effect with Cohen’s \(d\):
score <- rowSums(a$scored) # the keyed total score
EScalc <- function(dv, cond) { # McKnight's teaching function for Cohen's d
lv <- levels(factor(cond))
x1 <- dv[cond == lv[1]]; x2 <- dv[cond == lv[2]]
(mean(x1) - mean(x2)) / sqrt((var(x1) + var(x2)) / 2)
}
round(EScalc(score, d$sex), 2)[1] 2.62A large \(d\). But the point is why it was detectable: because the scale was reliable. Had \(\alpha\) been 0.5 instead, half the total-score variance would be pure error, the groups would blur together, and the same real difference could sink below significance. A reliable measure is the cheapest power you will ever buy: you pay once, in the design, and it pays back in every analysis that follows. (In practice you’d reach for psych::alpha(X, check.keys = TRUE), which does the keying, the drop statistics, and much more in one call - but now you know exactly what it is doing under the hood.)
Picture throwing darts. Reliability is how tightly grouped your throws are - consistency. A tight cluster is reliable even if it is nowhere near the bullseye. That “near the bullseye” part - accuracy, being on target - is validity, and it is a completely separate question. A bathroom scale that always reads five pounds heavy is perfectly reliable and perfectly wrong. Reliability is necessary for validity (you cannot hit the bullseye if your throws scatter randomly) but nowhere near sufficient. That is the whole reason the next chapter exists.
Classical test theory splits observed variance into just two pieces, true and error: \(S^2_{obs} = S^2_{true} + S^2_{error}\). But “error” is a bucket, and often we want to know what is in it. Is the noise coming from the particular items we chose? The rater who scored the responses? The day the person happened to show up? Each of these is a source of variance, and lumping them all into one “error” term hides exactly the information you would need to improve the measure.
Generalizability theory (G-theory) opens the bucket. It treats each of these as a facet and estimates how much of the total variance each one contributes, so you can see which sources matter most and where to spend your effort. As the authors put it, the whole basis of G-theory is to estimate the sources of variability and determine which ones affect your results most. Reliability stops being one number and becomes an analysis of where the noise comes from - and, in the follow-up “decision study,” of how many items or raters you would need to pin the true score down. We build that machinery (variance components, the generalizability coefficient, the D-study) in the item response and generalizability chapter. For now, carry the idea: the \(O = T + E\) split is the first step; naming the facets hidden inside that error is the next.
Everything in this chapter lives inside classical test theory - the \(O = T + E\) world, where “reliability” is a single number for a whole test. It is the right place to start, but it has a blind spot: it treats every item as interchangeable and pins a person’s score to the particular test they happened to take. Modern measurement - item response theory and its Rasch model - drops that assumption, estimating each item’s difficulty and each person’s ability on a common scale, so that reliability can vary across the range of the trait. We flag it here as the road ahead; the intuitions you build with \(O = T + E\) are exactly the ones that make item response theory click later.
A reliable sensor is consistent, but consistency is worthless if it is consistently measuring the wrong thing. That is the bullseye question - are we measuring what we intend? - and it is the subject of validity.