Our first real study is one of our own - a collaboration (McKnight, Stuewig, Kashdan) that started from a point you have already met in this book: the mean throws away information. In the central tendency chapter that was a general idea. Here it becomes a research program, with real patients and a real depression scale. It also includes something you rarely see in a finished paper: a claim we made, tested, and then had to take back. We want you to follow the decisions along the way, not just the results.
The instrument is the Beck Depression Inventory (BDI-II): 21 symptoms (sadness, guilt, sleep, appetite, …), each scored 0 (absent) to 3 (severe). Clinicians add the 21 items into one number, call a 15 “mild depression,” and move on. This study asks what that number leaves out.
Start with pure arithmetic - no data and no model, just counting. How many different 21-item profiles produce a total of exactly 15? The answer comes from the inclusion–exclusion principle, and it is worth computing yourself because the number is larger than most people expect:
n_patterns <- function(k, a, n) { # k items, each 0..a, summing to n
imax <- floor(n / (a + 1))
sum(sapply(0:imax, function(i)
(-1)^i * choose(k, i) * choose(n - i * (a + 1) + k - 1, k - 1)))
}
format(n_patterns(21, 3, 15), big.mark = ",") # ways to score 15 on the BDI-II[1] "1,653,955,415"1.65 billion distinct symptom profiles all land on “15 = mild depression.” Across all possible totals there are \(4^{21} \approx 4.4\) trillion profiles, peaking at 338 billion different ways to score a 31. The clinical convention - “minimal / mild / moderate / severe” - treats all of them as the same. That is a large amount of discarded information, and the question is whether any of it mattered.
The clinical and student profiles behind this project are the authors’ own research collections and are not public. So this chapter illustrates the mechanics on a simulated BDI-II-like sample - 21 items scored 0–3, generated from a single latent severity dimension so the symptoms covary the way real ones do - while every finding it quotes comes from the real six-sample analysis. Watch the machinery here; trust the numbers to the published study.
The billion is abstract; two people make it concrete. First we build the simulated sample, then pull two who both scored exactly 15:
set.seed(2026)
N <- 1200; k <- 21
symptoms <- c("sadness","pessimism","past_failure","loss_pleasure","guilt","punishment",
"self_dislike","self_critical","suicidal","crying","agitation","loss_interest",
"indecision","worthlessness","loss_energy","sleep","irritability","appetite",
"concentration","tiredness","loss_sex_interest")
theta <- rnorm(N, mean = -0.5) # latent depression severity (one per person)
diff <- sort(rnorm(k, mean = 2.6, sd = 1.3)) # item "difficulty": most symptoms are rare
bdi <- as.data.frame(sapply(seq_len(k), function(j) {
p <- plogis(outer(theta, diff[j] + c(-1.1, 0, 1.1), `-`)) # P(score >= 1, 2, 3)
rowSums(runif(N) < p) # a graded 0..3 response
}))
names(bdi) <- symptoms
totals <- rowSums(bdi)
pair <- which(totals == 15)[c(1, 2)] # two people, identical total of 15
t(bdi[pair, ]) 17 24
sadness 3 3
pessimism 3 3
past_failure 1 1
loss_pleasure 2 2
guilt 0 0
punishment 2 1
self_dislike 0 0
self_critical 1 1
suicidal 0 1
crying 1 0
agitation 0 0
loss_interest 1 0
indecision 1 2
worthlessness 0 1
loss_energy 0 0
sleep 0 0
irritability 0 0
appetite 0 0
concentration 0 0
tiredness 0 0
loss_sex_interest 0 0One person’s distress concentrates in a few symptoms; the other’s spreads thin across many. Same “15,” but different clinical pictures, and a sum score cannot tell them apart. The profile can.
If 4.4 trillion profiles are possible, how many actually occur? Almost none - because depression symptoms do not vary independently; they hang together. Count the distinct profiles:
key <- apply(bdi, 1, paste, collapse = "-")
c(people = nrow(bdi),
distinct_profiles = length(unique(key)),
pct_unique = round(100 * mean(!(duplicated(key) | duplicated(key, fromLast = TRUE))), 1)) people distinct_profiles pct_unique
1200.0 973.0 74.8 Roughly three-quarters have a profile no one else shares, and in the real six-sample analysis it was 79.7%. Even our simple simulation reproduces the effect: of the trillions of mathematically possible patterns, people occupy only a small fraction, what the study called the manifold collapse. The space of possible profiles is huge, but real people use very little of it. That raises a genuine question: are the profiles that never appear merely rare, or are some of them impossible in principle, incoherent given how depression actually works?
A profile can be numerically fine and clinically nonsensical - maximum worthlessness with zero sadness, say. Borrowing a term from Configural Frequency Analysis, we call these antitypes: possible on paper, absent in reality because they violate the internal structure of the construct. To catch them we need a notion of which symptoms are “easy” (commonly endorsed) and which are “hard” (rare), and then flag people who endorse hard symptoms while skipping easy ones. Here is the idea in miniature - a light ordering-violation count you can run:
difficulty <- colMeans(bdi) # higher = more commonly endorsed = "easier"
ovi <- apply(bdi, 1, function(r) # count "harder scored above easier" pairs
sum(outer(seq_along(r), seq_along(r),
function(i, j) difficulty[i] < difficulty[j] & r[i] > r[j])))
c(median_violations = median(ovi), max_violations = max(ovi),
cor_with_total = round(cor(ovi, totals), 2)) # NOT just severitymedian_violations max_violations cor_with_total
10.0 51.0 0.7 Incoherence is its own dimension - only weakly tied to how high the total is. The real study replaces this rough count with a proper Graded Response Model (an IRT model, cousin of the Rasch model) that estimates each item’s difficulty and each person’s severity, then scores every profile’s log-likelihood. The two measures - our Ordering Violation Index (OVI) and the GRM profile log-likelihood - agree strongly, and that agreement is stable across samples:
# The real analysis (needs the mirt package — shown, not run on the book's server)
library(mirt)
fit <- mirt(bdi, 1, itemtype = "graded") # fit the Graded Response Model
theta <- fscores(fit) # each person's estimated severity
ll <- personfit(fit)$logLik # how probable each profile is
cor(ovi, ll) # OVI and GRM-likelihood track each otherAcross all six independent samples, the OVI–likelihood correlation held between −0.73 and −0.83, and the model flagged 13 strong antitypes in a pooled \(n = 1{,}479\). Incoherent profiles are not a fluke of one dataset; they are a stable, measurable property.
Finding structure that a sum score hides is a recurring theme in the authors’ work. In a related vein, Kashdan, McKnight, and colleagues (Kashdan et al. 2009) used latent class analysis on the National Comorbidity Survey to uncover a neglected, risk-prone subtype of social-anxiety disorder - a whole subgroup of people invisible to anyone who only looked at total symptom counts.
A finding earns a place in a paper when it predicts something. In 207 patients tracked across four therapy sessions, we asked whether the coherence of a patient’s intake profile forecast how much they improved, over and above their initial severity. It did: more incoherent intake profiles improved less (\(p = .017\)), controlling for baseline depression. And treatment did not merely lower the total; it reorganized the profile (cognitive and emotional symptoms improved ~20%; somatic symptoms barely moved). The shape of depression changed, not just its size. The information the sum score discarded was clinically meaningful.
Two habits separate research from storytelling: replicate, and try hard to disprove your own finding. We did both here.
Replication. We re-ran the entire pipeline on a completely different scale - the CES-D, 20 different items - in a completely different population (a multigenerational community panel). No shared items, no shared people. Same result: 78.7% unique profiles (BDI: 79.7%), OVI–likelihood \(r = -0.77\). The manifold collapse is a property of depression, not of Beck’s particular questionnaire.
The retraction. Encouraged, we noticed the two scales reproduced nearly the same configural geometry (difficulty spread, profile uniqueness, incoherence) at a structural concordance of about 0.95, despite sharing nothing. We were ready to call it convergent validity: two windows onto the same construct.
Then we ran the control that could undermine the claim. We took an anxiety scale of the same 0–3 format (the STICSA), a different construct, and measured its concordance with the depression scales.
The anxiety scale concorded just as highly as the two depression scales did with each other. If a different construct fits the pattern equally well, the pattern is not about the construct; it is about the shared response format (21-ish items, 0–3, summed). Our claim of convergent validity did not hold up. The concordance is real, but it is a format-general regularity, not evidence that the scales read the same thing. We cut the claim.
This is one of the most important moves in statistics: a discriminant test, deliberately giving your hypothesis a chance to fail by showing something it shouldn’t explain that it explains anyway. We wanted the convergent-validity story. The data said no. In a textbook you usually see only the claims that survived; here you have seen one that did not. That is a normal and healthy part of the work.
The next case study builds a measure from scratch, study by study - and shows what it looks like to preregister a prediction before the data arrive.