Multigroup comparison and measurement invariance

Loading the data

Paste columns directly from the sheet (including the header row) – select them in Excel, copy, and paste here:

Model specification

When to use?

Whenever groups are compared on a construct measured by a questionnaire – men and women, cultures, languages of the questionnaire, modes of administration, experimental and control. Kline (2023, p. 393) defines it: measurement invariance concerns the question "whether scores on indicators of theoretical variables have the same meaning or interpretation" under different conditions; without it, "true differences between people cannot be unambiguously separated from differences due to the conditions under which the scores were obtained". Putnick and Bornstein (2016, p. 2) illustrate with depression: if crying and weight gain indicate the severity of depression in women but not in men, a scale score that includes them will show that men are "less depressed" – because they cry less – even though these items are not related to depression in men to begin with. And in the opposite direction, Kline (ibid.) notes that invariance does not mean the groups are equal at the level of the construct: groups can truly differ – and invariance is what allows that difference to be estimated without confounding by the measurement.

Brown (2015, p. 241) divides the questions in two: measurement invariance – the loadings, the intercepts and the residual variances, that is, how the items measure; and population heterogeneity – the variances, covariances and means of the factors themselves, that is, how the groups differ. The page answers both, in this order. In the sample data file: do the three scales (belonging, trust, participation) measure the same thing in the old neighborhood and in the new neighborhood – and if so, how do the neighborhoods differ. Related pages: confirmatory factor analysis (the model in one group), structural equation model, independent-samples t-test (a comparison of observed scale scores – which assumes, without testing, what this page tests).

The logic: the same model, with parts of it equal

The model. In each group g, each item is an intercept plus a loading times the factor plus a residual: x = ν_g + Λ_gξ + δ. From this follow two matrices implied by the model – covariances Σ_g = Λ_gΦ_gΛ_g′ + Θ_g and means μ_g = ν_g + Λ_gκ_g (Vandenberg and Lance, 2000, p. 42; Kline, 2023, p. 394, calls this a MACS model – mean and covariance structure). The invariance models differ only in which parameters are constrained to be equal across the groups:

The names "weak" and "strong" are not uniform: Kline (2023, pp. 396-397), Brown (2015, p. 243) and Putnick and Bornstein (2016, pp. 3-4, following Widaman and Reise, 1997, as cited in Putnick and Bornstein) call equal loadings "weak" and equal intercepts "strong"; Vandenberg and Lance (2000, p. 12, following Horn and McArdle, 1992, as cited in Vandenberg and Lance) call the configural model "weak factorial invariance" and equal loadings "strong". So Brown (ibid.) prefers descriptive names ("equal form, equal factor loadings, equal intercepts"), and so does the page.

What each step means. Equal loading – a one-unit increase in the factor raises the item by the same amount in all groups; Kline (2023, p. 396) notes that rejection can stem from the construct itself being different across cultures, or from response styles (preferring the extremes or the middle of the scale). Equal intercept – participants from different groups with the same level on the factor are expected to get the same score on the item; a different intercept is a constant bias in the item, "differential item functioning" (Brown, 2015, p. 261, Figure 7.4), for example a tendency to agree with every item (Kline, p. 397). Vandenberg and Lance (2000, p. 38) add that a difference in intercepts is not always bias – sometimes it is an expected difference between the groups; this is a substantive question.

Estimation and verification. Maximum likelihood (ML) with a mean structure, like cfa(…, group =, group.equal =) in lavaan: χ² = Σ N_gF_g, Fisher scoring with the expected information until the gradient is almost zero, and equality constraints as a shared parameter. The page reproduces to the printed digit Kline's example (2023, chapter 22) – divergent thinking data of Guo et al. (2021, as cited in Kline) in China and the USA: Table 22.3 (all models, differences and fit indices), Table 22.4 (loadings, residual variances and factors in effects coding), Table 22.5 (implied means) and Table 22.6 (intercepts and factor means); Brown's (2015) Table 7.11 – the six comparisons in a sample of 375 men and 375 women with mood disorders – to within tenths in χ² (the book prints the correlations to three digits) and identical in RMSEA, CFI and TLI; and section 6 of Rosseel (2012): χ², AIC, BIC and Δχ²(12) = 48.251. Against lavaan 0.6-17 – 37 models (including four groups, three methods of setting the scale, releases and score tests), thousands of checks. The reproduction uncovered several misprints in the books: in Kline's Table 22.2 the LM3 mean in China is printed as 1.88, but Tables 22.3 and 22.5 are reproduced only with 1.86; and in the effect-size computation (pp. 406-408) the Americans' mean appears as 1.106 instead of the 1.016 in Table 22.6 – so the d is about .40 and not .33. In Brown, the RMSEA for equal intercepts in Table 7.11 (.046) is inconsistent with the χ² in the same row (.043).

Order of tests and decisions

From the bottom up. Brown (2015, pp. 243-244) recommends starting from the least constrained model and adding constraints: (1) the model in each group separately; (2) configural; (3) loadings; (4) intercepts; (5) residual variances (optional); and if there is substantive interest – (6) factor variances, (7) covariances and (8) means. The reasoning: when a model with all constraints does not fit it is hard to know what the problem is, and "tests of certain aspects of invariance rest on the assumption that other aspects hold". Kline (2023, p. 398) calls this the "free baseline" approach, as opposed to the reverse approach that releases constraints from a full model. Vandenberg and Lance (2000, p. 12) add, before everything, an omnibus test of the equality of the covariance matrices; the page does not compute it, because its rejection "is not informative about the specific source" of the inequality (ibid., p. 36; see also Kline, p. 395, and Brown, p. 244).

When does a step "hold"? There is no consensus here, so the page shows all the indices side by side. The classical test – Δχ² between the two nested models; Vandenberg and Lance (2000, p. 46) recommended staying with it as the main tool. The drawback: with a large sample it also detects negligible differences (Putnick & Bornstein, 2016, pp. 9-10). Hence the move to changes in the indices: ΔCFI ≤ −.01 of Cheung and Rensvold (2002) – the most common; Chen (2007) added ΔRMSEA ≥ .015 and ΔSRMR ≥ .030 (metric) or .015 (scalar and residual); Meade et al. (2008) proposed a stricter cutoff of −.002 in CFI; and Rutkowski and Svetina (2014) – more lenient cutoffs for loadings when comparing 10-20 groups (all as cited in Putnick and Bornstein, 2016, p. 9, and in Kline, 2023, p. 399). The page flags all of them, and in the "Decision" column combines Δχ² and ΔCFI – and when they disagree writes "mixed". Kline (p. 399) is skeptical: "there is little hope" of finding decision rules based on changes in indices "that would apply universally", and it is "poor practice" to ignore the residuals and χ². In his example (pp. 404-405) the Δχ² for the loadings is significant but ΔCFI = .009, and he retains the model after examining the residuals in each group.

In the sample data file (without releases): the configural model fits – χ²(102) = 118.87, p = .121, CFI .992, RMSEA .023; the loadings are equal – Δχ²(9) = 2.87, p = .969; but the intercepts are not – Δχ²(9) = 71.57, p < .001, ΔCFI = −.029.

Setting the scale

A factor has no units of its own, and in a mean structure it also needs an origin (Brown, 2015, p. 256; Kline, 2023, pp. 399-401). Three ways, with exactly the same fit ("it changes nothing in the global fit", Kline, p. 400) and the same effect size d (p. 401):

Marker (the lavaan default): the loading of the first item = 1 in all groups, and the factor mean = 0 in the reference group; in the other groups the mean is estimated once the intercepts are equal – and it is the difference from the reference group (Brown, pp. 256-257, the Mplus method). Reference group (std.lv): in the first group the factor variance is 1 and its mean 0, and in the others they are estimated when the loadings and intercepts are equal (Kline, p. 400) – the differences are obtained in units of the standard deviation of the reference group. Effects coding (Little et al., 2006, as cited in Kline, p. 400): the mean of the loadings of each factor = 1 and the sum of the intercepts = 0, so the factor is on the scale of the items – suitable when all items are on the same scale, such as Likert 1-7.

The reference-item problem. In the marker method the loading of the first item is fixed and equal in all groups, so it "must be assumed" to be invariant – it cannot be tested (Kline, 2023, p. 400). If it happens to be the problematic item, "all the other items on the factor may appear non-invariant" (Putnick & Bornstein, 2016, p. 7; Vandenberg & Lance, 2000, p. 46). Solution on the page: choose effects coding (then any loading can be released), or move another item to the head of the factor. Putnick and Bornstein (ibid.) also warn against fixing the variance to 1 and the mean to 0 in all groups – then the tests of loadings and intercepts are wrong if the groups truly differ in variance and mean; the page, like lavaan, fixes them only in the reference group.

Partial invariance

When the loadings or intercepts are not all equal, the constraint can be released for one or several items, continuing with the rest – partial invariance (Byrne et al., 1989, as cited in Kline, 2023, p. 401). This is common: in the review by Putnick and Bornstein (2016, p. 12) – 32% of the tests. How to find what to release: in the "backward" method, release the constraint with the highest modification index (score test), rerun and repeat, one at a time (Kline, p. 402; Putnick & Bornstein, p. 5); according to Yoon and Kim (2014, as cited in Putnick and Bornstein, p. 14), sequential release has fewer Type I errors than releasing all the problematic constraints at once, and Jung and Yoon (2016, as cited ibid.) found that the method works well with a cutoff of 6.635. On the page – a "Release" button in the score-test table. In the sample data file the score test for the intercept of partic_3 ("I attend residents' meetings") is χ²(1) = 62.86 – far above all the others (partic_1: 12.30); after releasing it the other intercepts are already equal (Δχ²(8) = 7.08, p = .528), and partic_1 also drops below 3.84 – it was "dragged along" by partic_3 on the same factor. This is why one is released at a time. And the substantive explanation: in the new neighborhood the municipality convenes planning meetings, so residents attend them more at the same level of participation (this is how the simulated data were built).

How much may be released? There is no agreed rule. Steenkamp and Baumgartner (1998) proposed that more than half of the items on a factor be invariant, and Vandenberg and Lance (2000) – most of them; Byrne et al. (1989) only required that it "make substantive sense" (all as cited in Putnick and Bornstein, 2016, p. 9, who note that none of the guidelines rests on empirical research). Vandenberg and Lance (2000, p. 38) recommend releasing loadings "only for a minority of the indicators", on as strong a theoretical basis as possible, and when there is cross-validation; Byrne (2016, p. 277) – that there be more than one invariant intercept "beyond the reference variable". And what is released matters: according to Steinmetz (2013, as cited in Putnick & Bornstein, pp. 10, 14) unequal loadings hardly bias a comparison of means, but a single unequal intercept on a factor of 4-6 items led to a spurious "significant" difference in 13% of the samples. And what depends on a release: for an item whose loading was released, the intercept is also released at the scalar step, "because there is no point in testing equal intercepts if the metric of the items differs between the groups", and at the residual step – also the residual variance of items with a released loading or intercept (Putnick & Bornstein, pp. 5-6). The page does this automatically (a checkbox; lavaan does not do it by itself, so the R script includes these releases explicitly).

Comparing the factors: variances, correlations and means

"Once the configural, metric and scalar steps have been passed, the researcher is free to compare group means on the latent factors" (Putnick & Bornstein, 2016, p. 7), and Brown (2015, p. 243) adds that this is possible even when only some of the loadings and intercepts are equal. Vandenberg and Lance (2000, p. 40) list the advantages over a t-test on scale scores: the comparison is justified only if the measurement is equivalent, differences between latent means are "corrected for measurement error", and partial invariance can be controlled. With two groups, the z test of the factor mean (the means-comparison table) and the omnibus test of equal means give almost the same result (Brown, pp. 261, 263: Δχ² = 1.91 versus z = 1.38); with more than two groups – an omnibus test followed by pairwise comparisons, with a correction to the significance level (Vandenberg & Lance, p. 40; Brown, p. 263).

Effect size. Kline (2023, p. 401, Equation 22.2): d = (κ̂₁ − κ̂₂) / √((φ̂₁ + φ̂₂)/2) – the difference in means divided by the square root of the mean of the factor variances; identical across the three methods of setting the scale, and it assumes equal variances and equal groups. When the factor variances differ (test M5), Kline (p. 408) suggests two d values, each based on the standard deviation of one group – the page prints both in the note. Putnick and Bornstein (2016, p. 7) ask that such an effect size be reported in any case. In the sample data file, in the partial scalar model: the new neighborhood is lower in belonging (d = −0.30, p < .001), higher in participation (d = 0.32) and does not differ in trust (d = 0.04, p = .669). In the observed scale scores the difference in participation is larger (d = 0.46) – because partic_3 is inflated in the new neighborhood; a comparison of item means would have exaggerated by about 40%.

Variances and correlations. Equality of factor variances tests whether the groups use the same range of the continuum; such a comparison is meaningful only when the loadings are equal, and a comparison of covariances – only when the variances are also equal (Brown, 2015, p. 262; Vandenberg & Lance, 2000, p. 39). The page tests the variances against the scalar model (M5), the covariances against M5 (M6), and the means against the scalar model (M7) – a site convention; Brown (Table 7.11) tests the means on top of the equal-variances model, and when the variances are equal the difference is negligible.

How to read the output?

Table 1 – the size of each group, the means and standard deviations of the items – the summary data that allow others to reproduce the analysis. Table 2 – the model in each group separately, with the largest residual; start here. Table 3 – the heart: the sequence of models in the structure of Table 3 of Putnick and Bornstein (2016, p. 29): the fit of each model, the model it was compared with, Δχ², Δdf, the changes in CFI, RMSEA and SRMR (negative in CFI and positive in RMSEA and SRMR – worse fit), and a decision. The score-test table (appears when there is something to release) – by how much χ² is expected to drop if one constraint is released, and the EPC: the expected value of the parameter in each group after the release; Kline (2023, p. 402): if the change is negligible, "there may be little harm in retaining the constraint" even when the test is significant. The estimates table – loadings, intercepts and residual variances in each group, in the selected model (the default – the highest step not rejected); = marks a shared parameter. The factors table – variances, means and correlations in each group. The means-comparison table and the chart of d – the latent factor versus the observed scale score. The implied-means table – like Kline's Table 22.5 (p. 407): from M3 the intercepts are shared, and a nonzero mean residual is a local sign of a problematic intercept.

Assumptions and warnings

Sample size in each group. Kline (2023, p. 403) cites the simulations of Meade and Bauer (2007, as cited in Kline): with 400 in each group power was generally high, with 200 – "quite variable", and there is no simple ratio of participants to items that guarantees power. So non-rejection in a small sample is not strong evidence of invariance, and with Δχ² in a large sample – every negligible difference is rejected. Unequal groups: allowed, but balanced groups are preferable – the larger group contributes more to χ², to the indices and to the score tests (Brown, 2015, p. 251). Likert items: the page treats items as continuous (ML), and Kline (p. 409) notes that this "is not ideal" with 5 categories or fewer; in a categorical analysis there are thresholds instead of intercepts, and the order of tests and the identification are different (ibid., pp. 409-410) – in R: ordered = in lavaan. Data-driven release "is subject to capitalization on chance" (Vandenberg & Lance, 2000, p. 37); "with enough tests and releases, any model can be made to fit" (Kline, p. 402) – report every release, and preferably confirm in a new sample. Missing values – listwise deletion in each group (the lavaan default).

What the page does not compute. Ordinal or binary data (thresholds, WLSMV); invariance over time in the same group (Kline, 2023, p. 394); MIMIC – group as a predictor in a single model, which tests only intercepts and means (Brown, 2015, p. 242); the omnibus test of equal matrices; the factor-ratio test of Cheung and Rensvold and the "forward" method (Kline, p. 402); approximate (Bayesian) invariance and alignment for many groups (Kline, p. 411); effect-size indices of non-invariance such as d_MACS (Putnick & Bornstein, 2016, p. 17); robust estimation (MLR) and FIML; cross-loadings.

And how is it done in SPSS and AMOS?

SPSS Statistics has no confirmatory factor analysis; in AMOS groups are defined via Analyze › Manage Groups – a name and a data file for each group (Byrne, 2016, pp. 235-236). The sequence is run under Analyze › Multiple-Group Analysis: AMOS offers ready-made models – "measurement weights" (equal loadings), followed also by "structural covariances" and "measurement residuals" (p. 249). For intercepts and means: under Analysis Properties › Estimation check Estimate means and intercepts, and in the Multiple-Group Analysis window add "measurement intercepts"; AMOS fixes the factor means to 0 in all groups, and the 0 is deleted in one group so that its means are estimated against the reference group (pp. 277-278). Byrne (p. 249) notes that constraining the residual variances "is considered an overly strict test" and is therefore rare.

And how do you do it in R?

In lavaan: group = for the group column, group.equal = for the types of constraints ("loadings", "intercepts", "residuals", "lv.variances", "lv.covariances", "means") and group.partial = for releases (Rosseel, 2012, Section 6); lavTestLRT compares nested models, and lavTestScore gives the score tests for each equality constraint. Note: in lavaan 0.6-17 effect.coding = TRUE ignores group.partial and group.equal = "means" – with effects coding and releases, or when testing equality of means, the constraints have to be written manually, as in Kline's script (2023, chapter 22). The "Download R script" button in the output writes the sequence you ran, with the releases. The script below was run as is on the sample data file (lavaan 0.6-17).

# Group comparison and measurement invariance – Professor Stat. This file is saved as UTF-8; every comment line starts with an English word
library(readxl)   # Reading Excel
library(lavaan)   # cfa, lavTestLRT, lavTestScore

# Loading: the standard layout – row = participant; the sheet "data"
d <- read_excel("C:/Rdata/DATA_invariance_example_eng.xlsx", sheet = "data")
items <- c(paste0("belong_", 1:4), paste0("trust_", 1:4), paste0("partic_", 1:4))
d <- as.data.frame(d[, c("neighborhood", items)])
names(d) <- c("grp", paste0("b", 1:4), paste0("t", 1:4), paste0("p", 1:4))   # Short English names for the model
labs <- c("established", "new")   # The first – the reference group

model <- "
  belong =~ b1 + b2 + b3 + b4
  trust  =~ t1 + t2 + t3 + t4
  partic =~ p1 + p2 + p3 + p4
"
fit <- function(...) cfa(model, data = d, group = "grp", group.label = labs, ...)
m1 <- fit()                                                  # Configural
m2 <- fit(group.equal = "loadings")                          # Metric
m3 <- fit(group.equal = c("loadings", "intercepts"))         # Scalar
lavTestLRT(m1, m2, m3)
# χ²(102) = 118.87; Δχ²(9) = 2.875, p = .969; Δχ²(9) = 71.567, p < .001

# Score tests for the intercepts in the scalar model – what to release
st <- lavTestScore(m3)$uni; pt <- parTable(m3)
st$param <- paste(pt$lhs, pt$op, pt$rhs)[match(st$lhs, pt$plabel)]
head(st[order(-st$X2), c("param", "X2", "p.value")], 3)    # p3 ~1: 62.864; p1 ~1: 12.296; p2 ~1: 5.113

# Partial scalar: the intercept of p3 free in each group
m3p <- fit(group.equal = c("loadings", "intercepts"), group.partial = "p3 ~ 1")
lavTestLRT(m2, m3p)                                          # Δχ²(8) = 7.082, p = .528
pe <- parameterEstimates(m3p)
pe[pe$op == "~1" & pe$lhs %in% c("belong", "trust", "partic") & pe$group == 2, c("lhs", "est", "se", "z", "pvalue")]
# belong -0.299 (SE 0.090, z = -3.311); trust 0.041 (0.097); partic 0.393 (0.111, z = 3.549)
m7 <- fit(group.equal = c("loadings", "intercepts", "means"), group.partial = "p3 ~ 1")
lavTestLRT(m3p, m7)                                          # Omnibus test of the means: Δχ²(3) = 40.868

Reporting the results

Putnick and Bornstein (2016, p. 16) found that in about a fifth of the studies even the degrees of freedom were not reported, and propose a uniform minimum: (1) the sample size in the models, if different from the whole sample; (2) the handling of missing values; (3) the number of groups and the size of each; (4) the fit criteria for the configural model and for the comparisons; and (5) a table of all the models – df, the fit indices, which model was compared with which, Δdf, the differences and the decision – like Table 3 on the page. Kline (2023, pp. 408-409) adds: explain the rationale for every change to the model, check the code three times, know the software's defaults (in Mplus, for example, loadings and intercepts are equal by default), preregister the plan and publish data and code, and describe the residuals in each group. From the page: Table 3, the releases and their rationale, the means-comparison table with d, and the findings paragraph. APA (2020, section 6.44) – define abbreviations (CFA, CFI, RMSEA) at first use; section 6.36 – no zero before the decimal point for p, CFI and RMSEA.

Common mistakes

Comparing means of scale scores without testing. A t-test on the mean of the items assumes – without testing – full scalar invariance. In the sample data file it exaggerates the difference in participation by about 40%.

Comparing standardized loadings across groups. They depend on the variances of each group, and differ even when the unstandardized ones are equal (Kline, 2023, p. 405; Brown, 2015, p. 262) – compare unstandardized loadings.

Comparing factor means when the scalar step is rejected. First release the problematic intercepts (if there is a justification for it), and only then compare.

Releasing many constraints at once, or until it fits. One at a time, with a rationale, and report every release; when most of the items on a factor are not invariant – the factor is not the same factor.

Deciding only by ΔCFI, or only by Δχ². Report both, and examine the residuals in each group (Kline, 2023, pp. 399, 409).

Requiring equal residual variances as a condition. Residual invariance is not needed for comparing factor means (Putnick & Bornstein, 2016, p. 6).

A non-invariant marker. If all the loadings "fail", the problem may be in the reference item – try effects coding or a different marker.

References

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). https://doi.org/10.1037/0000165-000

Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford Press.

Byrne, B. M. (2016). Structural equation modeling with AMOS: Basic concepts, applications, and programming (3rd ed.). Routledge.

Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

Kline, R. B. (2023). Principles and practice of structural equation modeling (5th ed.). Guilford Press.

Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71-90. https://doi.org/10.1016/j.dr.2016.06.004

Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02

Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4-70. https://doi.org/10.1177/109442810031002

Byrne et al. (1989), Chen (2007), Cheung and Rensvold (2002), Jung and Yoon (2016), Meade et al. (2008), Rutkowski and Svetina (2014), Steenkamp and Baumgartner (1998), Steinmetz (2013), Widaman and Reise (1997) and Yoon and Kim (2014) – as cited in Putnick and Bornstein (2016); Byrne et al. (1989), Guo et al. (2021), Little et al. (2006) and Meade and Bauer (2007) – as cited in Kline (2023); Horn and McArdle (1992) – as cited in Vandenberg and Lance (2000). By the site's rule, the reference list includes only sources read in the original. The page numbers of Putnick and Bornstein (2016) – according to the authors' manuscript in PubMed Central. Site conventions: the "Decision" column (Δχ² and ΔCFI together, "mixed" when they disagree), the cutoff for showing score tests, the automatic release of intercepts and residuals following a release, the d of the observed scale score, the shading of mean residuals above .10 standard deviations, the comparison of variances and means against the scalar model, and the sample data file (simulated data, with one deliberately non-invariant item). The computation on the page is verified against the lavaan 0.6-17 package in R and against the printed output of Kline (2023), Brown (2015) and Rosseel (2012).