Design of experiments

What is an experiment, and why design it

An experiment is "a series of runs in which purposeful changes are made" to the input variables of a process, in order to see what changes in the outcome and why (Montgomery, 2020, p. 1). Unlike correlational research, the researcher sets the levels – teaching method, dosage, condition – and assigns them to participants. Therefore an experiment can answer a causal question, provided that the assignment is random: Fisher's insight was that randomization ensures that "no uncontrolled factor will bias" the result of the statistical test (Maxwell et al., 2018, p. 26).

Two common ways of working without a design – the "best guess" (changing a factor according to what happened in the previous run) and "one factor at a time" – waste runs and miss interactions: what works at one level of one factor does not necessarily work at another (Montgomery, 2020, p. 4). The recommendation is to plan in advance, and as a team: state the problem, choose the dependent variable, the factors, levels and ranges, and then the design – and only then run and analyze (ibid., p. 13, Table 1.1). "The method of analysis depends directly on the design" (ibid., p. 11) – and so the page is built of two parts that talk to each other: the run sheet downloaded in the design is loaded back in the analysis.

Three principles: randomization, replication, blocking

Randomization is "the cornerstone": both the assignment of participants to conditions and the order of the runs are determined at random. It justifies the assumption that the errors are independent, and "averages out" extraneous factors – for example, if all the tired participants happened to receive the same condition (Montgomery, 2020, p. 11). Replication – an additional independent run of each combination; it allows estimating the experimental error and makes the means more precise. Repeated measurements of the same unit are not replicates (ibid., p. 12). Blocking – groups of similar units (class, day, operator, batch), within each of which the treatments are compared, in order to remove the differences between them from the error (ibid., pp. 12-13). All three are "part of every experiment" (ibid., p. 13), and sometimes a fourth principle is added – the factorial design (ibid., p. 11).

Which design is suitable?

Independent groups (completely randomized design) – one factor, each participant in one condition. The simplest; the analysis – one-way analysis of variance.

Randomized blocks – when there is a known nuisance factor that can be controlled: every block contains all the treatments, and the order within the block is random. The block is "a restriction on randomization" (Montgomery, 2020, p. 117). Three lessons in each of four classes, for example.

Latin square – two nuisance factors, each at p levels, and p treatments: each treatment once in every row and every column (Montgomery, 2020, p. 133). In psychology – mainly for counterbalancing the order of presentation in a within-subjects design: the rows are participants, the columns the position in the order (Maxwell et al., 2018, pp. 645-648).

Factorial – two or more factors, all combinations. It allows testing interaction, and without interaction – each participant "works" for both factors at once, so the same power is obtained with fewer participants than in two separate experiments (Maxwell et al., 2018, pp. 355-356; Montgomery, 2020, pp. 155-156).

‎2^k‎ and fractions – many factors, each at two levels ("low" and "high"). For screening: which of eight factors has an effect. When all ‎2^k‎ combinations cannot be run, half or a quarter of them are run (a fraction) – at the cost of aliasing between effects (Montgomery, 2020, pp. 274-278).

Designs with a within-subjects factor (repeated measures) – paired-samples t-test or repeated-measures ANOVA; random, nested and split-plot factors – on the page "Random, nested and split-plot factors"; quantitative factors and the optimum point – on the page "Response surface methodology".

Sample size and power

Power is the probability of rejecting the null hypothesis when it is indeed false. When the hypothesis is false, the F statistic follows a noncentral F distribution, with a noncentrality parameter λ that combines the effect size, the variance and the sample size (Maxwell et al., 2018, pp. 144-145). The page computes the power directly from this distribution – the same computation as G*Power and pwr.anova.test in R – and searches for the smallest n that reaches the power you asked for. "The most difficult part of power analysis is usually conceptual, not computational": you need to decide what the effect size is (ibid., p. 145).

Three ways to specify it, on the page and in the book: (1) Cohen's f – the standard deviation of the population means divided by the within-group standard deviation; small .10, medium .25, large .40 (Cohen, 1988, as cited in Maxwell et al., 2018, p. 146). (2) Expected means and standard deviation – from which f is computed; in the book's example, 400, 450 and 500 with σ = 100 give f = .41 and 21 participants per group for power .80 (ibid., pp. 147-149). (3) The difference between the two extreme means, in standard deviation units (δ), with all the others in the middle – Table 3.10 in the book (p. 148); the page reproduces all 96 of its cells.

With a single factor, power depends only on f, n and the number of groups. In a factorial design each effect has its own power, and for an interaction of the same size more participants are usually needed: the main effect relies on marginal means of many participants, and the interaction – on cell means (Maxwell et al., 2018, pp. 354-355). That is why you choose in the settings the effect for which power is computed. With blocks, the error degrees of freedom are smaller – (a − 1)(b − 1) – and this too is in the computation. In a ‎2^k‎ experiment the effect size is the difference between the high and low levels in σ units; as in Montgomery's example – two replicates of ‎2³‎ for an effect of 2σ give power .937 (Montgomery, 2020, pp. 209-210).

Be careful with an effect size from a pilot experiment: it is biased upward and imprecise, and using it to plan a follow-up study "is probably a wild guess" (Maxwell et al., 2015, as cited in Maxwell et al., 2018, p. 149). It is better to compute for several sizes, or by the lower bound of a one-sided confidence interval (ibid., pp. 150-151). Montgomery (2020, pp. 94-95) also suggests another way: choose n so that the confidence interval for the difference between two means has the desired width.

Random assignment in practice

In the design, the page produces a run sheet: for each run (or participant) – a number in the random order, a number in the standard order, the block and the condition, and an empty column for the outcome. The order is determined with R's random number generator (Mersenne-Twister with sample), so that set.seed(seed); sample(N) in R gives exactly the same order; the script the page downloads rebuilds the sheet. It is worth saving the seed with the data.

Complete randomization – all runs in one random order. With blocks – the order is random within each block separately. In a Latin square – a "standard" square (whose first row and column are in alphabetical order) is chosen at random, and then the rows, the columns and the treatment names are shuffled (Maxwell et al., 2018, pp. 647-648; Montgomery, 2020, p. 137). Up to ‎6×6‎ the page chooses from all the standard squares (1, 4, 56 and 9,408; ibid., Table 4.13); from ‎7×7‎ – from the cyclic square. The cyclic square alone (A B C D, B C D A...) "is not a good design" for order of presentation: B always follows A (Maxwell et al., 2018, p. 646). When carryover effects of one condition on the next are a concern, choose a square balanced for order (Williams): each treatment immediately follows every other exactly once; with odd p two squares are needed (ibid., p. 648). An experiment with more participants than p – several squares, each drawn separately (ibid.).

Factorial experiment and interaction

Interaction means that the effect of one factor depends on the level of the other – in the plot, lines that are not parallel. When the interaction is large, the main effects "have little practical meaning", and a significant interaction can mask a main effect (Montgomery, 2020, p. 155). Then examine the effect of one factor at each level of the other (simple effects). With a significant three-way interaction, "usually neither" the two-way interactions nor the main effects are interpreted (Maxwell et al., 2018, p. 422).

Each effect (main, two-way, three-way) is a separate "family", and is tested at a significance level of .05 without correction across families (Maxwell et al., 2018, p. 329). Each cell needs at least two observations for there to be an error estimate; with one observation per cell there is no test for interaction, and it is assumed that there is none – Tukey proposed a one-degree-of-freedom test to check this (Montgomery, 2020, pp. 171-173). The page runs it automatically in a two-way table with one observation per cell, including blocks.

Unequal cells: Type III or II sum of squares

When all cells have the same number of participants, the factors are orthogonal, and the three types of sum of squares are identical. When the cells are unequal – a participant who dropped out, a missing score – they test different hypotheses (Maxwell et al., 2018, p. 370): Type I ignores the other factor and is appropriate only when the cell sizes reflect the population; Type II takes the other factor into account but not the interaction, and is more powerful if there is no interaction in the population; Type III tests the unweighted marginal means, and is easy to interpret even when there is an interaction. "Our general recommendation" – Type III; "some researchers prefer" Type II when there is a theoretical reason to expect no interaction and its test is far from significant (ibid., p. 371). This is "both" – and on this page: Type III by default (as in SPSS), Type II in the settings. The marginal means in the table – the unweighted ones (like emmeans in R), which are what Type III tests. Under no circumstances are participants deleted to equalize the cells (ibid., p. 372).

Blocks and Latin square

The block removes the differences between blocks from the error: in Montgomery's example, the same data, when analyzed without the blocks, double the error mean square (from 7.33 to 15.11), and the test is weakened (Montgomery, 2020, pp. 121-122). The model assumes no interaction between the blocks and the treatments (ibid., p. 126). On this page, a factor marked "block" enters the model as a main effect only, and the experimental factors – with all the interactions between them (as in a factorial experiment in blocks; ibid., pp. 188-190). Latin square – two blocking factors (rows and columns); Graeco-Latin square – three (ibid., pp. 140-142); and also a factorial experiment in which each combination is a letter in the square (ibid., pp. 191-193).

The test for blocks. Montgomery (2020, p. 119) omits the F test for blocks from the ANOVA table: randomization was done only within the blocks, so this is not an exact test of the difference between the blocks – but the ratio MS_blocks/MS_error shows whether the blocks helped. SPSS and R (and also JMP in his book, p. 124) print F and p. "Both": the page shows the ratio and the p, with a note.

Missing values. When an observation is missing, the blocks and the treatments are no longer orthogonal. Instead of estimating the missing value (an approximate analysis that inflates significance), the page performs the exact analysis – a regression model (Montgomery, 2020, pp. 129-132).

Blocks or ANCOVA? When the blocks are built by splitting a continuous variable (age into decades), Maxwell and colleagues prefer analysis of covariance (ANCOVA) with the continuous variable itself – it uses all the information and fewer degrees of freedom; blocks have their place when the nuisance factor is categorical by nature (Maxwell et al., 2018, pp. 502-507). Both: in the design – assignment by blocks (which the page builds) is good in any case; in the analysis – if the block was built from a continuous variable, consider regression with the continuous variable.

Latin square for order of presentation. "The analysis should match the design" – analyzing a Latin square as if it were a design without order wastes what the design provided (Maxwell et al., 2018, p. 648). The page analyzes the participant and the position as blocking factors – Montgomery's additive model (2020, p. 134), with (p − 2)(p − 1) error degrees of freedom in a single square. With several squares – each participant with their own ID (ibid., pp. 137-139).

‏2^k, fractions and aliases

In an experiment in which each factor is at two levels, the effect of a factor is the difference between the mean at the high level and the mean at the low level; for each interaction – an effect in the same way, from its column of signs (Montgomery, 2020, p. 196). When there are replicates, each effect is tested with an F test (one degree of freedom). Without replicates there is no estimate of error. One then relies on the sparsity of effects principle: most systems are dominated by a few main effects and low-order interactions (ibid., p. 218). Two ways, and the page does both: a half-normal plot of the absolute values of the effects – the prominent ones move away from the line (ibid., p. 224); and Lenth's method: a "pseudo standard error" (PSE) from the small effects, and against it a margin for a single effect (ME) and a simultaneous margin for all the effects (SME) (ibid., p. 226). Montgomery recommends using Lenth "as an adjunct to the plot, not a replacement" (ibid., p. 227), and points out that the margins are conservative (ibid., p. 226). After choosing the effects, run a reduced model, and the remaining effects become error (ibid., p. 223) – in the settings, "Reduced model".

A fraction of a ‎2^k‎ runs only some of the combinations, chosen by a "generator": for example in a ‎2⁴⁻¹‎, D = ABC – the column of D is the product of A, B and C. The cost: each effect is "aliased" with another effect, and they cannot be separated – what is estimated is their sum (Montgomery, 2020, pp. 275-276). The resolution summarizes this: in III a main effect is aliased with a two-way interaction; in IV – main effects are clear of two-way interactions, but two-way interactions are aliased with each other; in V – the two-way interactions are clear too (ibid., p. 278). The page uses the generators of Table 8.14 (ibid., p. 299) – the highest resolution, and among those "minimum aberration" (ibid., pp. 297-298) – and shows the defining relation and all the alias chains.

Blocks in a ‎2^k‎. When all the runs do not fit into one block, the high-order interactions are confounded with the blocks: in a ‎2⁴‎ in two blocks – ABCD (Montgomery, 2020, pp. 259-265). The page builds the blocks according to Table 7.9 (ibid., p. 270), and in the analysis marks which effect is confounded with the blocks and is not estimated.

Effect sizes

In a multifactor design you need to decide exactly what is divided by what (Maxwell et al., 2018, pp. 329-335). Partial eta squared – SS_effect/(SS_effect + SS_error) – is suitable when the other factors are "extrinsic" to the question (this factor could have been studied alone); this is what SPSS prints. But it is a description of the sample and is biased upward. Partial omega squared – (SS − df·MS_error)/(SS + (N − df)·MS_error) – estimates the population, and is positive only when F is larger than 1 (ibid., p. 334, equation 44); when it is negative, 0 is shown (ibid., p. 421). In Table 7.8 in the book (p. 335) – .22 versus .17 for the same effect. The page shows both.

How to read the output?

In the design: the design table (number of runs, error degrees of freedom), the power table (for each n – the power for each effect), the power chart, and the run sheet. In a Latin square – the squares themselves; in a ‎2^k‎ – the matrix, the defining relation, the resolution and the aliases. Download the run sheet to Excel, fill in the outcome column, and load it in "Analysis".

In the analysis: the table of means in each cell; the ANOVA table – for each effect SS, degrees of freedom, F, p and two effect sizes; the marginal means with confidence intervals; an interaction plot. In a two-level experiment – also the effects table and the half-normal plot. "Interpreting the result" points to what is important to check in your case – a significant interaction, unequal cells, additivity in blocks.

Assumptions

The assumptions of analysis of variance: independent errors (which randomization justifies), normally distributed, and with equal variance in all cells (Maxwell et al., 2018, pp. 133-141; Montgomery, 2020, pp. 68-76). The F test is robust to departures from normality, and to unequal variances – as long as the cells are equal in size; with unequal cells, different variances "create significant problems" (Maxwell et al., 2018, p. 379). The page runs Levene's test for equality of variances (around the mean, as in SPSS) when there are at least two observations in each cell. In a design with blocks, additivity is also assumed – that there is no interaction between blocks and treatments (Montgomery, 2020, p. 126).

And how do you do it in SPSS?

Analysis: Analyze → General Linear Model → Univariate (the UNIANOVA command). Type III is the default; a model with blocks – Model → Custom, with the block as a main effect only. The page downloads syntax with an appropriate /DESIGN, marginal means (/EMMEANS), partial eta squared and Levene's test. SPSS has no generator for fractional factorial designs and no Lenth – they are on the page and in the R script. The syntax was tested in PSPP (GLM).

And how do you do it in R?

Type III in R requires sum-to-zero contrasts – otherwise car::Anova(type = 3) gives incorrect numbers. The script below was run verbatim on the sample file (the values in the comments). For the design – set.seed and sample give the same order as the page, and for fractions – the FrF2 package.

# Design of experiments – Professor Stat. This file is saved in UTF-8; every comment line starts with an English word
library(readxl)     # reading Excel
library(car)        # Anova of Type III/II
library(emmeans)    # marginal means
options(contrasts = c("contr.sum", "contr.poly"))   # required for Type III

# Design: power and n (three groups, f = .41) and the random order – the same order as on the page with the same seed
pwr::pwr.anova.test(k = 3, f = 0.4082, sig.level = .05, power = .80)   # n = 20.3 – that is, 21 per group
set.seed(2026); sample(24)   # run order of 24 runs: 1 6 13 15 12 4 ...

# Analysis: the "Data" sheet; two missing scores
d <- read_excel("C:/Rdata/DATA_doe_example_eng.xlsx", sheet = "Data")
d <- d[!is.na(d$score), ]   # 46 of 48
d$class <- factor(d$class); d$method <- factor(d$method); d$feedback <- factor(d$feedback)
m <- lm(score ~ class + method * feedback, data = d)   # class – block, main effect only
Anova(m, type = 3)   # method: F(2, 37) = 9.92, p < .001; feedback: F(1, 37) = 0.86, p = .360; interaction: F(2, 37) = 1.75, p = .187
emmeans(m, ~ method)   # 66.8, 79.0, 71.1 (SE 1.92-2.00)

Reporting the results

Report: the design (how many factors, levels, participants in each cell, blocks), how the random assignment was done, and the power analysis that determined the sample size – the assumed effect size, α and the power. In the results – for each effect F(df₁, df₂), p and an effect size, and the type of sum of squares when the cells are unequal; means (and not only tests) – in a table or a figure. A significant interaction is described before the main effects. Do not report "observed power" computed from the results – it is only a translation of p (Maxwell et al., 2018, pp. 151-152).

Common mistakes

Not randomizing – running the conditions by convenience (all of condition A in the morning) and letting an extraneous factor enter the result (Montgomery, 2020, p. 11).

Analyzing blocks as if they did not exist – all the variance of the blocks goes into the error (Montgomery, 2020, pp. 121-122); and conversely – analyzing a Latin square without the rows and columns (Maxwell et al., 2018, p. 648).

Interpreting a main effect when there is an interaction – the mean across the levels of the other factor may not describe any of them (Montgomery, 2020, p. 155).

Type I with unequal cells – the default of aov and anova in R; the result depends on the order of the factors (Maxwell et al., 2018, pp. 370-371).

"Observed power" as evidence that there is no effect – when the result is not significant, it is always below .5 (Maxwell et al., 2018, p. 152).

Computing sample size only for the main effect when the question is the interaction (Maxwell et al., 2018, p. 355).

Forgetting the aliases in a fraction – reporting "the AB interaction is significant" in a ‎2⁴⁻¹‎, when what was actually estimated is AB + CD (Montgomery, 2020, p. 276).

Repeated measurements instead of replicates – three measurements of the same unit are not three replicates, and do not estimate the experimental error (Montgomery, 2020, p. 12).

References

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). https://doi.org/10.1037/0000165-000

Maxwell, S. E., Delaney, H. D., & Kelley, K. (2018). Designing experiments and analyzing data: A model comparison perspective (3rd ed.). Routledge.

Montgomery, D. C. (2020). Design and analysis of experiments (10th ed.). Wiley.

Cohen (1988), Maxwell et al. (2015) and Harris and Quade (1992) – as cited in Maxwell and colleagues (2018). According to the site rule, the reference list includes only sources that were read in the original; citations are by the printed page. Built from the two books in the "Design of experiments" folder (Montgomery – from the deskewed version of the scan, with a full text layer). Site conventions: a random seed and a generator identical to R's; Type III by default; F and p for blocks too, with a note; two digits after the decimal point in tables. The computation is verified against R 4.3.3 (car, emmeans, effectsize, pwr, FrF2, agricolae) and PSPP, and against the numbers printed in the two books: Table 3.10 and the power examples in Maxwell and colleagues, Tables 7.7, 7.8, 7.25, 8.14, 8.15 and 9.12 there; and in Montgomery – Examples 4.1-4.3, 5.1-5.3, 5.6, 6.1, 6.2, 7.1, 7.2, 8.1, Tables 4.13, 7.9, 8.13 and 8.14, and Lenth's method (p. 226).