Simple and multiple linear regression

Loading the data

Paste columns directly from the sheet (including the header row) – select them in Excel, copy, and paste here:

Variable selection

When to use?

When you want to predict one quantitative variable – a score, age, time, income – from another variable (simple regression) or from several variables together (multiple regression), and to know how much of its variance they explain and which of them contributes. Gravetter and Wallnau (2017, p. 530) present it as a direct continuation of correlation: a correlation says there is a linear relationship, and regression finds the straight line that describes it and allows Y to be predicted for every value of X; with two or more predictors (p. 544) "several predictors can be combined to obtain a more accurate prediction". Pallant (2020, chapter 13) lists the questions multiple regression answers: how much variance a set of variables explains, which of them is the best predictor, and whether a predictor still predicts when another is controlled for. Field (2024, section 9.2) shows that this is the general "linear model" – the same model behind the t-test and analysis of variance, only with quantitative predictors.

The page computes standard regression – all the predictors enter at once (the Enter method; Pallant, 2020, chapter 13 – "the most common"). Hierarchical regression (blocks by theory, with R² change), which Field (section 9.9.1) prefers when there is prior knowledge, is not on the page; stepwise regression – Pallant "strongly recommends against" it, and Field: "statisticians frown". Interaction between predictors (moderation) and mediation – the Moderation and mediation page; a dichotomous outcome variable (yes/no) – the Logistic regression page; a relationship between two variables without a direction of prediction – the Correlations page; categorical predictors with three or more groups – analysis of variance, or dummy coding, which is not on the page. A multi-item scale score (the Cronbach's alpha and scale score page) is treated as an interval variable per the site convention (the Variables and levels of measurement page); a single Likert item – not.

The logic: the line that minimizes the squared errors

The regression equation Ŷ = bX + a (Gravetter & Wallnau, 2017, Equation 16.5) is the straight line that passes "closest" to all the points: for each participant the vertical distance between their actual value and the line is the error, Y − Ŷ, and the line is chosen so that the sum of the squared errors is the smallest possible – "the method of least squares" (pp. 533-535; Field, 2024, Section 9.2.4). The slope b – how much Y changes for each one-unit increase in X (b = SP/SS_X, Equation 16.2); the constant a – the predicted value when X = 0. In Gravetter and Wallnau's Example 16.1 (p. 535; eight pairs) the result is Ŷ = 2X − 1. With several predictors (pp. 545-546): Ŷ = b₁X₁ + b₂X₂ + a, and each b describes the contribution of the predictor when the others are held constant.

How good is the line? Field (section 9.2.4) compares it with a "null" model: predicting the mean for everyone. The spread around the mean (SS_T) breaks down into what the line explains (SS_M, "regression") and what remains (SS_R, "residual") – and R² = SS_M/SS_T is the proportion of variance explained (Equation 9.17; Gravetter & Wallnau, pp. 540-541: r²SS_Y). Each sum of squares is divided by its degrees of freedom (k for the model, N − k − 1 for the residual), giving F, which tests whether the model is better than the mean (Field, Equations 9.19-9.20; Gravetter & Wallnau, "analysis of regression", pp. 541-543). With a single predictor this is exactly the significance test of the Pearson correlation: F = t² (p. 543). Gravetter and Wallnau's examples are computed on the page and give the numbers in the book: Example 16.5 (pp. 542-543) – SS_regression = 112.01, SS_residual = 43.99, F(1, 6) = 15.28; Example 16.6 (pp. 546-551, SPSS output in Figure 16.8) – R² = .557, F(2, 7) = 4.392, p = .058, b values of 2.552, .672 and .293.

How to read the output?

Pallant (2020, chapter 13) proposes four steps – checking the assumptions, evaluating the model, evaluating each predictor, and using the results – and the tables on the page are arranged accordingly (the assumptions: the "Assumptions, diagnostics and warnings" section and Table 4).

Table 1 – Model Summary. R² – "how much of the variance in the dependent variable is explained by the model"; in Pallant's example .468, "a quite respectable result". Adjusted R² – in a small sample R² is "rather optimistic", and the adjusted value corrects it toward the population estimate (Pallant; Field, 2024, section 9.2.4 – "shrinkage"); Field adds that it corrects for the number of predictors, because every additional predictor raises R² even when its contribution is nil. R – the correlation between the predicted and the observed values (with a single predictor: the Pearson correlation between predictor and outcome; Field, section 9.8.1). The standard error of the estimate – "the standard distance" between the points and the line, in units of the outcome variable (Gravetter & Wallnau, 2017, pp. 538-539, 548-549): small – accurate prediction. Durbin-Watson – see the assumptions section.

Table 2 – ANOVA. Tests "H₀: the multiple R in the population equals zero" (Pallant): p smaller than .05 – the model predicts better than the mean alone. Field (section 9.8.1): "the linear model overall significantly predicts". A significant F still does not say which of the predictors contributes.

Table 3 – Coefficients. Pallant: scan the Sig. column – a predictor with p smaller than .05 "is making a significant unique contribution"; a nonsignificant predictor may be so "due to overlap with other predictors in the model". B – "how much the outcome changes for each one-unit increase in the predictor, with the others held constant" (Field, section 9.11.3 – in the album sales example: every £1,000 of advertising = 85 albums); the constant completes the prediction equation. β – the standardized coefficient, "the number of standard deviations by which the outcome changes for each standard deviation of the predictor"; since it does not depend on units, it compares the importance of the predictors (Gravetter & Wallnau, p. 550 – "a larger b does not mean a better predictor"; Pallant; Field). t and p – test H₀: b = 0 ("a flat line"; Field, section 9.2.5); a confidence interval for B that contains zero means that the population value "could be positive but also negative" (section 9.11.3). The three correlations (Zero-order, Partial, Part in SPSS): zero-order – the simple Pearson correlation; partial – with the other predictors removed from both variables; semipartial (sr, "Part") – Pallant emphasizes it: squared, it is "how much of the total variance is uniquely explained by the predictor, and how much R² would drop without it" (in her example −.40² = 16% and −.30² = 9%, which together make 25% and not 46.8% – the rest is shared variance). Gravetter and Wallnau (pp. 550-551) arrive at the same unique contribution through "R² with both predictors minus r² with the first alone", and show that the t of the predictor is the square root of the F for the addition. VIF – see the assumptions section.

Table 4 – diagnostics of residuals and influential cases and the residual plot – see the "Assumptions, diagnostics and warnings" section. The "Interpreting the result" box summarizes: the effect size, the sample size against the rules of thumb, unique contributions, multicollinearity and the diagnostics.

Effect size

R² is the effect size of the whole model – the proportion of variance explained, which Field (2024, section 9.2.4) calls "a gauge of the substantive size of the fit". The yardstick: Cohen (1988, pp. 413-414) defines for multiple regression f² = .02, .15, .35 for small, medium and large effects, and translates them into R² of .02, .13 and .26 (R of .14, .36 and .51) – thresholds "somewhat higher" than those for a single correlation, because the model usually has several predictors; Field (section 9.5) uses the same three values for sample-size calculations. Cohen notes that the thresholds are a compromise: "small" in clinical psychology may be "large" in sociology. For each predictor separately the effect size is β (standard deviation units) and sr² (the unique variance; Pallant, 2020, chapter 13), and B with its confidence interval is the effect size in the original units – Field (section 9.14) asks that it be reported first of all. In a large sample a tiny R² comes out significant; report R² even when the model is not significant.

Assumptions, diagnostics and warnings

Types of variables. Field (2024, section 9.4.1): the predictors are quantitative or dichotomous categorical, and the outcome is "quantitative, continuous and unbounded". A dichotomous variable (e.g. sex 1/2, group 0/1) enters as a predictor and its B is the predicted difference in means between the groups; a categorical variable with three or more levels is not a number – dummy coding is required (k − 1 0/1 variables), and the page warns about this. An ordinal variable (education 1-4) in practice enters as a number, assuming that every step is equal – common, but it calls for caution in interpretation.

Sample size. Pallant (2020, chapter 13) gives two rules: Stevens – "about 15 participants per predictor", and Tabachnick and Fidell – N > 50 + 8m (m – the number of predictors; five predictors – at least 90 participants), and more when the outcome is skewed. Field (section 9.5) argues that rules of "10 or 15 per predictor" oversimplify "to the point of uselessness": what is needed depends on the effect size – for a large effect 77 participants suffice even for 20 predictors, for a medium effect 160 (and with up to six predictors about 100), and for a small effect "don't bother without hundreds of participants"; and the R expected for random data is k/(N − 1) – six predictors on 21 participants give R = 0.3 out of nothing. The interpretation box checks both of Pallant's rules and prints the expected random R.

Multicollinearity. Predictors that are very highly correlated with each other (Pallant: r of .7 and above; Field, section 9.9.3: .8-.9) "make it difficult for the regression to separate the unique contribution of each", inflate the standard errors, and can turn important predictors nonsignificant. The table prints VIF for each predictor (tolerance = 1/VIF): Pallant and Field – VIF above 10 or tolerance below .10 are "a serious problem" (Bowerman & O'Connell, 1990; Myers, 1990); Field adds: tolerance below .2 – "a potential problem" (Menard, 1995), and an average VIF substantially greater than 1 – possible bias. Pallant warns that the thresholds "still allow for quite high correlations" and suggests checking the correlation matrix as well. A scale score together with its items, or a total together with its subscales, constitute "singularity" – the page stops with an error.

Four assumptions about the residuals (Pallant, following Tabachnick & Fidell; Field, section 9.4.1, in order of importance): linearity and additivity – the relationship is straight, and the predictors add up; a violation "makes the model invalid" (Field). Independence of the residuals – a violation spoils the standard errors, confidence intervals and tests; Durbin-Watson (Table 1) tests the correlation between adjacent residuals: 2 – none; "a very conservative rule of thumb" – below 1 or above 3 is a concern; and it is meaningful only when the order of the participants has meaning (time, location), not in a file whose order is arbitrary. Equality of variances (homoscedasticity) – the spread of the residuals is equal across the predicted values. Normality of the residuals – important in a small sample; in a large sample the central limit theorem covers it (Field), and the predictors themselves "do not need to be normally distributed". The last three are checked in the residual plot (*ZRESID against *ZPRED, Figure 2 in the output): Pallant – one hopes for "a roughly rectangular distribution, with most of the scores concentrated in the center along the zero point"; Field (section 9.11.7) – "a random array of dots"; a funnel – heteroscedasticity, a curve – nonlinearity. The histogram and the P-P plot of the residuals, from which normality is checked, are not on the page (in SPSS from the syntax; on the Outliers and assumptions page – for the variables themselves).

Outliers and influential cases (Table 4). Field (section 9.3.1) distinguishes between an outlier – a participant the model predicts poorly – and an influential case – a participant without whom the model would be different (Box 9.1). The standardized residuals (ZRESID, z-scores of the errors): in an ordinary sample about 5% lie beyond ±1.96 and 1% beyond ±2.58; more than that – "the level of error within the model may be unacceptable"; a value beyond ±3.29 – "cause for concern" (Pallant: 3.3, and in a large sample "it is not uncommon to find a number"). In Field's album example (section 9.11.6; 200 participants): 12 above 2 (6%) and 2 above 2.5 (1%) – "as expected". Cook's distance – the influence on the model as a whole: above 1 "may be cause for concern" (Cook & Weisberg, 1982, as cited in Field, section 9.3.2; Pallant: in her example the maximum is .094). The leverage: its mean is (k + 1)/N, and cases above twice the mean are checked (Hoaglin & Welsch, 1978, as cited in Field). And what to do? Field (section 9.3.3, following Belsey et al., 1980): diagnostics are meant for understanding the fit, "not a way of justifying the removal of data points to effect some desirable change in the regression parameters"; an outlier that is not influential "is not a cause for concern", but it is worth understanding why it did not fit. Pallant: a case with a Cook's distance above 1 – check it, "you may need to consider removing" it. Mahalanobis distance (Pallant, Table 13.1 – critical values by the number of predictors) and DFBeta (Field) are not on the page, but the syntax saves them to the file.

Missing values. A participant who is missing a value on one of the selected variables is not included at all – listwise deletion (/MISSING LISTWISE), the SPSS default, which Field (section 9.10.5) asks to keep, "because pairwise deletion can produce absurdities such as an R² that is negative or greater than 1"; Pallant (chapter 13) selects Exclude cases pairwise in the dialog, so the numbers in her example may differ slightly from what the page gives on the same data. Hence N in the model may be smaller than the number of rows (the page reports this). What to do about missing values – the Missing values page.

Prediction outside the range. Gravetter and Wallnau (2017, p. 536): the equation is valid only within the range of the observed predictor values – "there is no information about the relationship outside the range"; the interpretation box prints this range in simple regression. Causality. Regression describes prediction, not cause: "predictor" and "outcome" are roles in the model (Field, section 9.4.1 – unmeasured "external variables" can explain the relationship).

What this page does not compute. Hierarchical regression and R² change, stepwise regression, automatic dummy coding, interactions, Mahalanobis distance, DFBeta and DFFit, the histogram and P-P plot of the residuals, partial plots, bootstrap confidence intervals and robust regression (Field, Section 9.12), Bayesian regression (section 9.13), cross-validation (section 9.4.2).

And how do you do it in SPSS?

The procedure (Pallant, 2020, Chapter 13; Field, 2024, Sections 9.7 and 9.10): Analyze › Regression › Linear… – the outcome variable to Dependent, the predictors to Independent(s), Method: Enter; Statistics… – Estimates, Confidence intervals, Model fit, Descriptives, Part and partial correlations, Collinearity diagnostics, Durbin-Watson, Casewise diagnostics (Pallant: outliers beyond 3 standard deviations; Field lowers it to 2); Plots… – *ZRESID to Y and *ZPRED to X, and Normal probability plot; Save… – Mahalanobis and Cook's; Options… – missing values: listwise (Field) or pairwise (Pallant). The syntax the page generates (with the listwise default):

REGRESSION
  /MISSING LISTWISE
  /STATISTICS COEFF OUTS CI(95) R ANOVA COLLIN TOL ZPP
  /CRITERIA=PIN(.05) POUT(.10)
  /NOORIGIN
  /DEPENDENT attitude_score
  /METHOD=ENTER age education sex group
  /SCATTERPLOT=(*ZRESID ,*ZPRED)
  /RESIDUALS DURBIN NORMPROB(ZRESID)
  /CASEWISE PLOT(ZRESID) OUTLIERS(3)
  /SAVE MAHAL COOK.

Reading the output. Model Summary = Table 1 (R, R Square, Adjusted R Square, Std. Error of the Estimate, Durbin-Watson); ANOVA = Table 2 (Regression, Residual, Total); Coefficients = Table 3 – B, Std. Error, Beta, t, Sig., the confidence interval limits, Correlations: Zero-order, Partial, Part, and Collinearity Statistics: Tolerance, VIF; Collinearity Diagnostics (eigenvalues and condition indices; Field, section 9.11.5) – not on the page; Casewise Diagnostics – the participants with |Std. Residual| > 3 (the page lists those above 1.96 in the note to Table 4); Residuals Statistics – the minimum and maximum of Std. Residual, Cook's Distance and Centered Leverage Value (= the leverage in Table 4 minus 1/N); the Scatterplot and the Normal P-P Plot at the end of the output. The "Download SPSS syntax" button in the output generates the loading, the dictionary (if there was a codebook sheet) and the REGRESSION command on the selected variables.

And how do you do it in R?

In base R: lm() fits the model, summary() gives the coefficients, R², adjusted R², the standard error of the estimate and F (Tables 1-3), confint() the confidence intervals, anova() the ANOVA table; rstandard() gives internally standardized residuals (slightly different from the SPSS ZRESID – see the script), cooks.distance() and hatvalues() Cook's distance and the leverage; vif() and durbinWatsonTest() from the car package. β and sr are computed by hand. The script was run as is on the sample data file and gave the table values (in the comments). The site rule for R: a UTF-8 file, every comment line starts with an English word, and a UTF-8 locale for Hebrew names (on Windows: R 4.2 or later, or Sys.setlocale('LC_ALL', 'Hebrew')).

# Linear regression – Professor Stat. This file is saved in UTF-8; every comment line starts with an English word
library(readxl)   # Reading Excel
library(car)      # vif, durbinWatsonTest

# Loading: the standard template – row = participant, column = variable; the sheet "data" (data)
d <- read_excel("C:/Rdata/DATA_regression_example_eng.xlsx", sheet = "data")
v <- c("attitude_score", "age", "education", "sex", "group")
d <- d[complete.cases(d[, v]), v]                       # Listwise deletion (113 of 120)
names(d) <- c("y", "age", "edu", "sex", "grp")           # Short names for the formula

# Tables 1-3 – the model, R squared, F, the coefficients, the confidence intervals
m <- lm(y ~ age + edu + sex + grp, data = d)
summary(m)                                               # R² = .138, adjusted R² = .107, standard error 0.713, F(4, 108) = 4.337, p = .003; B: constant 3.064, age 0.012 (p = .028), education 0.022, sex 0.197, group −0.406 (p = .003)
confint(m)                                               # Confidence intervals 95%: age [0.001, 0.022], group [−0.673, −0.138]
anova(m)                                                 # Residual: SS 54.915, df 108, MS 0.509; regression = 3.376 + 0.003 + 0.847 + 4.594 = 8.820 (Table 2)
# Note: anova(m) splits the regression by predictor (Type I); the SPSS Regression row = the sum of the four predictor rows

# Standardized beta and semipartial correlation (sr) – by hand
b <- coef(m)[-1]; sds <- sapply(d[, c("age", "edu", "sex", "grp")], sd)
b * sds / sd(d$y)                                        # β: age .200, education .030, sex .130, group −.270
t <- summary(m)$coefficients[-1, "t value"]; r2 <- summary(m)$r.squared; dfe <- df.residual(m)
t * sqrt((1 - r2) / dfe)                                 # sr: age .199, education .030, sex .129, group −.268

# Multicollinearity and independence of the residuals
vif(m)                                                   # VIF: 1.013, 1.006, 1.012, 1.012 – no multicollinearity
durbinWatsonTest(m)                                      # Durbin-Watson 1.956

# Table 4 – diagnostics: SPSS ZRESID = residual/standard error of the estimate; rstandard() = internally standardized residual (also divided by √(1 − h))
z <- resid(m) / sigma(m)
c(over196 = sum(abs(z) > 1.96), over258 = sum(abs(z) > 2.58), over329 = sum(abs(z) > 3.29), maxZ = max(abs(z)))   # 4 above 1.96, 2 above 2.58, 0 above 3.29; the maximum 3.14
c(maxCook = max(cooks.distance(m)), maxLeverage = max(hatvalues(m)), avgLeverage = mean(hatvalues(m)))             # Maximum Cook's distance .129; maximum leverage .114, mean .044 = (k + 1)/N

# The residual plot: ZRESID against ZPRED
plot(scale(fitted(m)), z, xlab = "ZPRED", ylab = "ZRESID"); abline(h = c(0, -1.96, 1.96), lty = c(1, 2, 2))

Reporting the results

Report the type of analysis (standard/hierarchical), N, and then the model – R² (or adjusted), F with both degrees of freedom, exact p – and then each predictor: B with its standard error or confidence interval, β, p, and according to Pallant also sr (Pallant, 2020, Chapter 13; Field, 2024, Section 9.14; American Psychological Association [APA], 2020, Section 6.43). The APA example (section 6.43): "High school GPA predicted college mathematics performance, R² = .12, F(1, 148) = 20.18, p < .001, 95% CI [.02, .22]". Pallant's paragraph: "Hierarchical multiple regression was used to assess the ability of two control measures to predict levels of stress, after controlling for the influence of social desirability and age. Preliminary analyses were conducted to ensure no violation of the assumptions of normality, linearity, multicollinearity and homoscedasticity. … The total variance explained by the model as a whole was 47%, F(4, 421) = 94.78, p < .001. … In the final model, only the two control measures were statistically significant, with the Mastery scale recording a higher semipartial correlation value (sr = −.36, p < .001) than the PCOISS (sr = −.26, p < .001)." Field (section 9.14): with several predictors "you can't beat a summary table" – B, standard error, confidence interval, p and β for each predictor including the constant ("so that readers can reconstruct the full model"), and R² below the table; APA (2020, Tables 7.15-7.18) gives sample tables with the same structure, without confidence intervals, with confidence intervals in brackets or in separate columns, and for hierarchical regression.

Format rules (APA, 2020, Section 6.36): B, F, t and the confidence interval for B with a zero before the decimal point; R², β, r, sr and p without one (they cannot exceed 1); a confidence interval in the format 95% CI [LL, UL]. The report sentences in the output are built this way, in English, with the prediction equation in simple regression and with the significant predictors (ordered by the size of β) in multiple regression.

Common mistakes

Interpreting B as a measure of importance. B depends on the units; compare predictors by β or by sr² (Gravetter & Wallnau, 2017, p. 550; Pallant, 2020, Chapter 13).

Summing the sr² values and expecting R². The sum is smaller than R² – the difference is the variance the predictors share (Pallant: 25% versus 46.8%).

A multi-category predictor as a number. "Education 1-4" as a number assumes equal steps; "region 1-5" as a number is nonsense – dummy coding is required (Field, 2024, Section 9.4.1).

"The model is significant, so all the predictors are important" – or the reverse. F speaks about the model as a whole; a predictor can be nonsignificant just because of overlap with others (Pallant; Field, Section 9.9.3).

Deleting participants to improve p. Diagnostics are for checking the model, not "a way of justifying removal" (Belsey et al., 1980, as cited in Field, section 9.3.3).

Stepwise regression. It is affected by chance and often "does not replicate"; Pallant (chapter 13) and Field (section 9.9.1) warn against it – enter predictors according to theory.

Predicting outside the range and talking about causality. The equation is valid within the observed range (Gravetter & Wallnau, p. 536), and a "predictor" is not a "cause".

Ignoring sample size. Six predictors on 21 participants give R = 0.3 even from noise (Field, Section 9.5); adjusted R² hints at how inflated the fit is.

References

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). https://doi.org/10.1037/0000165-000

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Field, A. (2024). Discovering statistics using IBM SPSS statistics (6th ed.). SAGE.

Gravetter, F. J., & Wallnau, L. B. (2017). Statistics for the behavioral sciences (10th ed.). Cengage Learning.

Pallant, J. (2020). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (7th ed.). Routledge.

Tabachnick and Fidell (2013), Stevens (1996, 2002), Cook and Weisberg (1982), Hoaglin and Welsch (1978), Belsey, Kuh and Welsch (1980), Bowerman and O'Connell (1990), Myers (1990), Menard (1995), Durbin and Watson (1951), Miles and Shevlin (2001) and Studenmund and Cassidy (1987) – as cited in Field (2024) and in Pallant (2020). According to the site rule, the reference list includes only sources that were read in the original. Field (2024) and Pallant (2020) in a reflowable digital copy – cited by section and chapter; Gravetter and Wallnau (2017), Cohen (1988) and APA (2020) – by page and section of the print edition.