Outliers and assumptions: what biases the results, and what to do

Guide

On this page

Almost all the tests on the site – correlation, t, ANOVA, regression – are versions of one model, the linear model, and so they have the same assumptions and the same enemies. Field (2024, chapter 6) calls this the "beast of bias", with two heads: outliers and violations of assumptions. The page explains what each head damages – because not everything is damaged to the same extent: the estimate itself is usually fine, and what breaks is the standard error and with it the confidence interval and the test – orders the assumptions by importance (linearity first, normality last), shows how to identify problems in plots and numbers, and what to do: robust methods, not deletion and not transformation of the outcome.

1. What bias is – and what exactly is damaged

A "biased" judge is one whose summing-up does not match the objective truth; and so in statistics – the parameters we estimate can differ from their true values, and tests can err (Field, 2024, Section 6.3). What matters for this page is that "bias" is not one thing. Field breaks it down according to what we want from the model: if we want to estimate – an unbiased estimator is one that on average, across samples, hits the population value; a consistent estimator converges on it as the sample grows; an efficient estimator has the smallest variance among the alternatives. If we want a confidence interval – what is checked is coverage: a 95% interval should contain the population value in 95% of samples, and if not – it is biased. And if we want to test a hypothesis – an unbiased test keeps the Type I error rate that was set (5%) and the Type II error rate (20%). This distinction is the key to everything that follows: most violations leave the estimate intact and damage the standard error – and with it the confidence interval and the test.

2. Outliers – the first head

An outlier is a score very different from the rest of the data. Field's example (2024, section 6.4) is the first seven reviews of the first edition of his book on Amazon: 2, 5, 4, 5, 5, 5, 5. The mean is 4.43; without the first review – 4.83. One score dragged the mean down by 0.4, and a new buyer would conclude that the book is worse than the population thinks. In a model with a predictor – a very loud concert with relatively short ear ringing – the outlier flattened the regression line: the slope fell from 9.94 to 6.01. But Field's important point is that outliers damage the error much more than the estimate: in the example of the five lecturers (1, 3, 4, 3, 2), replacing 4 with 10 moves the mean from 2.6 to 3.8, but the sum of squared errors from 5.2 to 50.8 – 38.44 of it due to the outlier. The reason: the errors are squared, and a large deviation is multiplied by itself. And the sum of squares is what the standard deviation, the standard error, the confidence interval and the test statistic are computed from. One outlier – a wide confidence interval and a weak test.

Even so, Field is unequivocal: outliers stay in the data, unless there is solid evidence that they are not from the population we are interested in. The fact that a score is extreme is not a reason to delete it. The question to ask is why: if the loud concert ended with short ringing, and it turned out that this person was the only one wearing earplugs – that both explains it and moves the person to another population, and they can be removed, provided it is reported that they were removed and why; and preferably the exclusion criteria are set in advance, in preregistration. And if there is no such explanation – the default is that the outlier reflects natural variability, and we want to keep it. Field adds a warning about "sneaky outliers" (Box 6.5): not an extreme, isolated score, but a small subpopulation with heavy tails hiding inside a distribution that looks like a perfect bell – a contaminated normal distribution (Tukey, 1960, as cited in Field). It inflates the variance estimate and lowers the power of tests, and it is very common.

3. The assumptions – in order of importance, not in the order in which they are taught

An assumption is a condition that ensures that what we are trying to do works properly (Field, 2024, Section 6.5). Statistical procedures are usually presented as separate tests with their own assumptions, but most of them are versions of the linear model estimated by least squares, and so they share a common set of assumptions. Field orders it by importance: (1) linearity and additivity; (2) the expected value of the errors is zero – almost always met; (3) "spherical errors" – equal variance (homoscedasticity) and independence; (4) normality "of something". And his Table 6.1 adds what each assumption buys: linearity is needed for everything; for an unbiased estimate it is enough that the errors are zero on average; for the best estimate – spherical errors as well; for an unbiased standard error – spherical errors and also normality of the errors or a large sample. The confidence interval and the test inherit the assumptions of the standard error.

Linearity and additivity. The assumption that the outcome is related to the predictors in a straight line, and that several predictors act additively. Field (2024, section 6.6) places it above all the others: if it is wrong, the model describes the world incorrectly – even when all the other assumptions hold – and there is no point interpreting the coefficients or worrying about significance. His analogy: calling a cat a dog. You can try to put it in a kennel or throw it a ball, but it will remain a cat. A curved relationship described by a straight line is a wrong model, not a model with a problem.

Homoscedasticity (equal variance). The spread of the errors around the model is the same at all values of the predictor – the cloud of data runs parallel to the line, and does not widen like a funnel (Field, 2024, Section 6.7.1). When the assumption is violated, the coefficient estimates remain unbiased – but the standard errors, confidence intervals and tests are not accurate, and the confidence intervals can be "extremely inaccurate" (Wilcox, 2010, as cited in Field, Section 6.7.2). That is: if you are only estimating – not terrible; if you are testing significance or reporting a confidence interval – it matters, and there is a solution (section 6). In group comparisons this assumption is called homogeneity of variances.

Independence of the errors. The prediction error of one subject is unrelated to that of another. Field's example (2024, section 6.7.3): Paul and Julie, who consult each other on which pictures they saw – the answers, and therefore the errors, are not independent. The assumption is always violated when data are collected from the same entities over time – repeated measures, longitudinal research – because a person's answer at one time point is correlated with their answer at another; and for the site's purposes also when subjects are nested in groups that influence them (families, localities, classrooms). The consequences are as with homoscedasticity – the standard error is wrong – and the solution is a model that takes the dependence into account, not ordinary least squares.

4. Normality – the least important assumption, and the most misunderstood

Field (2024, section 6.8) opens by saying that this assumption is perhaps the most misunderstood thing in statistics, and takes some of the blame upon himself. The common mistake: thinking that "the assumption of normality" means that the data must be normally distributed. No. What needs to be normal is the errors of the model, and above all – the sampling distribution of the estimate: to know the standard error, a formula is used that is correct when the estimates are normally distributed across samples; if not – the standard error, the confidence interval and the test are not accurate. The confusion arises because if the outcome is normal, the errors will be too, and so the outcome serves as a stepchild for the thing that matters.

And the three reasons why this is the least important assumption (Field, 2024, Section 6.8.3): estimation does not require normality – least squares gives an unbiased estimate that minimizes error even when the errors are not normal; normality only makes it the "best". In a large sample normality does not matter – the central limit theorem ensures that the sampling distribution will be normal regardless of the shape of the population. And in a small sample the residuals of the model are checked: if they are normal, you can proceed. How large is "large"? The customary value is 30, but there is no simple answer (Box 6.4): in distributions with light tails even 20 is enough; with heavy tails – in which outliers are common – 100 or even 160 may be needed, and with a lot of skewness and kurtosis – even more. Hence Field's practical conclusion: in a large sample, outliers are a more urgent concern than normality.

5. How to check – plots before numbers, numbers before tests

Outliers are seen in the data; violations of assumptions – in the residuals of the model, and as a first approximation also in the outcome itself (Field, 2024, Section 6.9). Field's example: hygiene scores of 810 attendees of a metal festival, on a scale of 0 to 4, over three days. A scatterplot, a histogram and a boxplot all show one case far from all the rest – a score of 20.02 on the first day, on a scale that ends at 4. Go back to the original record: 2.02, a typing error. That is the first lesson: an outlier is first of all a question about the data point, and only then a statistical question. On the site: From SPSS to Excel warns about values outside the range defined by the codebook, and the frequencies and descriptive statistics calculator shows the minimum, maximum and a chart for each variable – that is where to start.

Standard scores (Box 6.6): convert to z-scores and compare with what is expected in a normal distribution – about 5% above 1.96 in absolute value, 1% above 2.58, and almost none above 3.29. In Field's example: 6.8% above 1.96 (compared with 5%), 2.3% above 2.58 (compared with 1%), 0.8% above 3.29 – "about what we would expect". A boxplot marks outliers and extreme cases with the row number. A Q-Q plot compares the data with the diagonal line of a normal distribution – points on the line: normal; bending upward: positive skew (section 6.9.4-6.9.5). All of these – z-scores, histogram, Q-Q, skewness and kurtosis with their standard errors, and also the two tests in the next paragraph, weighted by sample size – are produced by the normality check calculator, for the whole sample or by group. Skewness and kurtosis: SPSS gives values for which zero is normal; positive skew drags the mean above the median, so a mean larger than the median is a sign of positive skew; positive kurtosis – heavy tails. In the example: day 1 is symmetric (skewness almost zero), days 2-3 have a skew of about 1 – more people with low scores, "people became smellier as the festival progressed".

And significance tests for normality – no. Field (Box 6.7) covers Kolmogorov-Smirnov, Shapiro-Wilk and Levene "mainly because people expect to see them in introductory books, and not because they are a good idea": in a large sample they are significant even for small, unimportant deviations – exactly when normality no longer matters; in a small sample they lack the power to detect violations – exactly when it does matter. In the example, K-S on day 1 (N = 810) is almost significant (p = .097) although the histogram is remarkably normal. The advice: in a large sample – do not use them, and do not worry about normality at all; in a small sample – note whether they are significant, and do not be reassured if they are not. And for all samples: plot. Linearity and homoscedasticity are checked together in one plot – standardized residuals against predicted values (Figure 1): an even cloud – fine; a funnel – heteroscedasticity; an arc – nonlinear. With a single predictor, the ordinary scatterplot is a reasonable approximation.

And missing values, along the way (Field, 2024, Tip 6.1): listwise deletion removes a subject from the whole analysis if they are missing one value; pairwise deletion removes them only from the analyses the missing value concerns – the site's correlation matrix works this way, and reports N for each pair. Replacing a missing value with the variable mean – which SPSS sometimes offers – shrinks the standard deviation and the standard error, because each replaced value has no distance from the mean, and with many missing values it produces "significant" results that are a product of the replacement. In more detail – on the missing values page.

6. What to do – robustness instead of deletion and transformation

The historical route was nonparametric tests, which convert scores to ranks – and lose information – and cover a limited range of designs (Field, 2024, Section 6.10). The best route today, according to Field, is to estimate parameters, standard errors, confidence intervals and p with methods that are not biased in the presence of outliers and violations – robust methods (robust methods):

Trimming (trimming): remove a fixed percentage from each end – 5%, 10%, 20% – and compute a trimmed mean; at the other end of the scale is the median, which is a trimming of everything except the middle (section 6.10.5). Trimming by standard deviations ("remove everything above 2.5 standard deviations") is a bad idea (Box 6.8): the mean and standard deviation themselves are inflated by the outliers, so the outliers bias the criterion that is supposed to deal with them – in Field's example the rule removed no score on a 0-10 scale, because the cutoff fell beyond the limits of the scale. Winsorizing: instead of removing, replace the scores beyond the cutoff with the cutoff value (section 6.10.6). Feels like cheating? Field: if the score does not represent the sample and biases the model, it is better than reporting a biased model. Cheating is not dealing with extremes when they bias in favor of the hypothesis, or changing scores systematically for a reason other than reducing bias. Heteroscedasticity-consistent standard errors (HC3 or HC4): estimate the standard error from the residuals without assuming equal variance (section 6.10.7). Bootstrap: treat the sample as the population, sample from it with replacement 1,000-2,000 times, compute the estimate in each resample, and from the distribution of the estimates derive a confidence interval (percentile) and a standard error – without assuming anything about the shape of the sampling distribution (Efron & Tibshirani, 1993; Field, 2024, Section 6.10.8). In SPSS there is a button for this in most procedures; Field notes that the results differ slightly in each run because the sampling is random – whereas on the site the bootstrap runs with a fixed seed, so the same input always gives the same interval (see intercoder reliability).

And do not transform the outcome (Field, 2024, section 6.10.3 – "don't do it"). A logarithm or square root to "normalize" the outcome variable: normality does not matter in a large sample, heteroscedasticity has robust solutions, and transformation creates new problems. Field quotes Glass et al. (1972): the payoff of normalizing transformations is "low, and they are rarely worth the effort", and the four points of Games (1984): in a sample above about 30 the sampling distribution is normal anyway; transformation changes the hypothesis – a logarithm compares geometric means, not arithmetic ones, and measures a different construct from the one that was measured; it is hard to decide about normality in a small sample; and the "wrong" transformation can do more harm than analyzing the raw data. The one exception: "straightening" a curved relationship – a transformation of the predictor so that the linear model describes the relationship correctly – and that is the linearity assumption, the most important of all.

7. How to report

The handling of outliers and assumptions is part of the method section or the beginning of the results, and it is reported as decisions, not as excuses: which checks were done (plots, standard scores, skewness and kurtosis – not just "the data were normally distributed"); how many cases were removed, by which criterion, whether it was set in advance, and what the result is with and without; which robust method was applied – "confidence intervals were computed with a percentile bootstrap, 2,000 resamples" or "HC3 standard errors" – and why; and if there are repeated measures – which model took the dependence into account. A result that changed substantially with the removal of one case is a finding in itself, and it is reported. Field points out (the footnote to section 6.10.6) that outliers are interesting in their own right: if they do not represent the population, the question of why they differ can be the subject of the next study.

Common mistakes

"The data must be normally distributed." Not the data – the errors, and above all the sampling distribution; and in a large sample not even that. A "non-normal" binary predictor is no problem at all.

K-S on 800 subjects. It will be significant even for an almost perfect distribution, and will send you to fix what is not broken. In a large sample – plots and nothing else.

Deleting an outlier because it "spoils" things. An outlier stays unless it has been shown not to be from the population. The evidence is a reason (earplugs, a typing error), not distance from the mean.

Trimming by standard deviations. The criterion is computed from the mean and standard deviation – both of which are inflated by the very outliers you want to deal with. Trim by percentages.

A logarithm on the outcome "to normalize". It changes the hypothesis and the construct measured, and is usually not worth the effort. Transformation is reserved for straightening the predictor.

Checking normality and forgetting linearity. The assumption above which there is none. An arc in the residual plot – the model is wrong, and no normality correction will help.

Replacing missing values with the mean. Shrinks the standard error and produces artificial significance. Pairwise deletion, or model-based imputation methods – on the missing values page.

An independent-samples t-test on repeated measures. The same people twice – the errors are dependent, the standard error is wrong. A paired-samples test or a repeated-measures model.

Checklist – before running a test

References

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman & Hall. https://doi.org/10.1007/978-1-4899-4541-9

Field, A. (2024). Discovering statistics using IBM SPSS statistics (6th ed.). SAGE.