Missing values: why they are missing, and what to do

Guide

On this page

Missing values occur in almost every serious analysis (Gelman & Hill, 2007, p. 529): someone who refused to answer the income question, someone who skipped a page, someone who did not come to the second wave. The problem is not that they are missing – it is that the person with missing data is not necessarily similar to the person who answered, so every way of handling missing values is an assumption about the reason for the missingness. The page follows chapter 25 of Gelman and Hill: four missingness mechanisms, from the simple to the hard; the methods that discard data and what they risk; the simple methods that keep everyone and what they distort; and the approach they recommend – random imputation from a regression model, and where needed – multiple imputation. At the end: what to do in practice, what the site and SPSS do by themselves, and how to report.

1. Why missing – four mechanisms, from the simple to the hard

To decide what to do with missing values, it helps to know why they are missing. Gelman and Hill (2007, pp. 530-531) distinguish between four mechanisms. Missing completely at random (missing completely at random): the probability of missingness is the same for all units – as if each respondent rolled a die and refused to answer the income question when it came up 6. Only in this case does discarding the missing cases not bias the conclusions. Missing at random (missing at random): most missingness is not completely random, and this is visible in the data themselves – in the New York Social Indicators Survey, nonresponse rates on income differ between ethnic groups. The more general assumption is that the probability of missingness depends only on available information: if gender, ethnicity, education and age were recorded for everyone, income is "missing at random" if nonresponse depends only on them. Then it is permissible to exclude the missing cases – provided the model controls for all the variables that affect the probability of missingness: a model for income will need to include ethnicity, otherwise it will be biased. Missingness that depends on unobserved predictors: if "nervous" people are less likely to answer about income, nervousness predicts income, and nervousness was not measured – income is no longer missing at random; likewise a patient who drops out of a trial because of discomfort that was not measured. Such missingness must be modeled explicitly, or imputed with bias. Missingness that depends on the missing value itself: people with high income are less likely to disclose it; in the extreme case – everyone who earns above a threshold refuses – this is censoring, but the probabilistic case is also hard. It can be modeled, or the problem can be reduced with more predictors that bring it closer to "missing at random": whites and college graduates earn more, and controlling for them corrects somewhat – "but probably only somewhat" – for the high nonresponse rate of high earners.

And the decisive point (Gelman & Hill, 2007, p. 531): it is impossible to prove that data are missing at random. The "lurking variables" on which missingness may depend are unobserved – by definition – and therefore can never be ruled out. One assumes, or checks against other studies that followed up nonrespondents. In practice one tries to include as many predictors as possible in the model so that the assumption is reasonable: assuming that nonresponse on income depends only on gender, ethnicity and education is a strong assumption, but much more reasonable than assuming the probability is constant for everyone.

2. Methods that discard data – and what they risk

Many approaches simplify the problem by discarding data; Gelman and Hill (2007, pp. 531-532) show how this biases, and point out that even without bias – the standard errors grow because the sample is smaller. Complete-case analysis (complete-case) – listwise deletion: exclude every unit that has a missing value on the outcome or on one of the predictors; R and SPSS do this automatically in regression. The two problems: if the units with missing values differ systematically from the complete ones – bias; and if there are many variables in the model – few complete cases will remain, and most of the data will be discarded for the sake of a simple analysis. Available-case analysis (available-case) – pairwise deletion: each aspect is examined on the subsample that has data for it. In the 2001 survey all 1,501 respondents reported education and 16% refused to report income – so education is summarized over everyone and income over 84%. The problem: different analyses are based on different subsamples and are not necessarily consistent with one another; and if the nonrespondents are different – bias as before: 90% of African Americans but only 81% of whites reported income, so the income summary represents a different population from the education summary. Dropping an entire variable because of a high rate of missing values is also available-case analysis – and in a causal context it may drop a variable that the assumptions require. Nonresponse weighting: a model that predicts who responded, with the inverse of the predicted probability as a weight, to restore representativeness to the complete sample – becomes complicated with more than one missing variable, and the standard errors become unstable when the probabilities are close to 0 or 1.

3. Simple methods that keep everyone – and what they distort

Instead of removing, one can fill in – "impute" (impute). The methods preserve the sample size, but produce biases of another kind, and one rule applies to all of them: when a single value is imputed, the standard errors come out too low, because there is a lot of uncertainty about the missing value, and by choosing one value one pretends that it is known with certainty (Gelman & Hill, 2007, p. 532). Mean substitution: the easiest – and it also severely distorts the distribution of the variable, reduces the standard deviation, and "pulls" correlation estimates toward zero. Pallant (2020, chapter 6) says of this option in SPSS: "it should never be used," and Field (2024, tip 6.1) explains why: a substituted value has no distance from the mean, and the shrunken standard error produces "significance" that is an artifact of the substitution. Last value carried forward (last value carried forward): in evaluating interventions, replacing a missing outcome with the baseline measurement. Considered conservative, but it can be the opposite: because of regression to the mean a decline in risky behavior is expected even without the intervention, so the old value looks worse than the truth – and if the missingness rates differ between the groups, the effect estimate is biased in the non-conservative direction. Information from related observations: the father's income from the mother's report – reasonable, but it transfers measurement error, and one should ask whether the reporter has an incentive to distort. A "missing" category for a categorical predictor: simple and often useful. A missingness indicator for a continuous predictor (a "missing" dummy variable + zero or the mean in place of the value): popular in the social sciences, but tends to bias the coefficients of the other predictors, because it forces the same slope on both missingness groups. Logical rules: someone who reported zero months of work – their income from work is zero; such imputation does not rest on strong assumptions, because the missingness mechanism is known.

4. Random imputation from a model – Gelman and Hill's approach

When more than a negligible fraction is missing, Gelman and Hill (2007, section 25.4) prefer to impute formally. The simple case: one variable with missing values – income – and variables observed for everyone. Simple random imputation – sampling from the observed values – ignores all the information in the other questions; it is only a starting point. Regression: fit a model to the observed cases and predict the missing ones. But deterministic imputation – the prediction itself – produces values that are too narrow: in their model R² = .44, meaning the explained standard deviation is 66% of that of the data, so the predicted values vary less than the original ones – and most nonrespondents would look as if their income were in the middle of the scale, mistakenly (p. 536). The correction: random regression imputation – add the prediction error to the prediction, a random value from a normal distribution with the standard deviation of the residuals – and the imputed values spread across the range like the real data (Figure 1).

Three practical remarks from them. Which predictors? Everything that predicts – including consequences of the missing variable: it is strange to model income by receipt of welfare benefits, which is partly a consequence of income, but for imputation this is acceptable – the goal is not causal inference but accurate prediction, and any input that serves it may be used (p. 537). Scale: they model the square root of income, and "topcode" (topcode) high values – not to correct the data but to improve prediction in the range of interest. Zero or positive: when a variable can be zero (did not work) or positive, impute in two stages – a logistic regression for whether it is positive, and a linear regression for the positive value (pp. 537-538). And there is also matching (hot-deck): for each missing unit, find a similar unit in the observed data and take its value – a nonparametric version of regression; their example – imputing risk factors for unsolved cases from the five most similar solved cases.

5. Several missing variables – and multiple imputation

Usually missing values are spread across several variables, and it is not possible to set one variable against complete predictors (Gelman & Hill, 2007, Section 25.5). Two ways: one multivariate model for all the missing variables – in practice an "off-the-shelf" model (multivariate normal) that software runs at the push of a button, "the imputations are only as good as the model, so they need to be checked – but this is a good starting point"; or iterative imputation: impute crudely, then for each missing variable in turn fit a regression on all the others (including the values just imputed), impute randomly, and repeat until convergence. Its advantage – each regression is understood separately and interactions can be included; its disadvantage – the models must be checked for consistency (not imputing age from income and then ignoring age when imputing income), and in any case: look at histograms and scatter plots of the observed values against the imputed ones, to make sure the imputations are reasonable (p. 540).

Multiple imputation (multiple imputation; Section 25.7, p. 542): instead of one imputed value – several (say five), each from a slightly different model, which reflect not only sampling variability but also the uncertainty about the coefficients of the imputation model. Run the usual analysis on each of the five imputed datasets, and combine: the final estimate is the average of the estimates, and the variance combines the variance within the imputations (the mean of the squared standard errors) with the variance between the imputations (how much the estimates differ from one another), with a small correction for the number of imputations. This way the standard error reflects that we did not know the missing values – which a single imputation hides. Pallant (2020, chapter 6) cites Tabachnick and Fidell that multiple imputation is currently "the most respectable method" for dealing with missing data.

6. What to do in practice – a decision guide

The order derived from the chapter, as the site's guide (the decisions – the researcher's): first count – how much is missing in each variable, and in what pattern (the same people? one question?) – and check whether there is a logical rule that imputes without assumptions (zero months of work – zero income; "not applicable" that is not missing). Then ask why: who did not answer, and is it related to the measured variables – a frequency table of "answered / did not answer" against gender, age, education (chi-square) is Gelman and Hill's simple check (exercise 25.1). And then according to the extent: little, scattered missingness – pairwise deletion, with N reported for each analysis; in a regression model with several predictors – remember that listwise deletion excludes anyone missing anything, and include in the model the variables that predict nonresponse. Considerable missingness in a key variable – random regression imputation with all available predictors, or multiple imputation, and report the method. Missingness that depends on the value itself (income, sensitive behaviors) – there is no clean statistical solution: add predictors, report the assumption and test sensitivity – what happens to the conclusion under different imputations. And in any case: not the mean, not last value carried forward, and not a single imputation reported as if the values had been measured.

7. On the site and in SPSS – what happens to missing values without your asking

In the site's standard data template an empty cell is a missing value. The correlation matrix computes each pair on those who have both values (pairwise deletion) and reports N for each pair – so N varies between the cells of the table, and that is information, not a malfunction. Intercoder reliability handles missing values in two deliberately different ways: Cohen's κ on units coded by both coders only, and Krippendorff's α on every value that has a partner in its unit – the difference between the Ns is shown in the table. From SPSS to Excel transfers missing-value codes defined in SPSS (user-missing) as empty cells – and warns when a code such as "Other, please specify" or "Refused" was not defined as missing and will therefore pass as a valid value, as happened in the sample data file in nine variables. In SPSS, Pallant (2020, chapter 6) recommends always clicking Options and checking what is selected, because the default varies between procedures: pairwise deletion unless there is a good reason; listwise only when a subsample that answered everything is wanted; mean substitution – never. She refers those who want imputation and missingness patterns to the Missing Value Analysis procedure.

8. How to report

The missing-data rate for each key variable, and if there is a pattern – who did not answer (the check from section 6); what is assumed about the mechanism and what supports that assumption; the method: "pairwise deletion; N for each analysis is reported in the tables", or "multiple imputation, five data sets, the imputation model included X, Y, Z; estimates and standard errors were pooled following Gelman and Hill (2007, section 25.7)"; and what happens to the conclusion in the complete-case analysis compared with the imputation – if they agree, say so; if not, that is a finding. A value imputed by a logical rule is reported as an imputation (ten respondents with zero months of work – zero income), not as a measurement.

Common mistakes

"The missing values are random" – without checking. One table of answered/did-not-answer against background variables shows whether missingness depends on what was measured. If it does – simple deletion biases the results.

Replacing with the mean. Shrinks the variance and produces artificial significance; Pallant: never.

Dropping everyone missing anything – in a regression with ten predictors. Half the sample is left, and no one knows which half. Count how many remain before interpreting.

Imputing a single value and reporting it as if it were measured. The standard errors are too small by definition. Use multiple imputation, or at least report that values were imputed.

Predicting from a regression and forgetting the error. All the non-respondents end up "in the middle" – an artifact of prediction without scatter. Add the prediction error.

Being afraid to use "outcomes" as predictors for imputation. For imputation the goal is prediction, and anything that predicts is allowed.

Treating a "refused" code as a value. A 99 not defined as missing enters the mean. Check the codebook before anything else.

Forgetting that N changed. Pairwise deletion means a different N for each analysis – report it, and do not copy the N of the full sample.

Checklist – before running with missing values

References

Field, A. (2024). Discovering statistics using IBM SPSS statistics (6th ed.). SAGE.

Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.

Pallant, J. (2020). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (7th ed.). Routledge.