Reliability and validity: the concepts

Guide

On this page

Every measurement in social research – a questionnaire, an attitude scale, coding of open-ended answers, ratings by judges – is a number that represents something that cannot be measured directly. Two questions are asked about every such number, and they differ from each other: is the measurement consistent – will it give the same result under identical conditions? That is reliability. And does it measure what it was intended to measure, and not something else? That is validity. The page defines the two concepts, breaks each into its types, explains why reliability is a necessary but not sufficient condition for validity, and points to the site's calculators that check the part that can be computed. Version 3 added two chapters from the Cambridge handbook of research methods in the social and behavioral sciences (Nichols & Edlund, 2024): Raykov (2024) on reliability – the formal definition, what alpha is and what it is not, and two warnings about "α if item deleted" and about standardized α; and Skowronski (2024) on validity – why it is a continuum and not yes/no, where invalidity comes from, and why it changes with context.

1. Measurement error – why both concepts are needed at all

Some things are measured directly – weight, height, profit – and some are measured indirectly: self-report, questionnaires, computerized tasks. In both cases there is almost always a gap between the number that represents the thing and its true value – the value we would obtain if we could measure it directly – and that is measurement error (Field, 2024, Section 1.5.3). Field's example: someone who weighs 83 kg steps on a scale that shows 80 – a measurement error of 3 kg. A calibrated scale errs a little; a questionnaire errs much more, because the answer is also influenced by factors that are not the thing being measured – for example, would you admit in a questionnaire that you shoplifted?

Measurement error breaks down into two parts that behave differently. A random part – noise that varies from measurement to measurement and cancels out on average: mood, fatigue, wording understood in two ways. And a systematic part – a bias that recurs in the same direction in every measurement: a scale that always shows 3 kg less, a "satisfaction with the lecturer" questionnaire that actually measures how easy it is to get a high grade. Reliability deals with the first part; validity – with the second. And so it is possible to be highly reliable and not valid: the broken scale gives the same number every morning.

2. Reliability – "the same result under identical conditions"

Reliability is the ability of the measuring instrument to yield the same results under the same conditions (Field, 2024, Section 1.5.4) – or, in Pallant's (2020, chapter 1) wording, the extent to which the scale is free of random error. Reliability cannot be seen in a single measurement; it is seen only when the measurement is repeated and compared. The question is what is repeated – and each answer has its own type of reliability, its own measure, and its own place on the site (Table 1).

The formal definition. Raykov (2024) starts from the identity of classical test theory: every observed score is a true score plus error, X = T + E – not a model that can be falsified but a decomposition that is always true, with the true score being the expected value of the measurement. Reliability is the ratio ρ = Var(T) / Var(X): the proportion of true differences between people out of the observed differences. Hence three readings of the same number: ρ is the R² of "predicting" the observed score from the true one; ρ is 1 minus the noise-to-signal ratio; and ρ is the correlation between two parallel measurements of the same thing, when their errors are uncorrelated – and this is the basis for the "repetition" interpretation at the start of the section, and for test-retest reliability in Table 1, which assumes exactly this (parallel measurements with uncorrelated errors – an assumption Raykov remarks is very hard to construct in practice, and so he prefers multi-item scales). And another result that connects this section to the correlations page: the observed correlation between two measures is smaller than the correlation between their true scores, and r_true = r_observed / √(ρ_U·ρ_V) – the attenuation formula: two scales with reliability .7 that correlate .40 measure things that correlate about .57. Finally, ρ is defined for any measurement with positive variance – a binary item, a Likert item, a continuous score – and depends both on the instrument and on the population, just like a mean and a variance (Raykov, 2024).

Test-retest reliability (stability over time). The same people, the same scale, two time points; the correlation between the two scale scores is the measure (test-retest reliability). This is the simplest way to check reliability, but it suits only what is supposed to remain stable: a personality trait – yes; mood, blood sugar level, output – no, because real change between the time points will look like unreliability (Field, 2024, Section 1.5.4; Pallant, 2020, Chapter 1). The time interval is a decision that is reported: too short – memory of the answers; too long – real change.

Internal consistency. A scale of several items is supposed to measure one trait, so the items are supposed to "go together" – someone high on one item is high on the others too. The common measure is Cronbach's alpha, which reflects the average correlation among the items; the accepted rule is alpha above .7 (DeVellis, 2012, and Nunnally, 1978, as cited in Pallant, 2020, chapters 1 and 9). But alpha depends on the number of items: in a short scale (fewer than ten items) it is low even when the items are well related, and then the mean inter-item correlation is reported, whose desirable range is .2 to .4 (Briggs & Cheek, 1986, as cited in Pallant, 2020, chapter 9). And conversely: a very long scale will reach a high alpha even with mediocre items – a high alpha is not proof that the scale measures one thing. Two practical notes from Pallant (2020, chapter 9): negatively worded items must be reversed before the computation, otherwise alpha will be low and wrong; and reliability depends on the sample – compute and report it on your sample, not only the value reported by the scale's developer.

What alpha is – and what it is not. Raykov (2024) clarifies its status: alpha (Cronbach, 1951, as cited in Raykov) is defined in any population in which the scale score has positive variance – without a model, without an assumption of unidimensionality, without an assumption of normality, and with binary, ordinal or continuous items. Its relationship to the scale's true reliability is complicated: when the item errors are uncorrelated, alpha does not exceed the reliability (hence its nickname "lower bound"), and equals it exactly only when all the items measure the same true score in the same units (tau-equivalence); when the loadings differ but are all high, the difference is negligible in practice; and when the errors are correlated, alpha can even be above the reliability. Hence his conclusion: the calls that have been heard to abandon alpha in favor of other measures are, in his view, mistaken – alpha remains a reliable and useful measure when the errors are uncorrelated, and an informative lower bound in all other cases. And two warnings that bear directly on the output of the alpha calculator: (1) "α if item deleted" – Raykov generally recommends not using it as a guide for changing a scale, because in a scale in which alpha is lower than the reliability, deleting an item by this statistic can lower both the reliability and the criterion validity; (2) standardized α (on standardized items) – avoid estimating, reporting or interpreting it unless there is a strong reason to standardize the items in the first place, because it can be much higher or lower than the reliability of the scale score actually analyzed. The site holds both: the calculator shows both, like SPSS and like Field (2024) and Pallant (2020) on which it is built – but as a diagnostic (an unmarked reversed item is revealed there immediately), not as a basis for deleting items and not as a number to report; report the ordinary α, of the scale score as it is summed. Finally, Raykov adds why "sample-dependent" is also "time-dependent": populations change, the latent structure of a scale can change, and a population that was homogeneous can become several subpopulations – so do not rely on the reliability in the scale's manual, but collect data and re-estimate, preferably with a confidence interval; and in scale development – a large sample split in two, so that the confirmatory part is not tested on the same data on which the changes were made (Raykov, 2024).

Intercoder (or interrater) reliability. When a person is the measuring instrument – a coder sorting open-ended answers into categories, a judge giving ratings – the noise is their interpretation, and the way to check is to set a second, independent coder against them, on the same units and the same instructions. Krippendorff (2019, section 11.2.1) distinguishes between three test designs by the source of disagreement: stability – the same coder repeats the work after some time (individual inconsistency only; the weakest design, because it does not reveal a consistent bias of the coder); reproducibility – two or more independent coders (also differences of interpretation between them; this is what is usually called intercoder reliability, and it is the standard); and accuracy – comparison with a standard considered correct (also deviation from the standard; the strongest design, but such a standard exists mainly in coder training). The measures – Cohen's kappa for a pair of coders (Cohen, 1960) and Krippendorff's alpha for any number of coders – correct the raw agreement for the agreement expected by chance, and are computed on the intercoder reliability page, including confidence intervals and the two accepted interpretation benchmarks.

3. Validity – "measures what was intended"

Validity is the question of whether the instrument measures what it was built to measure (Field, 2024, Section 1.5.4): a "lecturer helpfulness" scale – does it really measure helpfulness? A device that is supposed to measure sperm motility and actually measures sperm count is not valid. Field adds an important distinction: reaction time and skin conductance are valid in the narrow sense – reaction time really measures the time it took to react – but as soon as they are used to infer something else (skin conductance as a measure of anxiety), they are valid only if no factor other than the one we are interested in affects them. Unlike reliability, validity has no single, unequivocal measure; validating a scale is the gathering of empirical evidence about its use (Pallant, 2020, Chapter 1). The literature distinguishes several types of evidence (Table 2).

Validity is a continuum, not yes/no. Skowronski (2024) breaks "measures what was intended" into three properties: accuracy – the value obtained reproduces the true value; precision – the inaccuracies are small; and lack of bias – the inaccuracies are split evenly between overestimation and underestimation. Measures differ on each, and so validity is a degree, not a binary property – and hence even a flawed measure can be useful: a scale that always shows 5 kg less still orders people correctly and still detects weight loss after treatment; but the same scale will be wrong when the value itself is needed, and in anesthesia calculated by weight – it will endanger. Where invalidity comes from. Four sources that weaken the causal link between the property and the reading: the instrument (a defect, wear), the user (carelessness, lack of ability, lack of training – a coder who was not instructed), the conditions (noise, disturbance), and the measured object itself – and when the measured object is a person, the answer to "I hate Klingons" also reflects the wish to please the researcher, to spoil things, not to seem opinionated, to seem dangerous, or to conform to the prevailing view (Skowronski, 2024; his example about aliens from "Star Trek", so that no real prejudice interferes with it). Direct and indirect, tangible and hypothetical. Line length, salivary cortisol, time – are measured directly; narcissism, stress, stability of self-concept – are hypothetical constructs. And a measure can be indirect: one thing is measured (cortisol, reaction time, word-recognition errors) and something else is inferred from it (stress, ease of processing, what the respondent thought) – like carbon-14 dating, which measures carbon and infers age. An indirect measure requires validation at two levels: that the measurement itself is accurate, and that the meaning attributed to it is correct – reaction time can lengthen out of concern for one's impression and shorten out of boredom, and so be completely accurate and invalid as a measure of what was wanted. The more hypothetical the construct and the more indirect the measure, the more validation work is required – and this, in his view, is an ongoing activity at the center of every research program, not a stage that ends.

Content validity. The extent to which the items represent the concept and cover all its aspects (Field, 2024, Section 1.5.4) – the extent to which the scale has adequately "sampled" from the content domain it is supposed to cover (Pallant, 2020, Chapter 1). A "job satisfaction" scale all of whose items are about pay does not cover the domain. This is evidence of judgment – experts, theory – not of a number.

Criterion validity. Comparing the scores with an external criterion that can be measured: students' ratings of the lecturer's helpfulness against an objective observation of his helpfulness. When the criterion is measured at the same time – concurrent validity; when the scale predicts a later observation – predictive validity (Field, 2024, Section 1.5.4). Field warns that both checks are often impractical – an objective criterion that is easy to measure simply does not exist – and that in measuring attitudes we are sometimes interested in the person's perception of reality and not reality itself: not whether they were treated unfairly, but whether they think so.

Construct validity. When there is no single criterion, the scale is tested against hypotheses derived from theory about the nature of the variable: it should correlate highly with measures of related concepts – convergent validity – and correlate weakly with measures of unrelated concepts – discriminant validity (Pallant, 2020, Chapter 1). An anxiety scale that correlates .85 with another anxiety scale and .10 with intelligence behaves the way anxiety should behave. On the site: the correlation matrix is the tool; the interpretation – yours.

The question behind them all. Skowronski (2024) puts validation as one question: "Does the measure behave as we would expect if it really measured the property?" – and the evidence is the kinds of answer: prediction of what it is supposed to predict (predictive), discrimination between groups known to differ (concurrent), convergence with other measures of the same thing (convergent), and non-correlation with what is not supposed to correlate (discriminant) – including a low correlation with a measure of social desirability, as evidence that the scale does not measure a wish to look good. And another kind of evidence that is not a correlation: sensitivity to manipulation – the measure moves in the right direction when what it is supposed to measure is changed experimentally (Table 2). And which evidence is best? "All of them" – the best argument comes from several kinds of evidence across several methods, the spirit of Campbell and Fiske's multitrait-multimethod approach (1959, as cited in Skowronski). And when is it enough? Skowronski warns that validation is harder than it looks: even when the results are consistent, many studies are needed; and when they are not – sometimes it is the criterion (an attitude predicts other attitudes better than it predicts behavior), and sometimes validity really has changed, because the context has changed: a subject who knows they are being watched (a parent assessed for risk of abuse), a questionnaire validated in the lab and completed at home amid interruptions, a different population (West/East, novices/experts), and even a tiny detail in the procedure – his example: a student who could not replicate nonverbal mimicry between conversation partners, until he discovered that in the successful replications the folder of pictures had been closed, and the participants looked at each other instead of at the pictures. Time is also a context: every replication is already a measurement in a different context. And before all of these, the historical warning: phrenology – measuring bumps on the skull as a measure of personality – dominated academia and culture for decades before it was refuted, and the damage (craniometry, physiognomy, and what was built on them) persisted; a lesson, in his words, not only about the past: he also lists current measures whose validity is still disputed in the literature, and which are already used in hiring and in interventions (Skowronski, 2024). What to do in the meantime. Choose the most valid measure the constraints allow – and there is usually an existing validated measure; anyone building their own gathers a body of evidence before building a research program on it; and in any case: repeated calibration of instruments, thorough training of those who measure and code, with periodic checks that they have not "drifted", control of conditions – keeping the same circumstances across data-collection sessions, and rechecking after any change in equipment or staff – and repeated measurements: the same instrument twice, different instruments, several raters (Skowronski, 2024). The last three are exactly what intercoder reliability checks (section 2).

Face validity. "Looks right, reasonable, sensible" – in the eyes of subjects or readers. Krippendorff (2019, chapter 13) places it, together with social validity (the relevance of the research to public discourse), against empirical validity, and warns: when there is no empirical evidence, face and social validity – which researchers control through their rhetoric – remain the only recourse. They are not a substitute.

And in content analysis. Krippendorff (2019, chapter 13) breaks empirical validity down by the stages of content analysis: sampling validity – the extent to which the sample of texts represents the population of phenomena in whose place it was analyzed (texts usually come from the source's selection, not from sampling); semantic validity – the extent to which the analytical categories match the meanings the texts have in the chosen context, and not only for the researcher; structural and functional validity – the extent to which the inference rules rest on established theory or data, and the extent to which they work like previous successful analyses; and correlative and predictive validity – correlation with more valid measures (including convergent and discriminant) and prediction of events that did not enter the analysis. The list is longer than that of psychometrics for one reason: content analysis infers from text to context, and every step of that inference requires its own evidence.

4. The relationship between them – necessary, not sufficient

The two concepts are linked in a fixed order: to be valid, the instrument must first be reliable (Field, 2024, Section 1.5.4). Krippendorff (2019, section 11.1) puts this in two claims and a conjecture. Unreliability limits the chance of validity: if eyewitnesses disagree among themselves, a third party cannot know what happened – and if the coding is a product of chance, it may contain a correct description of the text, but it cannot be identified any better than by chance. Reliability does not guarantee validity: two observers with the same perception, the same prejudice or the same interest will agree with each other and still be wrong; even a computer, the most reliable device, can be wrong – reliably. And the conjecture, from the experience of many content researchers: in the pursuit of high reliability, validity tends to get lost. His example – a study that sought to measure a rising national consciousness in the American colonies through the press, and chose to count mentions of American versus English place names, because "national consciousness" is hard to define and identify: counting words instead of themes – fewer reliability problems, and thin validity.

Raykov (2024) gives "necessary, not sufficient" a number: when the errors of the measure and of the criterion are uncorrelated, reliability is an upper bound on criterion validity – the correlation between a scale and a criterion cannot exceed the square root of its reliability (this is the attenuation formula from section 2 seen from the other side). A scale with reliability .5 will not correlate with anything above .71, even if it measures exactly the right thing. And Skowronski (2024) adds the side that is not a number: a measure can be completely reliable and accurate and still invalid for what was wanted – when the meaning attributed to it is wrong (section 3).

The practical conclusion: report both, in this order, and do not settle for one. An alpha of .90 does not say that the scale measures the concept; a correlation of .60 with a criterion is not worth much if the measurement itself is not consistent. And in designing the coding of text: simpler coding instructions raise agreement – and with each simplification, ask what is left of the concept.

5. How to check on the site – what is computed and what is not

Reliability is the computational part, and its three types are checked in the site's calculators on an Excel file in the standard template (row = subject or coding unit, column = variable): test-retest reliability – on the correlations page, a column for each time point; internal consistency – on the Reliability – Cronbach's alpha page, a column for each item (after reversing the reversed items); intercoder agreement – on the intercoder reliability page, a column for each coder, with Cohen's κ, weighted κ for ordinal scales, Fleiss' κ and Krippendorff's α, confidence intervals, and the two interpretation benchmarks – Landis and Koch versus Krippendorff – side by side. What Raykov (2024) recommends: a confidence interval for α – from version 7 of the alpha calculator it is in Table 2 (distribution-free ADF, normal theory and Feldt's F, at the 95%, 99% and 99.9% levels; Maydeu-Olivares, Coffman & Hartmann, 2007) – and a model-based reliability estimate (ω and alternatives), which the calculator does not compute, and anyone who needs it is referred from there to R; the "α if item deleted" and standardized α in the output – diagnostic, not for reporting (section 2).

Validity has no calculator, and not by chance: the part that can be computed – the correlations for criterion, convergent and discriminant validity – is an ordinary correlation matrix, and its interpretation as validity evidence is a theoretical argument, not output. Content validity and semantic validity are not numbers at all. This page is where the argument is built; the calculators supply it with the numbers.

6. How to report

Reliability is reported in the method section, under "Instruments" or "Measures" (Pallant, 2020, Chapter 9): after describing the scale – the number of items, the response scale, the history of use – give the reliability reported by the scale's developer and by other studies, and then a sentence on the current sample. Pallant's example: "According to Pavot, Diener, Colvin and Sandvik (1991), the Satisfaction with Life Scale has good internal consistency, with a Cronbach alpha coefficient reported of .85. In the current study the Cronbach alpha coefficient was .89." A scale with subscales – reliability for each subscale and for the total score separately. Coefficients between -1 and 1 are written without a leading zero: α = .89, r = .72, κ = .65.

For test-retest reliability report the correlation, N and the time interval; for intercoder agreement – the number of coders, the number of units (and if it is a sample of all the material – how many), the measure with its confidence interval, and the cutoff chosen in advance. Validity is reported as an argument: what evidence was gathered, against which criterion or measure, and what came out – including low correlations that were supposed to be low (discriminant validity), which tend to be forgotten. A well-known scale taken from a previous study: cite its validation studies, and report the reliability in your sample anyway – it is sample-dependent.

Common mistakes

Thinking that a high alpha proves validity. It says that the items go together, not that they go to the right place. A scale of ten items about pay will give an excellent alpha for "job satisfaction" – and will measure satisfaction with pay.

A high alpha thanks to length. Alpha rises with the number of items; 25 mediocre items will reach .85. Also check the mean inter-item correlation, and in a short scale – report it instead of apologizing for an alpha of .6.

Not reversing reversed items. A surprisingly low alpha, and sometimes negative item-total correlations – the classic symptom. Check the scale's manual for which items are negatively worded, and reverse them before the computation.

Test-retest reliability for something that changes. A mood scale at two time points a month apart: a low correlation does not mean the scale is poor – the mood changed. For changing properties check internal consistency, not stability.

Reporting only the scale developer's reliability. Reliability depends on the sample; a different population, a different language, a translation – a different number. Compute it on your sample and report both.

One coder, with no agreement check. Coding of open-ended answers done by one person is an untested interpretation. A sample of units is coded also by a second, independent coder, and κ or α is reported – before the categories become frequencies.

Chasing agreement and losing the concept. The simpler the coding instructions, the higher the agreement – until words are counted instead of ideas. This is Krippendorff's conjecture: in the pursuit of reliability, validity gets lost. With each simplification, ask what is left of the concept.

Deleting an item because "α if item deleted" rises. This statistic can lead to a less reliable and less valid scale; an item whose correlation with the scale is negative is almost always an unmarked reversed item, not an item to delete (Raykov, 2024).

Reporting standardized α because it is higher. It is the reliability of a different scale – of standardized items that no one sums; report the α of the score being analyzed (Raykov, 2024).

Assuming that validity shown in the lab or in one population carries over as is. Context, population and time change validity; recheck, and at least report the difference (Skowronski, 2024).

A magic cutoff. ".7 for alpha", ".8 for kappa", ".667 for Krippendorff's alpha" – all are conventions, and not all from the same source or by the same logic; Landis and Koch themselves called their benchmark "clearly arbitrary". Set a cutoff in advance, justify it by the cost of error in this study, and report the number with the confidence interval – not just "passed".

Checklist – before using a scale or a coding scheme

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104

Field, A. (2024). Discovering statistics using IBM SPSS statistics (6th ed.). SAGE.

Krippendorff, K. (2019). Content analysis: An introduction to its methodology (4th ed.). SAGE.

Maydeu-Olivares, A., Coffman, D. L., & Hartmann, W. M. (2007). Asymptotically distribution-free (ADF) interval estimation of coefficient alpha. Psychological Methods, 12(2), 157-176. https://doi.org/10.1037/1082-989X.12.2.157

Pallant, J. (2020). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (7th ed.). Routledge.

Raykov, T. (2024). Reliability. In A. L. Nichols & J. E. Edlund (Eds.), The Cambridge handbook of research methods and statistics for the social and behavioral sciences: Vol. 2. Performing research (pp. 189-210). Cambridge University Press. https://doi.org/10.1017/9781009000796.010

Skowronski, J. J. (2024). Measurement validity in the social and behavioral sciences: Some "whys" and "hows". In A. L. Nichols & J. E. Edlund (Eds.), The Cambridge handbook of research methods and statistics for the social and behavioral sciences: Vol. 2. Performing research (pp. 211-231). Cambridge University Press. https://doi.org/10.1017/9781009000796.011

DeVellis (2012), Nunnally (1978), Briggs and Cheek (1986) and Pavot et al. (1991) – as cited in Pallant (2020); Cronbach (1951), Novick and Lewis (1967) and McNeish (2018) – as cited in Raykov (2024); Campbell and Fiske (1959) – as cited in Skowronski (2024). Following the site's rule, the reference list includes only sources read in the original; the two chapters of the Cambridge handbook – by the whole chapter, without page numbers in the body text.