ROC analysis

Loading the data

Paste columns directly from the sheet (including the header row) – select them in Excel, copy, and paste here:

Variables and settings

When to use?

When there is a gold standard – an accepted way of determining who has the condition and who does not (a clinical interview, actual dropout, passing an exam) – and a numeric measure that is supposed to predict it, and the question is how well the measure discriminates between the two groups. The classic example is validating a screening questionnaire against a diagnosis, but the question is broader: Pepe (2003, p. 74) notes that the ROC curve "describes the separation between two distributions", and is therefore useful "wherever the separation between distributions is of interest" – a dropout risk score against dropout, a predicted probability from logistic regression against the outcome, an entrance exam score against academic success.

Pepe (2003, p. 73, Table 4.1) lists four uses: a full description of the possible performance of the measure; comparing and combining studies that used different cutoffs; guidance in choosing a cutoff; and comparing measures that are measured on different scales – a 27-point questionnaire against a 1-5 scale. When the measure itself is already binary (positive/negative), there is no curve but a single point – a 2×2 table; its test of association – on the chi-square page. And the performance measures of a machine learning model – including ROC versus the precision-recall curve for a rare class – on the model performance and validation measures page.

The logic: from one cutoff to all cutoffs

One cutoff. To decide on the basis of a score, a cutoff is set: anyone above it is "positive". Pepe (2003, p. 67) explains that the choice of cutoff "will vary with circumstances" – resources, the cost of the work-up or treatment, the severity of the condition – and that it rests on the accepted trade-off "between failing to detect the condition and falsely identifying it". At a given cutoff there is a 2×2 table and two rates: sensitivity – the proportion identified among those who have the condition (the true positive rate), and specificity – the proportion of negatives among those who do not have the condition; 1 − specificity is the false positive rate (Fawcett, 2006, p. 862).

All cutoffs. The curve is the set of all (false positive rate, sensitivity) pairs that can be obtained by moving the cutoff (Pepe, 2003, p. 67, Equation 4.3) – "a device that simply describes the range of trade-offs that can be achieved by the test". The upper left corner, (0, 1), is perfect classification; the diagonal is random guessing – someone who guesses "positive" in 70% of cases will get 70% sensitivity and 70% false positives (Fawcett, 2006, pp. 862-863). Points on the left side of the curve are "conservative" cutoffs – positive only with strong evidence, few false alarms and few detections; on the upper right side – "liberal" ones (Fawcett, 2006, p. 863).

Only the order matters. The curve does not change under any increasing transformation of the score (Pepe, 2003, p. 69, Result 4.1), and the empirical curve "is a function of the ranks only" (Pepe, 2003, p. 98). That is why measures in different units can be compared with it, and why it also suits an ordinal score with few values – then the curve is stepped, with large jumps and diagonal segments where there are ties between those with the condition and those without (Pepe, 2003, p. 98).

The area under the curve

The area under the curve (AUC) is 1 for a perfect measure and 0.5 for an uninformative measure (Pepe, 2003, p. 77). It has two readings, and the sources are divided over which to prefer – so "both":

Probability of correct ranking. The area is the probability that a random participant from the condition group will get a higher score than a random participant from the other group (Hanley & McNeil, 1982, p. 29; Fawcett, 2006, p. 868), and this is exactly the quantity estimated by the Mann-Whitney test (the two-sample Wilcoxon): the area under the empirical curve equals the U statistic divided by the number of pairs (Pepe, 2003, p. 103, Result 5.3; Hanley & McNeil, 1982, pp. 31-32). A tie between two participants counts as half. Hanley and McNeil (1982, p. 31) stress that this relationship "makes no assumptions about the form of the distributions" – unlike the area of the smooth curve, which rests on two normal distributions.

Average sensitivity. Pepe (2003, pp. 78-79) is not convinced: the correct ranking of a pair is "not particularly compelling" as a clinical question – it is natural in a forced-choice experiment with a pair of images, not in the use of a test – and she prefers to read the area "as an average sensitivity, over the whole range of false positive rates". Both readings are mathematically correct; the first is easy to explain, the second is a reminder that the area also averages over cutoffs that no one would use.

What the area hides. Two measures with the same area can be very different in the region that matters to you: "it is possible for a high-AUC classifier to perform worse in a specific region of ROC space than a low-AUC classifier" (Fawcett, 2006, p. 868), and a curve that is better everywhere implies a larger area, but "the converse is not necessarily true" (Pepe, 2003, p. 77). In population screening, for example, only cutoffs with few false positives are feasible, and there only the left part of the curve is relevant (Pepe, 2003, p. 75) – this is what the partial area is for (pAUC; Pepe, 2003, p. 79), which the page does not compute. That is why the page always shows the curve and not only the area. The Gini coefficient, which appears in the machine learning literature, is 2·AUC − 1 (Hand & Till, 2001, as cited in Fawcett, 2006, p. 868).

Area below 0.5. "No realistic classifier should have an AUC less than 0.5" (Fawcett, 2006, p. 868): a measure below the diagonal "has useful information, but it is applying the information incorrectly" (Flach & Wu, 2003, as cited in Fawcett, 2006, p. 863) – almost always the direction was defined the wrong way round. The page warns in such a case.

Standard error, confidence interval and test

The DeLong standard error. DeLong, DeLong and Clarke-Pearson (1988, pp. 839-840) decompose the area into "structural components": for each participant from the condition group – the proportion of participants from the other group whom they exceed; for each participant from the other group – the proportion of participants from the condition group who exceed them. The variance of the area is the variance of the first components divided by the number of cases, plus the variance of the latter divided by the number of controls. Pepe (2003, pp. 105-106) calls these components "placement values" – where each score is placed in the distribution of the other group – and regards them as a concept "fundamental to ROC analysis". This is the main standard error on the page, and the one that R computes (pROC).

The Hanley and McNeil standard error. The older formula (Hanley & McNeil, 1982, p. 32, Equation 1) rests on two probabilities: Q₁ – that two participants from the condition group will both exceed a participant from the other group, and Q₂ – that a participant from the condition group will exceed two from the other group. The page computes them from the data, with ties as in Table II of the article (p. 31), and this is the standard error that PSPP prints. The two formulas are almost always close. Hanley and McNeil (1982, p. 33, Equation 2) also proposed an approximation that depends only on the area – Q₁ = θ/(2 − θ), Q₂ = 2θ²/(1 + θ) – for planning sample size before there are data; Table III in the article (p. 34) gives the numbers of participants needed to compare two areas.

Confidence interval – two kinds, "both". The common one – the area plus and minus 1.96 standard errors (DeLong), as in R. But the area is bounded at 1, and when it is close to 1 its distribution is "skewed toward 0.5" (Hanley & McNeil, 1982, p. 34), and the symmetric interval can exceed 1 (the page truncates it at 1, like R). Pepe (2003, p. 107) proposes an interval on the logit scale – asymmetric, always between 0 and 1 – and shows in a simulation that in small samples and with high areas its coverage is better (Table 5.1); "in practice we recommend" it. The page shows both; the findings paragraph uses the symmetric DeLong interval, which can be reproduced in R and in SPSS, and with an area of .90 and above or a group of fewer than 30 participants – it is better to report the logit interval.

Is the area greater than 0.5? The page reports z and p for the null hypothesis that the measure does not discriminate (area 0.5), with the standard error under the null hypothesis – this is the Mann-Whitney test in the normal approximation, without a correction for ties, and it is the Asymptotic Sig. of PSPP. With a correction for ties (the Mann-Whitney page) the p is slightly smaller. Such a test answers a modest question: in a large sample even an area of .58 is "significant", so what is reported is the area and its confidence interval.

Choosing a cutoff

Youden index. J = sensitivity + specificity − 1, that is, the vertical distance between the curve and the diagonal; the cutoff that maximizes it is the page's default. Pepe (2003, p. 80) presents it as a summary proposed "when the cost of false positive errors is similar in magnitude to that of false negative errors" – and this is an assumption, not a fact. When the costs differ, the right cutoff depends on the costs and on the prevalence of the condition (Pepe, 2003, p. 72, Result 4.5; Fawcett, 2006, p. 867, "iso-performance lines"): in screening for a dangerous condition, high sensitivity is preferred even at the cost of false alarms; before a costly intervention – the opposite. That is why the page lets you check any cutoff, and shows all cutoffs in a full table. And if there is an accepted cutoff for the measure in the literature – report that one, not the cutoff found in the data.

A cutoff chosen on the data is optimistic. The Youden cutoff is chosen to fit exactly this sample, so the sensitivity and specificity at it are slightly higher than what will be obtained in a new sample. By the site's convention, a cutoff chosen on the data is reported as a preliminary finding, and its performance is validated in another sample (on validation on new data – the model performance and validation measures page).

What is computed at the cutoff. Sensitivity and specificity with an exact confidence interval (Clopper-Pearson) – Pepe (2003, p. 22) prefers exact or logistic methods, because in good tests the rates are close to the boundary, and a symmetric normal interval "may extend outside (0, 1)". It is important to report both, and not overall accuracy: the two errors differ in cost, and overall accuracy depends on prevalence – a test that classifies everyone as negative will get high accuracy for a rare condition (Pepe, 2003, p. 15). The likelihood ratios (LR+ = sensitivity/(1 − specificity), LR− = (1 − sensitivity)/specificity) tell how many times a positive or negative result changes the odds of the condition, and do not depend on prevalence (Pepe, 2003, p. 20, Table 2.2); their confidence interval – on the log scale (Pepe, 2003, p. 24, Result 2.5).

Predictive values and prevalence. Positive predictive value – among those classified as positive, how many truly have the condition; negative – among the negatives, how many truly do not. They are what interests the person who received a result, but they "depend not only on the performance of the test... but also on the prevalence of the condition" (Pepe, 2003, p. 16). When the sample was built from a fixed number of cases and of controls, the sample proportion "will overestimate the positive predictive value" (Pepe, 2003, p. 25). So enter the prevalence of the condition in the population in which the measure will be used, and the page will compute the predictive values in it by Bayes' rule. The ROC curve itself, by contrast, "is insensitive to changes in class distribution" (Fawcett, 2006, p. 864).

Comparing measures

When several measures were taken on the same participants, their areas are correlated – the same people are "easy" and "hard" for all the measures – and a comparison that ignores this is wrong. DeLong and colleagues (1988) developed a nonparametric approach for this: a covariance matrix of the areas from the structural components, and a chi-square test for any contrast between them (pp. 840-841, Equation 5). For a pair of measures – z for the difference between the areas, with a confidence interval (Pepe, 2003, p. 108, Result 5.6); for three or more – an omnibus test of the equality of all areas (k − 1 degrees of freedom), followed by all pairs, with a Holm correction for multiple comparisons by the site's convention. A measure in the reverse direction (well-being versus symptoms) enters the comparison after a sign reversal, as DeLong and colleagues did with their score (p. 842).

The lesson of their example (p. 844): a special score for predicting surgical success was no better than two routine nutritional measures, but "the power in this situation is probably very low because of the small sample size" – 49 patients. The absence of a significant difference between areas in a small sample is not proof that the measures are equivalent. When the measures were taken on different groups of participants, the variance of the difference is simply the sum of the variances (Pepe, 2003, p. 108) – the page does not compute such a comparison.

How to read the output?

Table 1 – The area under the curve: for each measure – the direction, the area, the standard errors (DeLong; Hanley & McNeil), the two confidence intervals, and z and p against 0.5. Figure 1 – the curves, with the diagonal and the Youden cutoff of each measure.

Table 2 – The cutoff by the Youden index: for each measure – the cutoff (positive if the score is greater than or equal to it; for a measure in the reverse direction – less than or equal), sensitivity and specificity with confidence intervals, J, predictive values and likelihood ratios. The cutoffs are the midpoints between adjacent values, as in SPSS and R, so a cutoff of 10.5 on a questionnaire of whole numbers means "11 and above".

Checking a cutoff: select a measure and type a cutoff, and you get Table 3 – a 2×2 table and all the measures at that cutoff, including predictive values at the prevalence you entered. Below it, at a click – all the cutoffs of the measure. Table 4 – Comparison of the areas (when two or more measures were selected): the difference between each pair, the standard error, the confidence interval, z, p, Holm-adjusted p, and the correlation between the areas; in the note – the omnibus test.

Assumptions and warnings

The gold standard is correct. The entire computation assumes that the condition variable is not wrong. Pepe (2003, pp. 6-7, Table 1.3) lists the common biases in such studies: an imperfect gold standard; verification bias – only some of the participants, and not at random, were tested with the gold standard; spectrum bias – "a common mistake is to select cases with severe or chronic disease and controls that are healthier than average", which "can inflate the apparent accuracy of the test"; and interpretation bias – when whoever determines the condition knows the result of the measure, or vice versa (pp. 5-6).

Independent participants. The standard errors assume that each participant contributes one observation. When there are several observations per participant (two eyes, several tests per student), Pepe (2003, p. 104) recommends a bootstrap at the participant level – which the page does not run.

Ties. In scores with few values there are many ties, and Fawcett (2006, p. 867) warns that if they are not averaged, the curve depends on the order in which the participants appear in the file, and "different orderings can yield very misleading curves". The page averages them: a tie between a case and a control counts as half, and the curve passes along a diagonal line.

The direction is set in advance. Software that chooses a direction "automatically" from the data turns every weak measure into an area above 0.5. The page does not do this: the direction is your decision, according to what the measure is supposed to measure, and an area below 0.5 is shown as it is, with a warning.

Practical limitations. Missing values: listwise deletion on the condition variable and on all selected measures (the page reports how many were omitted). At least five participants are required in each group; with fewer than 30 in a group – the asymptotic confidence intervals are less accurate (Pepe, 2003, p. 107, Table 5.1). An area of 1 (complete separation) – the standard error is 0 and the confidence interval is meaningless, and the page warns; for exact intervals in such a case Pepe (2003, p. 107) refers to Obuchowski and Lieber (2002, as cited there). What the page does not compute: a smooth (binormal) curve, a partial area, confidence bands for the whole curve, a comparison between independent groups, and ROC for more than two groups.

And how do you do it in SPSS?

In SPSS Statistics: Analyze › Classify › ROC Curve (in older versions – Analyze › ROC Curve); move the measures to Test Variable, the condition variable to State Variable and type the value that indicates the condition; check ROC Curve, With diagonal reference line, Standard error and confidence interval and Coordinate points of the ROC Curve. The direction is set in Options › Test Direction – one for the whole command, so the syntax the page generates runs a separate command for measures in the reverse direction. The paths are general information about the software; the syntax was not run in SPSS.

The same command (ROC) exists in PSPP, the free alternative, and was run on the sample file: the area and Std. Error are identical to the page (the Std. Error is the Hanley and McNeil one, the "SE (H-M)" column in Table 1), and so is Asymptotic Sig.; but the limits PSPP prints under "95%" are 1.645 standard errors from the area and not 1.96 – that is, a 90% interval – and with the direction TESTPOS(SMALL) it returns NaN for the standard error and the interval. In PSPP – take the confidence intervals from the page. Comparing areas is not part of the ROC command; it is in the R script below.

And how do you do it in R?

The pROC package: roc builds a curve (levels – the negative value followed by the positive one; direction = "<" – a high value indicates the condition, ">" – a low value; without direction the package chooses a direction from the data – see the assumptions), ci.auc gives the area and the DeLong confidence interval, coords – the Youden cutoff and all cutoffs, and roc.test – the paired DeLong comparison. The page is validated against the package: the area, the standard error, the confidence interval, all the points of the curve, the Youden cutoff, the comparisons and the omnibus test. The script was run as is on the sample file (the values are in the comments).

# ROC analysis – Professor Stat. This file is saved in UTF-8; every comment line starts with an English word
library(readxl)    # Reading Excel
library(pROC)      # ROC curves, area, confidence intervals and DeLong comparison

# Loading: the standard layout – row = participant, column = variable; the sheet "Data"
d <- read_excel("C:/Rdata/DATA_roc_example_eng.xlsx", sheet = "Data")
v <- c("diagnosis", "screen_long", "screen_short", "wellbeing")
d <- d[complete.cases(d[, v]), ]                        # Listwise deletion: 218 of 220 – the same participants for all curves

# Curve for each measure; levels = c(negative, positive); direction "<" – a high value indicates the condition, ">" – a low value
r_long  <- roc(d$diagnosis, d$screen_long,  levels = c(0, 1), direction = "<")
r_short <- roc(d$diagnosis, d$screen_short, levels = c(0, 1), direction = "<")
r_well  <- roc(d$diagnosis, d$wellbeing,    levels = c(0, 1), direction = ">")

# The area and the DeLong confidence interval
ci.auc(r_long); ci.auc(r_short); ci.auc(r_well)       # Area: .817 [.757, .876]; .762 [.693, .831]; .677 [.596, .759]
sqrt(var(r_long))                                      # Standard error (DeLong): .030

# The Youden cutoff, and what happens at it
coords(r_long, "best", best.method = "youden",
       ret = c("threshold", "sensitivity", "specificity", "ppv", "npv"))   # Cutoff 10.5: sensitivity .875, specificity .676
binom.test(42, 48)$conf.int                            # Exact confidence interval for sensitivity at the cutoff: [.748, .953]
coords(r_long, "all", transpose = FALSE)               # All cutoffs

# Paired comparison of areas (same participants) – DeLong
roc.test(r_long, r_short, method = "delong", paired = TRUE)   # Difference .055 [.007, .103], z = 2.25, p = .025
# Well-being is in the reverse direction: the comparison is on the sign-reversed variable, so that all curves are in the same direction (DeLong et al., 1988, p. 842)
r_well2 <- roc(d$diagnosis, -d$wellbeing, levels = c(0, 1), direction = "<")
roc.test(r_long, r_well2, method = "delong", paired = TRUE)   # Difference .140 [.062, .217], z = 3.54, p < .001

# Plot
plot(r_long, legacy.axes = TRUE, print.auc = TRUE)
lines(r_short, col = "darkorange"); lines(r_well, col = "darkgreen")

Reporting the results

Report the gold standard and the number of participants in each group; for each measure – the direction, the area and the confidence interval (and which one); the cutoff and how it was chosen – in advance from the literature or on the data – with the sensitivity and specificity and their confidence intervals; predictive values only with the prevalence at which they were computed; and for the comparison – the test (DeLong, paired), the difference and its confidence interval. Numbers in APA format: no zero before the decimal point for the area, sensitivity and specificity, which cannot exceed 1 (APA, 2020, Section 6.36), and p to three digits. The paragraph the page generates is a starting point – complete it with the context and the meaning.

Common mistakes

Reporting only the area. An area of .82 averages over all cutoffs, including ones that no one would use (Pepe, 2003, pp. 78-79; Fawcett, 2006, p. 868), and what is actually decided on is a cutoff. Report the area, the curve, and the sensitivity and specificity at the cutoff. And the reverse – reporting only one point: reporting sensitivity and specificity at a single cutoff is "clearly limited" (Pepe, 2003, p. 71).

Reporting overall accuracy instead of sensitivity and specificity – for a rare condition it is high even for a worthless test (Pepe, 2003, p. 15).

Reporting a positive predictive value from a case-control sample as if it applied to the population – it is inflated (Pepe, 2003, p. 25).

Presenting the Youden cutoff as "the optimal cutoff" – it is optimal only when both errors have the same cost, and only in the sample in which it was found.

Comparing areas of measures taken on the same participants with a test for independent groups, or by overlap of confidence intervals – the correct comparison is paired (DeLong et al., 1988).

Letting the software choose the direction – and reporting "AUC = .58" for a measure that actually predicts the opposite of what was planned, or does not predict at all.

Choosing a cutoff, and then "validating" it on the same data – without another sample, the numbers at the cutoff are optimistic.

References

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). https://doi.org/10.1037/0000165-000

DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837-845. https://doi.org/10.2307/2531595

Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861-874. https://doi.org/10.1016/j.patrec.2005.10.010

Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29-36. https://doi.org/10.1148/radiology.143.1.7063747

Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford University Press.

Hand and Till (2001) and Flach and Wu (2003) – as cited in Fawcett (2006); Obuchowski and Lieber (2002) – as cited in Pepe (2003). By the site's rule, the reference list includes only sources that were read in the original; quotations follow the printed page. The computation on the page is validated against R 4.3.3 and the pROC 1.18.5 package (roc, ci.auc, coords, roc.test, cov), against binom.test, and against the ROC command of PSPP 2.0; and against the numbers in the sources – 0.893 and a standard error of 0.032 in Table II of Hanley and McNeil (1982, p. 31).