Performance metrics and validation: how to know the model is good

Guide

On this page

A prediction model is validated in only one way – on data it has not seen. The machine learning page explains the difference between a model that explains and a model that predicts, what the models are and how a project is built; this page deals with one question on which every report of such a model stands or falls: where the number comes from. Four sub-questions, in the order in which they arise: which metric (1) – because "accuracy" is usually the wrong metric; on which data (2) – training, validation and test, and why the test set is set aside before anything else; how to get not only a number but also its precision (3) – cross-validation; and what the number can be at all (4) – bias, variance, and Bayes error, the limit below which no model goes. Then the two ways in which the number becomes a lie without anyone cheating: data leakage (5), and subjects who are not independent of one another – families, classrooms, networks – which the usual training/test split ignores (6). Section 7 turns it all into a reporting paragraph. The sources are the four textbooks of the machine learning page – Géron (2022), Hastie, Tibshirani and Friedman (2009), Bishop (2006), Ishii et al. (2026) – and three papers written for researchers: on leakage (Kapoor & Narayanan, 2023), on the limits of prediction (Salganik et al., 2020) and on the difference between validating an explanatory model and validating a predictive one (Shmueli, 2010); and for section 6 – the foundational book of relational learning (Getoor & Taskar, 2007). And a fifth source, from the software side: Truong's (2026) textbook for SPSS Modeler, written "for learners from non-technical fields and experienced researchers alike" – which says what the same concepts are called in IBM's menus, what the software does by default, and how to write the report. And what happens when there is no dependent variable at all – and therefore no metric – in unsupervised learning.

1. The metric: by task and by the cost of error

The metric is chosen at the start of the project, not at the end – because it determines which model is "good" (Géron, 2022, Chapter 2, "Select a Performance Measure"). For regression – RMSE, the root mean squared error, in the units of the target, which gives high weight to large errors; or MAE, the mean absolute error, when there are many outliers. For classification – this is where the problem begins. Chapter 3 of Géron trains a binary classifier to detect the digit 5 in images of handwritten digits: cross-validation shows overall accuracy (accuracy, the proportion of correct classifications) above 95% in every fold. Then a dummy classifier that says "not 5" for every image – and gets above 90%, because only about a tenth of the images are 5s. "It beats Nostradamus." The conclusion: "accuracy is generally not the preferred performance measure for classifiers, especially when you are dealing with skewed datasets" (Chapter 3, "Performance Measures") – and the data sets of social research are almost always skewed: dropout, violence, unemployment, voting for a small party. Bishop gives the same example from medicine: cancer in 1 of every 1,000 X-rays – "a classifier that assigned every point to the normal class would already achieve 99.9% accuracy and it would be difficult to avoid this trivial solution"; his solution is to train on a balanced data set and then correct the probabilities back according to the class proportions in the population (Bishop, 2006, p. 45).

The confusion matrix (confusion matrix) is the tool: rows – the actual class, columns – the predicted class; in Géron's example 53,892 true negatives, 687 false positives (Type I error), 1,891 false negatives (Type II error) and 3,530 true positives – and it is computed on "clean", out-of-sample predictions that the model made for observations it did not see in training. From it are derived precision ("positive accuracy": when the model says 5 – what proportion of the time it is right; 83.7%) and sensitivity (recall: what proportion of the actual 5s it catches; 65.1%), and F1 – their harmonic mean (.73), which is high only when both are high. Which of them matters is a question of the cost of error, not of statistics: a video filter for children needs high precision even at the cost of low recall; a shoplifter detector can settle for 30% precision if recall is 99%. The two trade off against each other through the threshold: raising the threshold raises precision and lowers recall (Chapter 3, "The Precision/Recall Trade-off"). The ROC curve shows recall against the false positive rate across all thresholds, and the area under it (AUC) – 1 for a perfect classifier, 0.5 for a random one – compares classifiers without choosing a threshold; but with a rare class the ROC is flattering, and "as a rule of thumb, you should prefer the PR curve whenever the positive class is rare or when you care more about the false positives than the false negatives" (Chapter 3, "The ROC Curve"). The curve itself – with a confidence interval for the area, a threshold by the Youden index, sensitivity and specificity at each threshold and a comparison between measures – is computed on the ROC analysis page. Anyone familiar with the classification table in SPSS knows the 0.5 threshold as a default nobody changes; here it is a reported decision.

A rare class: what to do – and what not. Truong orders the remedies by priority: first multiple metrics instead of overall accuracy, then partitioning and validation, then cost-sensitive learning (cost-sensitive learning) – and only "if the previous methods do not effectively address the imbalance issues and there are concerns that our models are not valid, resampling becomes a reasonable option" (Truong, 2026, pp. 78-79). Cost-sensitive learning is the formal statement of "the cost of error": a decision matrix in which every error has a cost – C(FN) for a miss, C(FP) for a false alarm – and the model chosen is the one whose total cost, C(FP) × FP + C(FN) × FN, is lowest, not the one whose error rate is lowest; "most software includes an option to enter the decision matrix" (pp. 203-204). Oversampling of the rare class (duplicating observations, or SMOTE – synthetic observations between near neighbors) and undersampling of the common one – about the former Truong himself remarks that "duplicating the data makes the models more prone to overfitting" (p. 79) – were tested in a simulation on logistic regression: all three "resulted in poorly calibrated models: the probability to belong to the minority class was strongly overestimated", did not raise the AUC compared with an uncorrected model, and the balance between sensitivity and specificity they achieved "could equally be obtained by shifting the probability threshold instead"; the conclusion – "outcome imbalance is not a problem in itself, and imbalance correction may even worsen model performance" (van den Goorbergh et al., 2022, abstract). The two readings are compatible: the rare class is a problem of the metric and the threshold, not of the data. A researcher who needs probabilities – for risk scoring, for reporting – does not balance; one who balances anyway does so only inside the training fold (leakage L1.2, section 5) and reports that the probabilities the model produces are no longer probabilities. The Modeler textbook stands on the resampling side – the authors "tend to treat data sets with balancing or resampling methods" when the minority class is "10% or less" (a ratio of 10:1 or 20:1) – but they agree on the two things that matter here: "we should never rely solely on the overall accuracy", and "to separate a representative test subsample, the partitioning method should be used before applying a resampling procedure" (Wendler & Gröttrup, 2021, pp. 1148, 1150).

And between "yes" and "no" – not deciding. Bishop's decision theory adds the reject option (reject option): refraining from a decision in cases where the highest posterior probability is below a threshold, and leaving them "to a human expert" (Bishop, 2006, p. 42). For a researcher who codes open-ended answers with a model (NLP page, step 6), this is the right procedure: the model classifies the clear cases, a person the ambiguous ones, and the agreement between them is measured with intercoder reliability.

2. On which data: training, validation, test

The difference from validating an explanatory regression. Shmueli distinguishes: in explanatory modeling, validation checks that the model represents the theory and that it fits the data – specification tests, reliability and validity of items, residual analysis; in predictive modeling validation "focuses on generalization, which is the ability of the model to predict new data", and "the greatest danger to generalization is overfitting the training data" (Shmueli, 2010, Section 2.6.1). Therefore "the only way to know how well a model will generalize to new cases is to actually try it out on new cases" – a test set is set aside, customarily 20% of the data (Géron, 2022, Chapter 1, "Testing and Validating"). "Training error is not a good estimate of the test error" – it falls consistently with model complexity, down to zero, "however, a model with zero training error is overfit to the training data and will typically generalize poorly" (Hastie et al., 2009, pp. 220-221); and in the language of pattern recognition, the "resubstitution" error (resubstitution, testing the classifier on the same observations it learned from) is a lower bound of the true error and the test error an upper bound – the truth lies between them (Ishii et al., 2026, pp. 113-114).

Before exploration, not after. "It may seem strange to voluntarily set aside part of the data at this stage", writes Géron, but "your brain is an amazing pattern detection system, which also means that it is highly prone to overfitting": anyone who looks at the test set will choose a model according to what they saw there, and their generalization estimate will be too optimistic – data snooping bias (data snooping bias; Chapter 2, "Create a Test Set"). Appendix A of the book puts it as a command: "sample a test set, put it aside, and never look at it". Two technical details that matter for reproducibility: the split should be stable across runs (a fixed seed, or a hash of the subject's identifier – so that a subject who was in training does not move to the test set when data are added); and it should be stratified – Géron explains this through surveys: the US population is 51.1% women, so a sample of 1,000 should keep 511 women; with purely random sampling there is a 10.7% chance of a skewed sample (fewer than 48.5% or more than 53.5% women). In the same way the test set is stratified by a key variable – and with a rare class, by the outcome itself.

Three sets, not two. If a model is tuned – choosing between models, or setting a hyperparameter such as regularization strength – on the test set, we have measured the generalization error "multiple times on the test set, and adapted the model and hyperparameters to produce the best model for that particular set"; the model that seems to have 5% error produces 15% in the field. The solution is a validation set (validation set, also dev set): part of the training set is set aside, the candidates are compared on it, the chosen model is trained on the whole training set, and only then – once – is it measured on the test set (Chapter 1, "Hyperparameter Tuning and Model Selection"). And what happens after the measurement: "you must resist the temptation to tweak the hyperparameters to make the numbers look good on the test set; the improvements would be unlikely to generalize to new data" (Chapter 2, "Evaluate Your System on the Test Set"); Appendix A: "don't tweak your model after measuring the generalization error: you would just start overfitting the test set". The number on the test set also comes with a confidence interval: in Géron's example an RMSE of 41,424 and a 95% interval – with a t distribution on the squared errors – between 39,275 and 43,467.

The names change between programs. "While most software consistently calls the first partition sample 'training,' the names of the second and third samples vary, which can be confusing" (Truong, 2026, p. 85): in SPSS Modeler the partition is Train / Test / Validation – the "test" sample is used to compare and refine the models, and the "validation" sample is the final measurement – in the words of the documentation: "to train the model with one sample, test and refine the model on a second sample, and validate your results on a third" (IBM Corporation, 2016, p. 157); in SAS Enterprise Miner – Train / Validation / Test, the reverse (Table 3.8 there); and Géron and Hastie et al., whom this page follows, go with SAS: validation for selection, test for the last measurement. The order is what matters, not the name – and a researcher who reports "accuracy on the test set" from Modeler output is reporting, in this page's terms, the validation set. The ratios differ too: Truong recommends 60-20-20 for a sample of a thousand to a million observations, and "fewer than 1,000 observations – 60-40 without a third set" (p. 87), while his sample recommendation for data mining is "at least 100,000 cases" (p. 73). That is a different world from a survey of 300 respondents – where a fixed validation set of 60 respondents is a number that depends on chance, hence section 3. And what Truong contributes to those working at small scale: his distinction between a model's reliability – "internal consistency", the same results on different samples – and its validity – "accuracy", correct prediction of the target; "in fact, model validity is the same as predictive power", and reliability without validity is a scale that consistently shows about 22 pounds for a 25-pound sack (p. 186). The evidence for reliability, for him, is a small gap between the metric on training and the metric on testing – "if the discrepancy is less than 5%, it can be considered reliable" (p. 187) – and that is exactly the gap section 7 requires reporting.

What it looks like at scale. In the Fragile Families Challenge, the organizers split the half of the data not used for training into two sets: a "leaderboard" (leaderboard), on which the 160 teams could check themselves during the competition, and a "holdout" (holdout), whose accuracy "was not known to participants – or the organizers – until the end of the challenge" (Salganik et al., 2020, Section Methods). And even so the authors add a caveat: using the same holdout set both to select the best submission out of 160 and to evaluate it "produced slightly optimistic estimates" – a small bias in their case, but a real one (Section Results). For a researcher who tests once on a sample of 300, this bias is not small.

3. Cross-validation: not only a number, but also its precision

A single validation set gives a single number, which depends on chance – on which split came out. Cross-validation (k-fold cross-validation) divides the training set into k folds, trains k times on k − 1 and tests on the remaining one, and obtains "not only an estimate of the performance of your model, but also a measure of how precise this estimate is – its standard deviation" (Géron, 2022, Chapter 2, "Better Evaluation Using Cross-Validation"). In Géron's housing-price example: a decision tree with a training error of 0.0 gets, in 10-fold cross-validation, an RMSE of 66,868 (±2,061) – almost the same as linear regression (69,858 ±4,182); a random forest 47,019 (±1,034), but on the training set itself about 17,474 – still considerable overfitting. Two numbers for each model, and the same folds for all models.

How many folds. "Overall, five- or tenfold cross-validation are recommended as a good compromise": fewer folds train on too small a sample and bias the error estimate upward; leave-one-out (leave-one-out, k = N) is unbiased but variable (Hastie et al., 2009, p. 243). And the one-standard-error rule: among the models tested, choose the most parsimonious one whose error is no more than one standard error above the best model – because the cross-validation curve itself is estimated with error (p. 244). The cost, in Bishop: "one major drawback of cross-validation is that the number of training runs that must be performed is increased by a factor of S" (Bishop, 2006, p. 33) – for the site's calculators this is nothing, for a neural network it is days. And why cross-validation at all rather than a single validation set: in the holdout method (hold-out) whatever is given to testing is taken away from learning, and vice versa – "when the number of observations is small, the H method is not an accurate method" for estimating the error, whereas cross-validation "ensures that all members are used for both learning and testing" (Ishii et al., 2026, p. 89). Random forests get validation for free: each tree is trained on a bootstrap sample that leaves out about 37% of the observations, and the error estimate on the "out-of-bag" observations (out-of-bag) is "almost identical" to N-fold cross-validation (Hastie et al., 2009, p. 593).

Tuning too – inside the cross-validation. Grid search (all combinations of hyperparameters) or random search (a fixed number of combinations; preferable when the space of combinations is large – Appendix A: prefer random unless there are few values) runs cross-validation for each combination, and chooses by the mean. The rule: every decision based on performance – model, hyperparameter, threshold, feature filtering – is made inside the folds, not on them (section 5).

4. Bias, variance – and the limit of error

A model's generalization error is the sum of three: bias – wrong assumptions (a straight line for quadratic data) – which leads to underfitting; variance – excessive sensitivity to small changes in the training data, the lot of models with many degrees of freedom – which leads to overfitting; and irreducible error – the noise in the data themselves, which only cleaning the data reduces. Increasing model complexity raises variance and lowers bias, and vice versa – "this is why it is called a trade-off" (Géron, 2022, Chapter 4, "The Bias/Variance Trade-Off"). The formal decomposition – the expected prediction error at a point = noise variance + squared bias + variance of the estimator – is Equation 7.9 in Hastie et al. (p. 223), and in pattern recognition it is the "bias-variance dilemma": MSE = Bias² + Var, so when the degree of the discriminant function is raised and testing is done on new observations, the error falls "up to a certain degree, and then" rises (Ishii et al., 2026, pp. 111-112). The diagnosis: good on training and bad in cross-validation – overfitting; bad on both – underfitting. Learning curves – training error and validation error as a function of sample size – show this visually, and Bishop adds what helps: "for a given model complexity, the over-fitting problem become less severe as the size of the data set increases" (Bishop, 2006, p. 9). On the ratio between the number of observations and the number of parameters and features – Bishop's rule of thumb and the Hughes phenomenon – see the machine learning page, section 2.

And what the number can be at all. There is a limit below which no model goes: Bayes error – "the limit at which the error probability cannot become any smaller, in other words the 'overlap of the distributions', which is the 'unavoidable error' introduced by this feature extraction system" (Ishii et al., 2026, p. 101). When the features do not separate the classes, no algorithm will separate them. This is the right reading of the Fragile Families Challenge: 160 teams, 4,242 families and 12,942 variables on each – and an R² on the holdout data of about 0.2 for material hardship and grade point average and about 0.05 for the other four outcomes, "only slightly better" than a baseline model of a regression on four variables chosen by an expert; the prediction error was strongly related to the family being predicted and only weakly to the method (Salganik et al., 2020, abstract and section Results). "Because hundreds of researchers attempted this task and none could predict accurately", low accuracy "cannot easily be attributed to the limitations of any particular researcher or approach" (Section Discussion) – this is the limit, not failure. Kapoor and Narayanan list "a lack of understanding of the limits to prediction" as their first diagnosis of reproducibility failures: the successes in image recognition "give the impression" that models outperform humans at every task, and confuse researchers about what can be expected in predicting social outcomes; the recommendation – identify upper bounds on accuracy, and once they are reached, stop trying to exceed them and treat results claiming to have exceeded them with suspicion (Kapoor & Narayanan, 2023, Section Discussion, D1-R1). For the researcher: a model reporting AUC = .95 for dropout from background variables is not a breakthrough – it is a sign for section 5.

5. Data leakage: the impressive number that is a lie

Kapoor and Narayanan surveyed the literature of machine-learning-based science and found leakage (leakage) in 294 papers across 17 fields, "in some cases leading to wildly overoptimistic conclusions" (Kapoor & Narayanan, 2023, Section Summary). Their taxonomy is this page's checklist: [L1] no clean separation between training and test – no test set at all (L1.1); preprocessing on all the data – imputing missing values, standardization or oversampling before the split (L1.2; this is Géron's warning: "never use fit() or fit_transform() for anything else than the training set", Chapter 2, "Feature Scaling and Transformation"); feature selection on all the data (L1.3); duplicates (L1.4). [L2] illegitimate features – which are in effect the target: their example is a study that predicted hypertension from the use of antihypertensive medication. [L3] a test set not drawn from the distribution about which the claim is made – temporal leakage (testing on data earlier than the training data, L3.1), and broken independence (L3.2, Section 6). Their demonstration is political science: in predicting civil wars, where complex models "were thought to vastly outperform" logistic regression, "when the errors are corrected, complex ML models do not perform substantively better than decades-old LR models" (Section Summary). Their proposal – "model info sheets" in which the researcher answers for each type of leakage – and to readers: "treat results from ML-based science as provisional" (R2).

The experiment that showed this fourteen years earlier. Hastie et al. simulate 50 subjects in two equal classes and 5,000 random predictors that are not related to the class at all – the true classification error is 50%. "The wrong way": filter, on the whole sample, the 100 predictors most correlated with the class, and then estimate the error by cross-validation – a mean of 3% over 50 simulations, "far lower than the true error rate", because the predictors "have already seen" the subjects left out; "the right way": divide into folds first, and carry out the filtering, the building of the classifier and the prediction inside each fold (Hastie et al., 2009, pp. 245-246). This is exactly L1.3 – and it is the reason why filtering items by their correlation with the outcome, a common habit in scale construction, is forbidden before the split. The technical solution is a pipeline (Pipeline in scikit-learn): all preparation steps and the model as one unit, which is fitted on the training fold and applied to the test fold – and therefore cannot leak.

6. Data that are not independent: families, classrooms, networks

Everything above assumes that the observations are independent and identically distributed (i.i.d.) – and in social research they almost never are: students in the same classroom, members of the same family, friends in a network, repeated measurements of the same person. The problem was described as an obstacle to validation in section L3.2 of Kapoor and Narayanan: "non-independence between train and test samples constitutes leakage, unless the scientific claim is about a distribution that has the same dependence structure. In the extreme (but unfortunately common) case, train and test samples come from the same people or units" (Kapoor & Narayanan, 2023, L3.2). The relational learning literature describes the same thing as a property of the data, not as an error of the researcher: "the vast majority of work in statistical classification methods has focused on 'flat' data – data consisting of identically-structured entities, typically assumed to be i.i.d. However, many real-world data sets are innately relational: hyperlinked webpages, cross-citations in patents and scientific papers, social networks, medical records" (Taskar, Abbeel, Wong & Koller, 2007, p. 175); "unlike traditional machine learning settings where the learning system receives as input a sequence of i.i.d. observations, the input to a relational learning algorithm is usually a single, richly interconnected instance" (Getoor & Taskar, 2007, p. 4). The mechanism is autocorrelation (autocorrelation): "a statistical dependency between the values of the same variable on related entities", which "is a nearly ubiquitous characteristic of relational data sets" – Neville and Jensen list, among others, the industry classification of corporations that share board members, the fraud status of cellular customers who call common numbers, and tuberculosis infection among people in close contact (Neville & Jensen, 2007, p. 240). This is "a unique opportunity to improve model performance because inferences about one object can inform inferences about related objects" (ibid.) – and for exactly the same reason, an ordinary model that gets the student in training and their classmate in testing "guesses" from the autocorrelation, not from the features.

What to do. The split – into training and test, and into cross-validation folds – is made by the unit about which the claim is made: by subject in repeated measurements, by classroom or school in nested data, by family, by community (in the tools: GroupKFold and its relatives in scikit-learn). Metrics by unit: error per student is not error per classroom. Forward-in-time prediction – the test data are later than the training data (L3.1), not mixed with them. And anyone who wants to exploit the dependence – to predict a student from their friends – moves to relational models (collective inference, hierarchical models; the tools map, section 4), and reports that the claim is about a network with the same structure, not about an individual subject. The site's check: ask "if I take this subject from the test set, who in the training set is too similar to them?" – and if there is an answer, the split is wrong.

7. What to report

The complete paragraph, in order: the metric and why (the cost of error; the threshold chosen); the split – the test proportion, stratification, by which unit, seed; cross-validation – how many folds, mean and standard deviation for every model tested, including those that failed; the baseline – a dummy classifier and a regression on the known variables; the number that matters is the difference from it, not the metric itself; the test set – once, the metric and a confidence interval, the confusion matrix, and the gap from cross-validation; subgroups – Géron requires checking that the model works not only on average but "across all categories of districts, whether they're rural or urban, rich or poor, northern or southern, minority or not", and if it fails on a whole category, not deploying it for that category "because it may do more harm than good" (Géron, 2022, Chapter 2, "Analyzing the Best Models and Their Errors"); leakage – an explicit answer for each type in the taxonomy, in the style of a "model info sheet"; and what the model does not say – prediction is not cause. Everything can be reproduced from the attached notebook – the same rule as for all of the site's calculators: an undocumented computational result is not a finding.

And what it looks like in the report. Chapter 11 of Truong is the only chapter in the five books that deals with writing. His structure for an academic paper is the familiar one – title, abstract, introduction, literature review, method, results, discussion and conclusions – but the method has four subsections that regression does not have: the data source and description of the variables (including each variable's role – target or input – and its scale); the data mining process; "the algorithms and configurations" – a table giving each model's hyperparameters, as entered (maximum tree depth, splitting rule, stopping rule, the proportion of the "overfitting prevention set"), "the easiest way to do so is to summarize the configurations in a table"; and the evaluation metrics – which ones, and why (Truong, 2026, pp. 396, 414-415). In the results: a comparison of all models on the same set, identification of the champion model (champion model) – "which is only the champion among the models we trained and validated on the given data", and may be the relatively best rather than the best (p. 189) – and then, for the champion only, the metrics on all sets side by side, and a comparison of variable importance. One difference from this page is worth knowing: Truong writes that "there is no need to report the training sample results separately" (p. 188), whereas here the gap between training and cross-validation is reported – because it is the evidence for overfitting, and because Truong himself measures reliability by it (section 2). And what not to take from him literally: the rules of thumb – "a misclassification rate of 5% or less is considered excellent, and between 5% and 10% – good", and for accuracy, sensitivity, specificity and F1 "95% and above is excellent, 90-95% good" – are explicitly presented by him as personal opinion, after the sentence "there are no standard threshold values for these metrics; the decision depends on the industry and the tolerance for error" (pp. 199-200). In predicting human outcomes from background variables (section 4) these values do not exist, and a researcher who reaches them – goes back to section 5.

8. And how to do it in R

Two parts. A. The metrics of section 1 on out-of-sample predictions: logistic regression in tenfold cross-validation on the example file of the machine learning page (320 students, dropout 11.6%), and then the confusion matrix with all the metrics (caret's confusionMatrix, mode = "everything" – sensitivity, specificity, precision, F1, and the "no information rate", which is the accuracy of the dummy classifier), the ROC curve with a confidence interval for the AUC, the "best" threshold by Youden, and the precision-recall curve from the same predictions (pROC). The list of metrics – accuracy, sensitivity, specificity, precision, false positive and false negative rates – is that of Gutierrez (2015, Chapter 7, "Measuring Classification Performance"), who also warns: "if the performance of the algorithm is too good to be true, data leakage may be the cause" (Chapter 7, "Data Leakage"). B. A reproduction of the experiment of section 5 – Hastie et al.'s "the wrong and right way to do cross-validation" (2009, pp. 245-246) – in a hundred lines: 50 subjects, 5,000 random predictors, filtering of the 100 most correlated, one nearest neighbor, five folds. The path C:\Rdata\ – the site's convention.

# model_evaluation.R - performance measures and validation in R: confusion matrix, ROC, cross-validation - and leakage in numbers
library(readxl); library(caret); library(pROC); library(class)

# --- A. The measures - on out-of-sample predictions, not on the training set ---
d <- read_excel("C:/Rdata/DATA_ml_example_eng.xlsx")
d$dropout <- factor(d$dropout, levels = c(0, 1), labels = c("no", "yes"))
d$gender  <- factor(d$gender, levels = c(1, 2), labels = c("male", "female"))
d$id <- NULL
ctrl <- trainControl(method = "cv", number = 10, classProbs = TRUE,
                     summaryFunction = twoClassSummary, savePredictions = "final")
set.seed(1)
m <- train(dropout ~ . - school_id, data = d, method = "glm", family = binomial,
           metric = "ROC", trControl = ctrl, preProcess = c("medianImpute", "center", "scale"),
           na.action = na.pass)
cv <- m$pred                                   # one prediction per observation - from the fold where it was held out
confusionMatrix(cv$pred, cv$obs, positive = "yes", mode = "everything")   # at threshold 0.5
r <- roc(cv$obs, cv$yes, levels = c("no", "yes"), direction = "<")
auc(r); ci.auc(r)                              # AUC and confidence interval
coords(r, "best", best.method = "youden", ret = c("threshold", "sensitivity", "specificity", "precision"))
# precision-recall curve - for a rare class - from the same predictions:
pr <- coords(r, x = seq(0.05, 0.6, by = 0.05), ret = c("threshold", "recall", "precision"))
pr

# --- B. The wrong and right way to do cross-validation (Hastie et al., 2009, pp. 245-246) ---
# 50 subjects in two equal classes, 5,000 random predictors - true error 50%.
set.seed(20260920)
N <- 50; P <- 5000; K <- 5; n_keep <- 100
y <- factor(rep(c("A", "B"), each = N / 2))
X <- matrix(rnorm(N * P), N, P)
folds <- createFolds(y, k = K)
screen <- function(X, y, n) {                   # the 100 predictors most correlated with the class
  r <- abs(cor(X, as.numeric(y)))
  order(-r)[1:n]
}
# wrong way: screen on the whole sample, then cross-validate
keep_all <- screen(X, y, n_keep)
err_wrong <- sapply(folds, function(te) {
  tr <- setdiff(seq_len(N), te)
  mean(knn(X[tr, keep_all], X[te, keep_all], y[tr], k = 1) != y[te])
})
# right way: screening inside each fold, on that fold's training data only
err_right <- sapply(folds, function(te) {
  tr <- setdiff(seq_len(N), te)
  keep <- screen(X[tr, ], y[tr], n_keep)
  mean(knn(X[tr, keep], X[te, keep], y[tr], k = 1) != y[te])
})
c(wrong = mean(err_wrong), right = mean(err_right))   # ~0.03 vs ~0.50

What came out (R 4.3.3, caret 6.0-94, pROC 1.18.5, class 7.3-22; the full output – qa/ml/ev_out.txt). A. At the 0.5 threshold: overall accuracy .884 – exactly the no information rate (.884), that is, the model is no better than the dummy classifier at this threshold; sensitivity .03, precision .50, F1 .05. The same model, the same predictions: AUC = .66, confidence interval [.57, .76]; the Youden threshold .13 – sensitivity .57, specificity .72, precision .21; and across thresholds, from .05 to .50, sensitivity falls from .89 to .03 and precision rises from .14 to .50 – this is the trade-off of section 1 in numbers, and what a single metric hides. B. The wrong way: error 2%; the right way: 60% – on completely random predictors, whose true error is 50% (and 60% with 50 subjects is 50% within the noise of five folds). Hastie et al. reported 3% over 50 simulations; here one simulation, seed 20260920. Anyone who filters items by correlation with the outcome before the split gets the first number.

Common mistakes

Reporting overall accuracy for a rare class. 94% accuracy in predicting the dropout of 6% of students is the dummy classifier. Report a confusion matrix, precision and recall – against the dummy classifier (section 1).

Standardizing, imputing or selecting features on all the data – and then splitting. The most common leakage (L1.2-L1.3). The habit from SPSS – "first clean the file" – is the problem. The pipeline is fitted on the training data only (section 5).

Splitting rows when the unit is a subject, classroom or family. Three measurements of the same person in training and in testing – and the model "recognizes" them instead of predicting (section 6). Split by the unit about which the claim is made.

Tuning on the test set. Run, see that the result is not good, change, run again. After the second time the test set is a validation set, and the number reported is optimistic (section 2).

One number without spread. AUC = .78 from one fold can be .71 in another fold. Mean and standard deviation across the folds, confidence interval on the test set (section 3).

Being impressed by the model without a baseline. AUC = .78 sounds excellent until you see that a logistic regression on four variables gives .76. The number is the difference – and Salganik et al. (2020) showed how small it can be.

Treating the test R² as a regression R². On new data it can be zero or even negative; 0.2 in the Fragile Families Challenge was the best of 160 teams.

Balancing the classes by oversampling – and reporting probabilities. After SMOTE or duplication, the predicted probability for the rare class is overestimated, and the AUC does not rise (van den Goorbergh et al., 2022). Shift the threshold, or enter a decision matrix (section 1); and if balancing – only inside the training fold.

Reading a metric that is too high as success. When the result far exceeds the literature and the baseline, the first suspicion is leakage (section 5) – a feature that is the target in disguise, or a subject who is on both sides. "Treat results… as provisional" (Kapoor & Narayanan, 2023, R2).

Checklist – before reporting a prediction result

References

Bishop, C. M. (2006). Pattern recognition and machine learning. Springer.

Géron, A. (2022). Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems (3rd ed.). O'Reilly Media.

Getoor, L., & Taskar, B. (2007). Introduction. In L. Getoor & B. Taskar (Eds.), Introduction to statistical relational learning (pp. 1-12). MIT Press. https://doi.org/10.7551/mitpress/7432.003.0003

Gutierrez, D. D. (2015). Machine learning and data science: An introduction to statistical learning methods with R. Technics Publications.

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7

IBM Corporation. (2016). IBM SPSS Modeler 18.0 source, process, and output nodes. IBM. https://public.dhe.ibm.com/software/analytics/spss/documentation/modeler/18.0/en/ModelerSPOnodes.pdf

Ishii, K., Ueda, N., Maeda, E., & Murase, H. (2026). Pattern recognition and machine learning for self-study I: Supervised learning. Springer. https://doi.org/10.1007/978-981-95-1478-6

Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), Article 100804. https://doi.org/10.1016/j.patter.2023.100804

Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1-26. https://doi.org/10.18637/jss.v028.i05

Neville, J., & Jensen, D. (2007). Relational dependency networks. In L. Getoor & B. Taskar (Eds.), Introduction to statistical relational learning (pp. 239-268). MIT Press. https://doi.org/10.7551/mitpress/7432.003.0010

Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., Datta, D., Davidson, T., Filippova, A., Gilroy, C., Goode, B. J., Jahani, E., Kashyap, R., Kirchner, A., McKay, S., . . . McLanahan, S. (2020). Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences, 117(15), 8398-8403. https://doi.org/10.1073/pnas.1915006117

Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289-310. https://doi.org/10.1214/10-STS330

Taskar, B., Abbeel, P., Wong, M.-F., & Koller, D. (2007). Relational Markov networks. In L. Getoor & B. Taskar (Eds.), Introduction to statistical relational learning (pp. 175-200). MIT Press. https://doi.org/10.7551/mitpress/7432.003.0008

Truong, D. (2026). Demystifying AI: Data science and machine learning using IBM SPSS Modeler. Chapman and Hall/CRC. https://doi.org/10.1201/9781003468196

van den Goorbergh, R., van Smeden, M., Timmerman, D., & Van Calster, B. (2022). The harm of class imbalance corrections for risk prediction models: Illustration and simulation using logistic regression. Journal of the American Medical Informatics Association, 29(9), 1525-1534. https://doi.org/10.1093/jamia/ocac093

Wendler, T., & Gröttrup, S. (2021). Data mining with SPSS Modeler: Theory, exercises and solutions (2nd ed.). Springer. https://doi.org/10.1007/978-3-030-54338-9