Machine learning for the social researcher: predicting, not only explaining

On this page

machine learning is, in the definition with which Géron opens his book, "the science (and art) of programming computers so they can learn from data"; in Arthur Samuel's classic formulation (1959) – "the field of study that gives computers the ability to learn without being explicitly programmed"; and in Tom Mitchell's engineering formulation (1997) – a program "learns from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E" (both as cited in Géron, 2022, chapter 1). To the researcher coming from statistics these definitions sound all too familiar: regression also "learns" coefficients from data. The difference is not in the algorithm – logistic regression is the same logistic regression in both worlds – but in the goal and therefore in the validation: the statistics of the site's calculators asks which variable has an effect and how much, and validates this with a coefficient, a standard error and significance; machine learning asks what will happen to a new observation, and validates this in one way only – on data the model has not seen. The page is built around the practical textbook most widely used in the field, Aurélien Géron's Hands-On Machine Learning (third edition, 2022), and pairs it with the other side of the same coin – The Elements of Statistical Learning by Hastie, Tibshirani and Friedman (second edition, 2009), the book statisticians wrote to explain the same algorithms as statistical models: "this book is about learning from data," and "statistical learning plays a key role in many areas of science, finance and industry" (Hastie et al., 2009, p. 1). The third book, by Christopher Bishop (2006), adds the probabilistic perspective: pattern recognition "has its origins in engineering, whereas machine learning grew out of computer science," but "these can be viewed as two facets of the same field" – and in it Bayesian methods have moved "from a specialist niche to become mainstream" (Bishop, 2006, preface). A fourth, from the engineering tradition of pattern recognition: the self-study textbook by Ishii, Ueda, Maeda and Murase (2026) – written "for those trying to learn these topics on their own," declaredly prefers "classical topics" over "the most recent technologies," because "acquiring the fundamental knowledge… will significantly improve the efficiency of learning the latest technologies," and ends with one recommendation: work with real data (Ishii et al., 2026, preface). And a fifth, from the software side: Dothang Truong's textbook (2026) for SPSS Modeler, written out of the recognition that "learners from diverse fields, such as business, marketing, finance, aviation, social sciences and humanities, may feel left behind" in a field that emphasizes programmers, and that today "many applications allow users to build machine learning models without writing a single line of code" (Truong, 2026, preface). Géron says what to do, Hastie and colleagues say why it works and when it does not, Bishop says what the model assumes about the probabilities, Ishii and colleagues say what was already known before the term became fashionable, and Truong says what it is called in the menu; the page adds to the four of them what the textbooks – written for engineers and statisticians – do not say: what all this does to social research (section 1), and what happens when one tries to predict human lives (Salganik et al., 2020). The order: the distinction between explanation and prediction (1), a map of the types of learning and the challenges (2), the end-to-end project (3), a summary of validation (4) – the metrics, the test set, cross-validation, data leakage and the limits of prediction are a separate page, Performance metrics and validation – the models themselves (5), neural networks (6), the tools (7), a workflow recipe (8) – and how to do it in R (9) and in SPSS Modeler (10). A note on citation: the digital copy of Géron has no page numbers, so references to it are by chapter and section; Hastie and colleagues, Bishop, Ishii and colleagues and Truong – by page.

1. Two cultures: explain or predict

The distinction statistics did not make. Leo Breiman, the inventor of random forests, opened the century with an article explaining why statisticians and engineers do not speak the same language: "There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown" – and the statistical community, he said, "has been committed to the almost exclusive use of data models" (Breiman, 2001a, abstract). Galit Shmueli turned the distinction into a practical map: explanatory modeling is the use of a model to test a causal theory, and in the social sciences "statistical methods are used nearly exclusively for testing causal theory" – and in practice, almost always, with "association-based models applied to observational data. Regression models are the most common example" (Shmueli, 2010, Section 1.1); predictive modeling is "the process of applying a statistical model or data mining algorithm to data for the purpose of predicting new or future observations" (section 1.2). The common mistake is the assumption "that models with high explanatory power are inherently of high predictive power" (abstract) – and it is not true: explanatory power and predictive power are two dimensions, not two ends of one continuum (section 4.2); validation also differs – in explanatory modeling one checks that the model represents the theory and that it fits the data, and in predictive modeling one checks generalization to new data, and "the greatest danger to generalization is overfitting the training data" (section 2.6.1). Géron says the same thing from the other side of the fence: most machine learning tasks are about making predictions, and "having a good performance measure on the training data is good, but insufficient; the true goal is to perform well on new instances" (Géron, 2022, Chapter 1, "Instance-Based Versus Model-Based Learning").

And what this does to psychology and sociology. Yarkoni and Westfall argue that psychology's near-total focus on causal explanation has left it with "complex theories of psychological mechanism" that have "little (or unknown) ability to predict future behaviors," and propose adopting principles from machine learning – and according to them, it is precisely an emphasis on prediction that will lead to a better understanding of behavior (Yarkoni & Westfall, 2017, abstract). Hofman, Watts and colleagues – among them Susan Athey, Jon Kleinberg, Matthew Salganik and Tal Yarkoni – propose in Nature a two-dimensional schema for every study: the extent to which it explains (identifying and estimating causal effects) and the extent to which it tests predictions of outcomes, and call on computational social science for "integrative modeling" of the two (Hofman et al., 2021, abstract). Molina and Garip, for sociologists, list the uses: distilling measures from new sources such as text and images, characterizing heterogeneity in the population, improving causal inference, and predictions for policy and for theory development (Molina & Garip, 2019, abstract).

And what happens when one tries. The largest experiment in predicting human lives is the "Fragile Families Challenge": 160 research teams received data on 4,242 families – 12,942 variables per family, collected from the child's birth to age 9 – and were asked to predict six outcomes at age 15: grade point average, "grit" (perseverance), eviction from the home, material hardship, layoff of the primary caregiver and the caregiver's participation in job training. The result: even the best predictions were not accurate – R² on the holdout data of about 0.2 for material hardship and grade point average and about 0.05 for the four other outcomes – and they were only slightly better than a simple benchmark model: linear or logistic regression on four variables chosen by a domain expert (three about the mother at birth – race/ethnicity, marital status and education – and a measurement of the same outcome, or one close to it, at age 9). Prediction error was strongly related to the family being predicted and only weakly to the method (Salganik et al., 2020, abstract and Results section). The conclusion of the authors – 112 researchers, led by Matthew Salganik (author of Bit by Bit in the tools map) – is that life-course researchers must "reconcile the idea that they understand life trajectories with the fact that none of the predictions was very accurate" (Section Significance), and resolve what they call "the understanding/prediction paradox": the same data yielded more than 750 explanatory articles – and do not predict (Section Discussion). For the researcher: the two cultures do not replace each other. An explanatory model that predicts nothing deserves suspicion; a predictive model that no one understands deserves a different suspicion (section 5, "White box and black box"); and most of the time the answer is to report both.

2. Map of the field: types of learning and what can go wrong

Géron classifies machine learning systems along three axes (Géron, 2022, Chapter 1, "Types of Machine Learning Systems"). The first – the degree of supervision. In supervised learning (supervised) the training set includes the desired answers – the labels – and the two typical tasks are classification (spam / not spam) and regression (the price of a car by mileage, age and make); the boundary between them is permeable – logistic regression is a regression model used for classification, because it outputs a probability. In unsupervised learning (unsupervised) there are no labels, and the tasks are clustering (groups of similar visitors), visualization and dimensionality reduction (merging correlated features, such as mileage and age, into one feature of "wear and tear"), anomaly detection and association rule learning (people who buy barbecue sauce and chips also buy steak) – for the social researcher these are the factor analysis and cluster analysis already familiar, and they have a page of their own: Unsupervised learning – and calculators: Cluster analysis and Exploratory factor analysis. In between – semi-supervised learning (few labels, a lot of unlabeled data), self-supervised learning (the model generates labels for itself – hides part of an image and learns to reconstruct it – and is then fine-tuned for another task: this is the principle of the language models on the NLP page), and reinforcement learning (an agent, actions, rewards). The second – batch versus online: a model trained once on all the data, versus a model that learns on the fly from a stream; what concerns the researcher is the remark about "model rot" (model rot) – a model's performance decays over time "simply because the world continues to evolve while the model remains unchanged." The third – instance-based versus model-based: learning by heart and generalizing by similarity (nearest neighbors), or building a model of the examples. Géron's example is sociological: life satisfaction by GDP per capita (OECD and World Bank data). A straight line gives Cyprus 6.30; the three nearest neighbors in GDP – Israel (7.2), Lithuania and Slovenia (5.9 each) – give an average of 6.33. Two approaches, the same answer, and neither of them has seen Cyprus. But "neighbors" stop being close when there are many variables – the curse of dimensionality: in ten dimensions, capturing 10% of the data requires covering 80% of the range of each variable, so every "local" neighborhood is actually most of the space (Hastie et al., 2009, pp. 22-23); and therefore any method that tries to fit a locally varying function in small neighborhoods "will run into problems in high dimensions" (p. 33). "Our geometrical intuitions, formed through a life spent in a space of three dimensions, can fail badly" in high-dimensional spaces – in a sphere in twenty dimensions almost all the volume sits in a thin shell near the surface (Bishop, 2006, p. 36). For a researcher with a survey of 80 items and 300 participants this is not a technical detail – it is the reason why regularized models and trees (section 5) work there, and nearest neighbors do not.

What can go wrong: bad data and a bad model (ibid., "Main Challenges of Machine Learning"). Too little data – "even for very simple problems you typically need thousands of examples"; the famous paper by Banko and Brill (2001) showed that very different algorithms, including simple ones, reach almost identical performance when given enough data – but Géron himself qualifies that "small and medium-sized datasets are still very common," and therefore "don't abandon algorithms just yet." Nonrepresentative data – the example is our example: the Literary Digest poll in 1936, about ten million recipients, 2.4 million responses, a prediction of 57% for Landon; Roosevelt won with 62%. The lists (telephone subscribers, club members) were skewed toward the wealthy, and fewer than a quarter responded – sampling bias and nonresponse bias, in exactly the terms of a research methods course. Poor-quality data – outliers, noise, missing values: "most data scientists spend a significant part of their time" cleaning. Irrelevant features – "garbage in, garbage out"; feature engineering is selecting, extracting and collecting new ones. And on the model side: overfitting – a model too complex relative to the amount of data and the noise in it, which detects patterns in the noise itself; the example: in the training data, all the countries with a w in their name have life satisfaction above 7 – New Zealand (7.3), Norway (7.6), Sweden (7.3) and Switzerland (7.5) – "how confident are you that the w-satisfaction rule generalizes to Rwanda or Zimbabwe?" The solutions: a simpler model, more data, less noise – and regularization: constraining the model, for example forcing the slope to stay small, controlled by a hyperparameter. Bishop demonstrates the same thing with a polynomial: one of order 9 on ten points passes exactly through all of them, with zero training error, "but the fitted curve oscillates wildly" – "and this behavior is known as overfitting" (p. 6); "for a given model complexity, the overfitting problem becomes less severe as the size of the data set increases," and the rough rule of thumb he cites – and distances himself from – is that the number of observations should be no less than 5 to 10 times the number of adaptive parameters in the model (p. 9). The penalty on the size of the coefficients is called "shrinkage methods" in statistics, the quadratic case – ridge regression, and in neural networks – "weight decay" (p. 10): three names for one idea. And what Bishop adds as a fundamental claim: overfitting is "a general property of maximum likelihood," and in a Bayesian approach – in which "the effective number of parameters adapts automatically to the size of the data set" – it can be avoided even in a model whose number of parameters greatly exceeds the number of observations (p. 9). And the same question – how many observations are needed – pattern recognition asks from the side of the features: "unfortunately, there is no general answer," but there is a known phenomenon – the Hughes phenomenon (Hughes phenomenon): as long as the number of features d is small relative to the number of observations n, adding a feature improves classification; when d grows large enough relative to n, "statistical reliability decreases, and contrary to expectations, classification performance deteriorates" – the reason is "adding too many features, while forgetting that the number of learning observations is finite" (Ishii et al., 2026, pp. 84, 86-87). For the researcher: another item in a scale is not necessarily more information for the model. And underfitting – a model too simple for the structure of the data (a straight line for life satisfaction), which fails already on the training data. Finally, the "no free lunch" theorem (Wolpert, 1996, as cited in Géron, Chapter 1): whoever assumes nothing about the data has no reason to prefer one model over another; every choice of model has a hidden assumption (a straight line = "the distance from the line is noise"), and the only way to know which is better is to compare – on data not used for training.

3. The end-to-end project: eight steps

Géron's chapter 2 is one complete project – predicting the median house price in 20,640 census districts in California (1990 census) – and the book's appendix A turns it into an eight-step checklist: frame the problem, get the data, explore them, prepare them, shortlist promising models, fine-tune, present the solution, and launch and maintain (Géron, 2022, appendix A). What distinguishes this pipeline from ordinary statistical analysis is not the algorithm but four decisions made before looking at the data.

A. Framing: what is the goal, what is the metric, what exists today. The first question is what the goal is – "building a model is probably not the end goal" – because it determines the framing (supervised? regression? batch?), the algorithm and the performance measure; the second is what the current solution is – in the example, experts who estimate manually and are off "by more than 30%": that is the baseline the model must beat. Then check assumptions: if the system that will receive the prediction converts prices into categories ("cheap / medium / expensive"), the problem is classification and not regression – "you don't want to find this out after working on a regression system for months" (Chapter 2, "Frame the Problem", "Check the Assumptions"). The performance measure is chosen now, not at the end: for regression – RMSE, which penalizes large errors, or MAE when there are many outliers; for classification – according to the cost of error (Performance metrics and validation, section 1 and Table 1 there).

B. The test set – before exploration, not after it. "It may seem strange to voluntarily set aside part of the data at this stage," Géron writes, but "your brain is an amazing pattern detection system, which also means that it is highly prone to overfitting" – data snooping bias (data snooping bias; Chapter 2, "Create a Test Set"). About 20% for testing, in a stable and stratified split; the details – why three sets and not two, and by which unit to split – in Performance metrics and validation, sections 2 and 6.

C. Exploration and preparation – on the training set only, and in functions. The exploration is familiar: distributions, Pearson correlations with the target (in the example, median income correlates 0.69 with house price), scatter plots, and feature combinations (rooms per household instead of total rooms). The preparation – in code, not by hand, for four reasons: reproduction on new data, reuse in future projects, running in a live system, and easy experimentation with alternatives (Chapter 2, "Prepare the Data"). The three steps: missing values – drop rows, drop a variable, or impute (impute) a value – median, mean, mode – and the rule that distinguishes machine learning from SPSS habits: the median is computed on the training set and saved, so it can be applied to the validation set, the test set and any new data; categorical variables – one-hot encoding (one-hot, dummy variables); and feature scaling – "with few exceptions, machine learning algorithms don't perform well when the input numerical attributes have very different scales" (total rooms 6 to 39,320, income 0 to 15): min-max to the range 0-1, or standardization (mean 0, standard deviation 1), which is less sensitive to outliers. And the warning, in Géron's own capitals: "never use fit() or fit_transform() for anything else than the training set" (Chapter 2, "Feature Scaling and Transformation") – section 4 explains why this is not pedantry. Decision trees are the convenient exception: they require no feature scaling at all (chapter 6).

D. Train, compare – with k folds – and fine-tune. Linear regression on the training data gives an RMSE of about 68,600 dollars when most prices are between 120 and 265 thousand – underfitting. A decision tree gives an RMSE of 0.0 – "Wait, what!? No error at all?" – complete overfitting, which cannot be seen without data the model has not seen. The tool is cross-validation (k-fold cross-validation): split the training set into 10 folds, train 10 times on 9 and test on the tenth, and get not only an estimate but also its precision – a mean and a standard deviation. The tree drops to 66,868 (±2,061), almost like the regression (69,858 ±4,182); a random forest gives 47,019 (±1,034) – but on the training set itself about 17,474, meaning still considerable overfitting (Chapter 2, "Better Evaluation Using Cross-Validation"). How many folds, the one-standard-error rule, and what to do with a small sample – Performance metrics and validation, section 3. The guideline: try models from several families without tuning, shortlist 2-5 promising ones – and only then tune hyperparameters, by grid search (all combinations) or, when the space of combinations is large, by randomized search (a fixed number of combinations; appendix A: prefer random unless there are few values). After tuning, look into the model: a random forest reports feature importance – in the example, median income comes first (0.19) – and analyze the errors; and Géron adds a requirement every social researcher will recognize: make sure the model works not only on average but "on all categories of districts, whether they're rural or urban, rich or poor, northern or southern, minority or not" – and if it fails on an entire category, do not deploy it for that category, "because it may do more harm than good" (Chapter 2, "Analyzing the Best Models and Their Errors").

E. The test set – once, and after it hands off. The final model is measured on the test set with a confidence interval, and then: "you must resist the temptation to tweak the hyperparameters to make the numbers look good on the test set; the improvements would be unlikely to generalize to new data" (Chapter 2, "Evaluate Your System on the Test Set"). The presentation – what was learned, what worked and what did not, which assumptions, what the limitations are; and in the example, honestly: the system is "not much better" than the experts who were off by 30%. The last two steps – launch, monitoring and retraining – belong to live systems; what remains of them for the researcher is only the remark about model rot: a model is tied to the data of its time.

4. How to know the model is good – in brief

The full version – in Performance metrics and validation; here are the four lines everything else rests on. The metric according to the cost of error: overall accuracy (accuracy) is misleading for a rare class – a dummy classifier that says "no" to everything gets 90% when a tenth of the observations are positive, "it beats Nostradamus" (Géron, 2022, Chapter 3); report a confusion matrix, precision and sensitivity, and the threshold. Data the model has not seen: training error goes down to zero with complexity, "however a model with zero training error is overfit to the training data and will typically generalize poorly" (Hastie et al., 2009, pp. 220-221) – hence training, validation and test, and cross-validation that gives a mean and a standard deviation. Bias and variance: generalization error = bias² + variance + noise; complexity lowers bias and raises variance – "this is why it is called a trade-off" (Géron, Chapter 4). Leakage and limit: Kapoor and Narayanan found data leakage in 294 papers in 17 fields, and in civil war prediction, "when the errors are corrected, complex machine learning models do not perform substantively better than decades-old LR models" (Kapoor & Narayanan, 2023, Section Summary); and above everything sits the Bayes error – "the unavoidable error" of the feature system (Ishii et al., 2026, p. 101), the limit that Salganik and colleagues' 160 teams met at R² = 0.2. A result far higher than the baseline and the literature is a sign of leakage before it is a sign of a breakthrough.

5. The models: from a straight line to a random forest

Géron ends the example project with the advice that "it is probably preferable to be comfortable with the overall process and know three or four algorithms well rather than to spend all your time exploring advanced algorithms" (Chapter 2, "Try It Out!"). Table 1 arranges the families worth knowing, and what precedes it explains the three most important for the social researcher – the regressions they already have, the tree, and the forest.

The regressions – the same formulas, one addition. Linear and logistic regression are in Géron (chapter 4) exactly what they are on the site's logistic regression page – a weighted sum of the features, and in the logistic case – its logistic as a probability, and a threshold (usually 50%) that turns it into a classifier; the extension to several classes is softmax regression. What the book adds is regularization: ridge regression (ridge; a penalty on the sum of squared coefficients – shrinks them), lasso (lasso; a penalty on the sum of their absolute values – sets some of them to zero, that is, selects features), elastic net (a combination), and early stopping. Hastie and colleagues – Tibshirani is the inventor of the lasso – put precisely what it does: when the budget for the sum of coefficients is reduced, "some of the coefficients will be exactly zero. Thus the lasso does a kind of continuous subset selection" (Hastie et al., 2009, p. 69). For the researcher used to regression "as it is": when the goal is prediction, a model with shrunken, "wrong" coefficients sometimes predicts better than a model with the unbiased coefficients – this is the trade-off of section 4 in action, and a regularized coefficient is not a coefficient that can be interpreted as an effect.

Decision tree – a white box. A tree (chapter 6) is a sequence of questions: is the petal length less than 2.45 cm? – if so, Iris setosa; if not, is its width less than 1.75? – versicolor, otherwise virginica. The algorithm (CART) chooses at each node one feature and one threshold that split the set into the two purest subsets (weighted by their size), with the Gini index by default or with entropy – "most of the time it does not make a big difference," Gini is faster. The class probability in a leaf is the proportion of observations of that class in it (49 out of 54 = 90.7%). The advantages: almost no data preparation, no feature scaling, and interpretability – "nice, simple classification rules that can even be applied manually" – hence a "white box model," as opposed to forests and networks, which are considered "black boxes": they predict excellently, and every calculation they made can be checked, "yet it is usually hard to explain in simple terms why the predictions were made." The disadvantages: a tree is a nonparametric model – the number of parameters is not fixed in advance, and it will cling to the data to the point of overfitting if not constrained (maximum depth – the default: unlimited – number of leaves, minimum observations per leaf); it loves boundaries perpendicular to the axes, and is therefore sensitive to rotation of the data; and above all – high variance: a small change in the data or the hyperparameters gives a very different tree, and even retraining on exactly the same data may give a different tree, because the algorithm randomly samples the features it examines at each node (unless a seed is set; chapter 6, "Decision Trees Have a High Variance"). The statisticians agree from both directions: "One major problem with trees is their high variance. Often a small change in the data can result in a very different series of splits, making interpretation somewhat precarious" – because of the hierarchical structure, an error in the top split propagates to everything below it (Hastie et al., 2009, p. 312); and yet, "of all the well-known learning methods, decision trees come closest to meeting the requirements" of an "off-the-shelf" method for data mining – fast, interpretable (when the tree is small), handling a mixture of numeric and categorical variables and missing values, immune to monotone transformations and to outliers in the predictors (p. 352). On the same page is also the remark researchers need: data mining applications "generally require interpretable models. It is not enough to simply produce predictions" – and therefore "black box methods such as neural networks" are far less useful there (pp. 351-352). Whoever reports "the tree found that…" is reporting one tree out of many possible ones.

Random forest – the wisdom of crowds, at the price of interpretability. The solution to variance is to average: an "ensemble" of predictors (chapter 7). Géron's intuition is a biased coin that lands on "heads" with probability 51% – in a single toss almost random, but in 1,000 tosses the probability of a majority of "heads" is about 75%, and in 10,000 above 97%: the law of large numbers – provided the predictors are numerous enough and diverse enough. Diversity is achieved in two ways: bagging (bagging, "bootstrap aggregating"; Breiman, 1996, as cited in Géron) – each tree is trained on a sample drawn with replacement from the training set (on average about 63% of the observations; the remaining 37%, "out of the bag," serve for free validation – out-of-bag evaluation – without a separate validation set); and in addition, in a random forest, at each node the tree chooses the best split only from a random subset of the features (by default the square root of n out of n) – "which again trades a higher bias for a lower variance, generally yielding an overall better model" (Chapter 7, "Random Forests"). The formal source is Breiman (2001b): a combination of trees each of which depends on a random vector sampled independently, and its generalization error converges to a limit as the number of trees grows (abstract) – that is, adding trees does not produce overfitting. Hastie and colleagues sharpen this claim: it is true that "increasing B does not cause the random forest sequence to overfit," but the limit to which it converges "can overfit the data" – an average of fully grown trees may be too rich a model; in their experience, growing full trees "seldom costs much" and saves a tuning parameter (Hastie et al., 2009, p. 596). And their general assessment, which explains why the forest is the baseline of section 8: "on many problems the performance of random forests is very similar to boosting, and they are simpler to train and tune" (p. 587); the out-of-bag error estimate is "almost identical" to N-fold cross-validation (p. 593). The forest also measures feature importance: how much each feature reduces impurity on average over all the nodes that use it, weighted by the number of observations, and summed to 1 (for the irises: petal length 44%, its width 42%, the sepals 11% and 2%). What the forest does not give is direction: importance is not a coefficient, and does not say whether the feature raises or lowers the probability. A sister family: boosting – "one of the most powerful learning ideas introduced in the last twenty years" (Hastie et al., 2009, p. 337); Breiman called AdaBoost with trees "the best off-the-shelf classifier in the world" (as cited ibid., p. 340) – trees in sequence, each correcting its predecessor (AdaBoost reweights misclassified observations; gradient boosting fits each tree to the residuals of its predecessor; a histogram-based implementation for large data – hundreds of times faster; XGBoost, LightGBM and CatBoost outside scikit-learn); and stacking (stacking) – a model that learns to weight the predictions of the others.

White box and black box – and when a black box is not allowed. Cynthia Rudin argues that "it is a myth that there is necessarily a trade-off between accuracy and interpretability": the belief that more complex models are more accurate "is often not true, particularly when the data are structured, with a good representation in terms of naturally meaningful features" – and survey data are exactly that. Explanations fitted onto a black box after the fact (a second model that imitates the first) "must be wrong": if they were completely faithful to the original model, the explanation would equal the model and the original would not be needed; her example is COMPAS – a proprietary model for predicting recidivism in United States courts, which ProPublica accused of racial bias on the basis of a linear explanation model it built for it, while the model itself is probably nonlinear (Rudin, 2019, Section 2). Her conclusion: for high-stakes decisions about people – justice, health, welfare – interpretable models should be built from the outset, rather than explaining black boxes. For the researcher: on survey data of a few hundred participants, a regularized logistic regression or a shallow tree will usually predict as well as the forest – and will also say why. The forest is run as a baseline: if it is much better, there are nonlinearities or interactions in the data that the interpretable model is missing – and that is what gets reported.

6. Neural networks – when yes and when no

An artificial neural network (chapter 10) was inspired by the brain, but "just like planes were inspired by birds, they don't have to flap their wings to fly"; the first model is that of McCulloch and Pitts (1943), followed by two "winters" – the 60s and the 90s, when support vector machines seemed more promising. The current wave is different, according to Géron, for four reasons: the amount of data, computing power (Moore's law, GPU cards that the gaming industry produced by the millions, the cloud), small but decisive improvements in training algorithms, and the realization that theoretical limitations – such as local optima – are not harmful in practice. And what the network is statistically: "There has been a great deal of hype surrounding neural networks, making them seem magical and mysterious. As we make clear in this section, they are just nonlinear statistical models" – a regression or classification model in two stages (Hastie et al., 2009, p. 392). The basic architecture, the multilayer perceptron, is an input layer, one or more hidden layers, and an output layer – and Bishop remarks that the name "is really a misnomer, because the model comprises multiple layers of logistic regression models" (with continuous nonlinearities) rather than perceptrons (Bishop, 2006, p. 226): for a researcher who knows logistic regression, a network is logistic regressions feeding into one another; "deep" is a network with a deep stack of hidden layers, although "many people talk about deep learning whenever neural networks are involved (even shallow ones)." Training – backpropagation (backpropagation): reverse-mode automatic differentiation (Linnainmaa, 1970) that computes in two passes – forward and backward – how to change each weight to reduce the error, followed by a gradient descent step; and so on, again and again. A single hidden layer can, theoretically, model any function given enough units, but deep networks are more parameter-efficient by orders of magnitude, because they exploit hierarchy in the data – lines, then shapes, then faces – and this is also what makes transfer learning (transfer learning) possible: taking the lower layers of a trained network and training only the upper ones for a new task. "You will rarely have to train such networks from scratch" (Chapter 10, "Number of Hidden Layers").

And when not. The warning is already in the book's preface: "Don't jump into deep waters too hastily… most problems can be solved quite well using simpler techniques such as random forests and ensemble methods… Deep learning is best suited for complex problems such as image recognition, speech recognition, or natural language processing, and it requires a lot of data, computing power, and patience (unless you can leverage a pretrained neural network)" (Géron, 2022, preface, "Roadmap"). For the social researcher the conclusion is sharp: on a survey table – no; on transcripts, posts, images and recordings – yes, but almost always through a pretrained model that someone else trained on millions of examples, and which is fine-tuned or simply applied: DictaBERT, HeBERT and BEREL on the NLP page are exactly such networks, and the guideline there – a manual sample of 100-200 items and intercoder agreement before relying on the output – is section 4 of this page in other words. The tool for writing networks is Keras (Table 2) – "a beautifully designed and simple high-level API" on top of TensorFlow, and most needs will be met by it.

7. The tools

The whole book runs in Python, and its first part in scikit-learn – a library Géron praises for the design of its API: every object is an estimator (estimator) with fit(), some are transformers with transform() and some are predictors with predict(); the hyperparameters are visible, the learned parameters are accessible (with a trailing underscore), the data are NumPy arrays and not private classes, and everything can be chained into a pipeline (Pipeline) – which is also the mechanism that prevents the leakage of section 4, because the whole pipeline is fitted on the training set and applied to the rest (Chapter 2, "Scikit-Learn Design", "Transformation Pipelines"). Table 2 arranges the tools by need; those who do not write code will find three graphical tools in it – two open source and one paid, to which section 10 is devoted. The other statistical tools – R, PSPP, JASP, jamovi – in the tools map, section 8.

8. Workflow recipe: machine learning in social research

The typical case: a survey of a few hundred participants with a binary outcome (dropout, voting, seeking help), or an administrative file of thousands. The question: can the outcome be predicted from the background variables – and how well, compared with what is already known. Seven steps, and each one is documented; the file – in the site's standard data template, row = participant, column = variable, and from SPSS – through From SPSS to Excel.

9. And how do you do it in R

The recipe of section 8, as one script that runs from start to finish on the page's sample data file – DATA_ml_example_eng.xlsx (and a version with Hebrew names, DATA_ml_example_eng.xlsx): 320 students in eight schools, seven background variables and a binary outcome – dropout (11.6%) – with missing values in two variables, in the site's standard data template. The choice of tools follows Gutierrez's tutorial book, which is written around the same question – what a machine learning project looks like in R, from reading the file to evaluating the model: glm with family = binomial for logistic regression, randomForest with ntree and mtry, and importance() for feature importance (Gutierrez, 2015, Chapter 6, "Logistic Regression", "Random Forests"); splitting into training, validation and test – "60% training, 20% cross validation and 20% testing" in his wording – and the test set is "left completely alone, because it is never used in the process" of model selection (Gutierrez, 2015, Chapter 7, "Bias and Variance", "Cross Validation"); and for regularization – glmnet, in which alpha = 0 gives ridge and alpha = 1 lasso, and lambda controls the strength (Chapter 7, "Overfitting"). What the script adds to the book is the caret package (Kuhn, 2008): stratified splitting, cross-validation, tuning and preparation inside the folds – imputation of missing values and standardization in preProcess, fitted on the training fold only – in one interface for all the models, so that leakage cannot happen by mistake (Performance metrics and validation, section 5). The path C:\Rdata\ – the same convention as the R sections on the calculator pages.

# machine_learning.R - machine learning in social research: the seven-step recipe in R
# Data: DATA_ml_example_eng.xlsx - 320 students in 8 schools; outcome dropout (0/1).
library(readxl)        # read Excel
library(caret)         # split, cross-validation, tuning, confusion matrix
library(randomForest)  # random forest
library(pROC)          # ROC curve and AUC confidence interval

d <- read_excel("C:/Rdata/DATA_ml_example_eng.xlsx")
d$dropout <- factor(d$dropout, levels = c(0, 1), labels = c("no", "yes"))
d$gender  <- factor(d$gender, levels = c(1, 2), labels = c("male", "female"))
d$id <- NULL                      # the id column is not a variable

# --- Step 1: goal, metric, baseline ------------------------------------------
# Metric: AUC (no threshold) and sensitivity at the chosen threshold. First baseline - the dummy classifier:
prop.table(table(d$dropout))      # majority-class rate = accuracy of "always no"

# --- Step 2: the split - first, stratified by outcome, fixed seed -----------
set.seed(20260920)
idx   <- createDataPartition(d$dropout, p = 0.8, list = FALSE)
train <- d[idx, ]
test  <- d[-idx, ]                # test set - untouched until step 6

# --- Step 3: preparation inside the pipeline - imputation and scaling within folds ---
ctrl <- trainControl(method = "cv", number = 10, classProbs = TRUE,
                     summaryFunction = twoClassSummary, savePredictions = "final")
prep <- c("medianImpute", "center", "scale")   # caret fits these on the training fold only (na.action = na.pass: NAs imputed in the pipeline)

# --- Step 4: compare models by cross-validation - same folds for all ---------
set.seed(1)
m_base <- train(dropout ~ age + gender + ses, data = train, method = "glm",
                family = binomial, metric = "ROC", trControl = ctrl)           # baseline: background variables
set.seed(1)
m_glm  <- train(dropout ~ . - school_id, data = train, method = "glm",
                family = binomial, metric = "ROC", trControl = ctrl, preProcess = prep, na.action = na.pass)
set.seed(1)
m_tree <- train(dropout ~ . - school_id, data = train, method = "rpart",
                metric = "ROC", trControl = ctrl, preProcess = prep, na.action = na.pass)
set.seed(1)
m_rf   <- train(dropout ~ . - school_id, data = train, method = "rf",
                metric = "ROC", trControl = ctrl, preProcess = prep, ntree = 500, na.action = na.pass)
res <- resamples(list(baseline = m_base, logistic = m_glm, tree = m_tree, forest = m_rf))
t(sapply(res$values[, grep("~ROC$", names(res$values))],
         function(x) c(mean = mean(x), sd = sd(x))))   # mean and SD of AUC over the 10 folds

# --- Step 5: tuning - candidates only ----------------------------------------
set.seed(1)
m_rf_t <- train(dropout ~ . - school_id, data = train, method = "rf",
                metric = "ROC", trControl = ctrl, preProcess = prep, ntree = 500, na.action = na.pass,
                tuneGrid = data.frame(mtry = c(2, 3, 4, 6)))
m_rf_t$results[, c("mtry", "ROC", "ROCSD")]

# --- Step 6: the test set - once -----------------------------------------------
final <- if (max(m_rf_t$results$ROC) - max(m_glm$results$ROC) > m_glm$results$ROCSD) m_rf_t else m_glm
# threshold chosen on the training set's cross-validated predictions (savePredictions) - not on the test set
roc_cv <- roc(final$pred$obs, final$pred$yes, levels = c("no", "yes"), direction = "<")
thr    <- coords(roc_cv, "best", best.method = "youden", ret = "threshold")$threshold
thr                                                          # the threshold to apply to the test set
p_test <- predict(final, test, type = "prob", na.action = na.pass)[, "yes"]
roc_t  <- roc(test$dropout, p_test, levels = c("no", "yes"), direction = "<")
auc(roc_t); ci.auc(roc_t)                                    # AUC and 95% CI on the test set
pred_t <- factor(ifelse(p_test >= thr, "yes", "no"), levels = c("no", "yes"))
confusionMatrix(pred_t, test$dropout, positive = "yes", mode = "everything")   # at the chosen threshold
confusionMatrix(predict(final, test, na.action = na.pass), test$dropout, positive = "yes")$byClass[c("Sensitivity", "Specificity")]  # and at 0.5, for comparison

# --- Step 7: report - plus feature importance (no direction) ----------------
varImp(final)

# --- Dependent data: folds by school instead of by student -----------------
folds_g <- groupKFold(train$school_id, k = 5)
ctrl_g  <- trainControl(method = "cv", index = folds_g, classProbs = TRUE,
                        summaryFunction = twoClassSummary)
set.seed(1)
m_rf_g <- train(dropout ~ . - school_id, data = train, method = "rf",
                metric = "ROC", trControl = ctrl_g, preProcess = prep, ntree = 500, na.action = na.pass)
m_rf_g$results[, c("mtry", "ROC", "ROCSD")]    # compare with m_rf: the gap is what the dependence "contributed"

What came out (R 4.3.3, caret 6.0-94, randomForest 4.7-1.1, pROC 1.18.5; the full output – qa/ml/ml_out.txt). The dummy classifier: 88.4% overall accuracy – this is the floor. In ten-fold cross-validation, mean AUC: the baseline (age, gender, socioeconomic status) .59 (standard deviation .21), logistic regression on all the variables .66 (.22), tree .64 (.14), random forest .64 (.21); tuning the forest (mtry 2-6) reached at most .65. The rule of step 5 chose the logistic regression: the forest did not beat it by a difference exceeding the standard deviation – and "the forest is not better than the regression" is a finding, not a failure. On the test set (63 students, 7 dropouts): AUC = .74, 95% confidence interval [.50, .97] – wide, because seven positive cases are seven. And the threshold: at a threshold of 0.5 the model identifies no dropout at all (sensitivity 0, specificity 1 – "it beats Nostradamus"); the threshold chosen on the cross-validation predictions of the training set, not on the test set, is .12 – and at it sensitivity is .71, specificity .73, precision .25: the model catches five of the seven dropouts at the price of 15 false alarms. Feature importance – grades, socioeconomic status, absences – without direction. And finally, splitting by school instead of by student: the forest's AUC drops from .64 to .60 – this small gap is what the dependence between students of the same school "contributed" to the first number (Performance metrics and validation, section 6). The data are artificial, and that is exactly the point: this is what a real project on 320 participants looks like – not AUC = .95.

10. And how do you do it in SPSS Modeler

For the researcher whose research lives in SPSS, the short route to machine learning is not Python but the sister software: SPSS Modeler, in which "data mining involves passing the data through a sequence of nodes, called a 'stream.' Each node represents a particular operation applied to the data, and the connections between the nodes define the flow of the data" (Truong, 2026, p. 56). This section follows Truong's textbook and translates the seven steps of the recipe of section 8 – and the R script of section 9 – into nodes. One caveat, unlike section 9: Modeler is not installed in the site's testing environment, so what is written here is a description according to the book, not a run; a researcher who runs it documents the stream (the str file) just as a notebook is attached.

The process. Modeler is built around CRISP-DM – the "cross-industry standard process for data mining," six phases: business understanding, data understanding, data preparation, modeling, evaluation and deployment; "it is more popular in industry than in academia, and IBM adopted and implemented it in the software" (p. 32). In the version 19.0 user's guide: "Business understanding. This is perhaps the most important phase of data mining" (IBM Corporation, 2025, p. 23), and the stream – "a series of nodes… that represents operations to be performed on the data, while links between the nodes indicate the direction of data flow" (p. 31). For the researcher, "business understanding" is step 1 of section 8 – the goal, the metric, the baseline – and it is done before opening the software.

The stream, node by node. Statistics File – reads a sav file directly (and for an Excel file in the site's template – an Excel node). Type – "for viewing and changing the measurement levels and roles of the fields": here you mark the target (Target), the predictors (Input), and what will not enter the model – the participant ID, the variable that is the target in disguise – as None; this is step 3 of section 8, and this is the place where leakage L2 is prevented or gets in. Partition – "training, testing and validation" in the proportions entered (pp. 87-88), and in the words of IBM's documentation: "to train the model on one sample, test and refine the model on a second sample, and validate the results on a third" (IBM Corporation, 2016b, p. 157); note the names: in Modeler the "testing" set is the second set, for comparing models, and "validation" – the third, for the final measurement, the reverse of the convention of Géron and of this page (Performance metrics and validation, section 2; Table 3.8 in Truong, p. 85). For a sample of hundreds, Truong's own recommendation is two sets only, 60-40 (p. 87) – and cross-validation, which is the heart of section 9, is not Modeler's default: in Truong it is recommended for hyperparameter tuning (p. 522), but all the examples in the book are measured on a single split. The model nodes – from the Modeling group: Logistic (the baseline and also a candidate), C&R Tree and CHAID (a tree; CHAID splits according to a χ² test and allows non-binary splits), Neural Net, Random Forest and XGBoost Tree (forest and boosting – Python nodes, which require the extension), and Auto Classifier, which runs several algorithms and ranks them – convenient, provided that what it ranks by is the metric of step 1. The forest node has "hyperparameter optimization, which automatically searches for the optimal combination of parameters to minimize the error rate on the samples" (p. 499) – this is step 5, but note on which samples: if on Modeler's "testing" set, it has become a validation set, and that is fine – as long as the final measurement is on the "validation" set. The academic Modeler textbook recommends "70-80% of the data for the training set and the rest for the test set" and fixing the seed in the Partition node "to get the same model in every run" (Wendler & Gröttrup, 2021, p. 826) – and on model selection, what this page says in section 5: "there is no method that outperforms every other model for every kind of data and classification problem… we strongly recommend trying different approaches and methods on every dataset" (p. 760). Each model node produces a "nugget" (nugget) – the trained model – which is connected to the data. Analysis (from the Output group) – "allows you to evaluate the ability of a model to generate accurate predictions" by comparison "between predicted values and actual values" (IBM Corporation, 2016b, p. 276), in Truong "compares the model's performance between the training sample and the testing sample" (p. 231): check Coincidence matrices (the confusion matrix), Performance evaluation, Evaluation metric (AUC and Gini) and Separate by partition (p. 377), and from it take accuracy, misclassification rate, Gini and AUC for each set; "sensitivity, specificity and F1 can be calculated from the confusion matrix" (p. 300) – yourselves, with a calculator. And Evaluation (from the Graphs group) – "ROC curves and Lift curves for all the models" overlaid (p. 502). When all the nuggets are chained to the same data, one Analysis node compares them all – this is the table of step 4 of section 8.

And what the menu does not do for you. The threshold: the nugget's default is 0.5, and what section 9 did in one line of pROC is done here in a Derive node on the probability field the nugget adds next to the prediction – or in the cost matrix in the model node's settings, which is Truong's cost-sensitive learning (pp. 203-204). The rare class: there is a SMOTE node and there is Balance, and both come after Partition, not before it – and it is better without them (Performance metrics and validation, section 1). Dependence between participants: Partition has no split by group; with repeated measures or schools, split yourselves (a Derive node on the group ID, then Select), or move to section 9. And the reporting: the Analysis node gives the numbers; what to report and what not – there, section 7, including Truong's configuration table from chapter 11, which is the way to document a stream that has no code.

Common mistakes

The validation mistakes – overall accuracy on a rare class, preparation on all the data before the split, splitting by row instead of by participant, tuning on the test set, a number without spread and without a baseline – in Performance metrics and validation. Here – the mistakes of the models themselves.

Reading feature importance as coefficients. "Income is the most important predictor" – yes; "income increases the odds" – the forest did not say that. Importance is not direction and is not cause; and a regularized coefficient in lasso regression is not an unbiased estimate of an effect.

Reporting a single tree as a finding. Retraining on the same data can give a different tree (chapter 6). A tree is an illustration of the model, not of reality; what is stable is the forest – and it is not drawn.

Choosing k by inertia. In clustering, inertia always decreases as clusters are added; k is chosen by the "elbow," the silhouette score, the gap statistic (Unsupervised learning, section 5) – and by reading the clusters themselves. And without prior standardization the clusters are "stretched" and k-means performs poorly (chapter 9).

Checklist – before reporting a prediction result

References

Bishop, C. M. (2006). Pattern recognition and machine learning. Springer.

Breiman, L. (2001a). Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16(3), 199-231. https://doi.org/10.1214/ss/1009213726

Breiman, L. (2001b). Random forests. Machine Learning, 45(1), 5-32. https://doi.org/10.1023/A:1010933404324

Géron, A. (2022). Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems (3rd ed.). O'Reilly Media.

Gutierrez, D. D. (2015). Machine learning and data science: An introduction to statistical learning methods with R. Technics Publications.

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7

Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., Vespignani, A., & Yarkoni, T. (2021). Integrating explanation and prediction in computational social science. Nature, 595(7866), 181-188. https://doi.org/10.1038/s41586-021-03659-0

IBM Corporation. (2016a). IBM SPSS Modeler 18.0 modeling nodes. IBM. https://public.dhe.ibm.com/software/analytics/spss/documentation/modeler/18.0/en/ModelerModelingNodes.pdf

IBM Corporation. (2016b). IBM SPSS Modeler 18.0 source, process, and output nodes. IBM. https://public.dhe.ibm.com/software/analytics/spss/documentation/modeler/18.0/en/ModelerSPOnodes.pdf

IBM Corporation. (2025). IBM SPSS Modeler 19.0 user's guide. IBM.

Ishii, K., Ueda, N., Maeda, E., & Murase, H. (2026). Pattern recognition and machine learning for self-study I: Supervised learning. Springer. https://doi.org/10.1007/978-981-95-1478-6

Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), Article 100804. https://doi.org/10.1016/j.patter.2023.100804

Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software, 28(5), 1-26. https://doi.org/10.18637/jss.v028.i05

Molina, M., & Garip, F. (2019). Machine learning for sociology. Annual Review of Sociology, 45, 27-45. https://doi.org/10.1146/annurev-soc-073117-041106

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825-2830.

Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206-215. https://doi.org/10.1038/s42256-019-0048-x

Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., Datta, D., Davidson, T., Filippova, A., Gilroy, C., Goode, B. J., Jahani, E., Kashyap, R., Kirchner, A., McKay, S., . . . McLanahan, S. (2020). Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences, 117(15), 8398-8403. https://doi.org/10.1073/pnas.1915006117

Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3), 289-310. https://doi.org/10.1214/10-STS330

Truong, D. (2026). Demystifying AI: Data science and machine learning using IBM SPSS Modeler. Chapman and Hall/CRC. https://doi.org/10.1201/9781003468196

Wendler, T., & Gröttrup, S. (2021). Data mining with SPSS Modeler: Theory, exercises and solutions (2nd ed.). Springer. https://doi.org/10.1007/978-3-030-54338-9

Wolpert, D. H. (1996). The lack of a priori distinctions between learning algorithms. Neural Computation, 8(7), 1341-1390. https://doi.org/10.1162/neco.1996.8.7.1341

Yarkoni, T., & Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6), 1100-1122. https://doi.org/10.1177/1745691617693393