Unsupervised learning: clusters, components and factors

Guide

On this page

The two sister pages – Machine learning for the social researcher and Performance metrics and validation – deal with supervised learning: there is a dependent variable, the model learns to predict it, and it is validated on data it has not seen. This page deals with what happens when there is no dependent variable – and the social researcher has been doing this for years without calling it that: factor analysis of questionnaire items, cluster analysis that divides participants into types, principal component analysis that reduces twenty variables to three. In machine-learning books all of these are called learning without supervision, i.e. unsupervised learning (Hebrew has two terms for it), and the page has two aims: to translate – to show that the Modeler cluster is the SPSS CLUSTER is R's kmeans() – and to warn, because here the one thing that protects a researcher in predictive modeling disappears: the metric. There is no test set to tell whether the three factors are "right" or whether the four clusters exist. The order: what makes learning "unsupervised", and why this is an uncomfortable situation (1); dimensionality reduction – principal component analysis (2); the distinction the software blurs – components are not factors (3); clusters – the methods and what each one assumes (4); how many clusters, and whether at all (5); what to do with a cluster after it has been found (6); what everything is called in the SPSS and Modeler menus (7); and how to do it in R (8). The sources are the five textbooks of the two sister pages – Géron (2022, chapters 8-9), Hastie, Tibshirani and Friedman (2009, chapter 14), Bishop (2006, chapters 9 and 12), Truong (2026, chapters 12-13) and Gutierrez (2015, chapter 8) – and with them, from the psychometric side, the review of factor-analysis practice in psychology (Costello & Osborne, 2005), and the original papers of the three criteria the page recommends: parallel analysis (Horn, 1965), the silhouette (Rousseeuw, 1987) and the gap statistic (Tibshirani, Walther & Hastie, 2001); from the SPSS Statistics side – the factor-analysis chapters of the two books the site cites on the calculator pages: Field (2024, chapter 18) and Pallant (2020, chapter 15); and from the SPSS Modeler side – the IBM documentation and the academic textbook by Wendler and Gröttrup (2021, chapters 6-7). A note on citation: as on the sister pages – Géron, Gutierrez, Field and Pallant by chapter and section (the copies have no page numbers), the rest by page.

1. Without a teacher: what it is, and why it is an uncomfortable situation

Hastie and colleagues open chapter 14 with a distinction: supervised learning is "learning with a teacher" – the "student" has an answer for every observation and the teacher corrects; unsupervised learning is "learning without a teacher" – there are N observations of a vector X, and "the goal is to directly infer the properties of this probability density without the help of a supervisor or teacher providing correct answers or degree-of-error for each observation" (Hastie et al., 2009, p. 486). Then comes the sentence from which the rest of the page follows: "With supervised learning there is a clear measure of success, or lack thereof… In the context of unsupervised learning, there is no such direct measure of success. It is difficult to ascertain the validity of inferences drawn from the output of most unsupervised learning algorithms. One must resort to heuristic arguments not only for motivating the algorithms… but also for judgments as to the quality of the results. This uncomfortable situation has led to heavy proliferation of proposed methods, since effectiveness is a matter of opinion and cannot be verified directly" (p. 487). Gutierrez puts it simply: "With unsupervised learning there is no way to verify your work, because there is no correct answer. That is why it is called unsupervised learning" (Gutierrez, 2015, chapter 8, opening). Géron quotes Yann LeCun: if intelligence were a cake, unsupervised learning would be the cake, supervised learning the icing, and reinforcement learning the cherry; "the vast majority of the available data is unlabeled" (Géron, 2022, chapter 9, opening).

What to look for when there is no y. Hastie and colleagues sort by what the method looks for in the distribution of X: principal components, multidimensional scaling and the like "attempt to identify low-dimensional manifolds within the X-space that represent high data density", and so they tell "whether the variables can be considered functions of a smaller set of 'latent' variables"; cluster analysis "attempts to find multiple convex regions of the X-space that contain modes" of the distribution, and so it tells whether it can be represented "as a mixture of simpler densities representing distinct types or classes of observations" (p. 486). Truong puts the same division in a table: the "object of analysis" of component analysis is the variables, of cluster analysis – the observations; the first produces components that summarize, the second produces groups (Truong, 2026, Table 13.1, p. 459). This is the distinction the social researcher knows from their own data matrix: reduce columns (sections 2-3) or group rows (sections 4-6). Géron adds two sister tasks – anomaly detection, "learn what 'normal' data looks like, and then use that to detect abnormal instances", and density estimation (chapter 9, opening) – and both return in section 4 as by-products of the same algorithms.

2. Dimensionality reduction: principal component analysis

The problem the method solves is the curse of dimensionality (the machine-learning page, section 2): Géron notes that two random points in a unit square are on average 0.52 apart, and in a cube of a million dimensions – about 408; "there's just plenty of space in high dimensions", and so "high-dimensional datasets are at risk of being very sparse" (Géron, 2022, Chapter 8, "The Curse of Dimensionality"). Principal component analysis (PCA) projects the data onto a smaller space. Bishop has two definitions "that give rise to the same algorithm": the orthogonal projection "such that the variance of the projected data is maximized" (Hotelling, 1933), or equivalently – the linear projection "that minimizes the average projection cost, defined as the mean squared distance between the data points and their projections" (Pearson, 1901) (Bishop, 2006, p. 561); in the wording of Hastie and colleagues, the principal components "provide a sequence of best linear approximations to that data, of all ranks" (Hastie et al., 2009, p. 534). Géron explains it with a picture: of the possible axes, PCA chooses "the axis that preserves the maximum amount of variance", then a second axis, orthogonal to the first, that preserves the largest amount of remaining variance, and so on; "the i-th axis is called the i-th principal component" (Chapter 8, "Preserving the Variance", "Principal Components"). What you hold after the projection is a table of explained variance ratios: in Géron's example 76% on the first component, 15% on the second, 9% on the third – "so it is reasonable to assume that the third PC probably carries little information" (Chapter 8, "Explained Variance Ratio").

How many components. It depends on what for: "if you are reducing dimensionality for data visualization, you will want to reduce the dimensionality down to 2 or 3" – otherwise, Géron suggests choosing the number of dimensions "that add up to a sufficiently large portion of the variance – say, 95%", or plotting the cumulative variance as a function of the number of dimensions and looking for "an elbow, where the explained variance stops growing fast"; and if PCA is a preprocessing step for a supervised model, "tune the number of dimensions as you would any other hyperparameter" – by cross-validation, according to the performance of the model that follows (Chapter 8, "Choosing the Right Number of Dimensions"). And a warning that comes before everything: dimensionality reduction "does cause some information loss… so I recommend that you first try to train your system with the original data before considering using dimensionality reduction" (chapter 8, opening). Two technical details the software will not do for you: standardization – PCA on variables in different units is PCA on the variable with the largest variance; Gutierrez demonstrates this with the US crime data, in which the variance of assaults is 6,945 and the variance of murders 19, and instructs: "You should, however, specify scale=TRUE" (Gutierrez, 2015, Chapter 8, "Principal Component Analysis"); and direction – the sign of a component's loadings is arbitrary, and a "negative" component is not a finding. And what does not belong here: t-SNE, which Géron notes "is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space", and its sibling UMAP – drawing tools, whose picture distances are not distances (Chapter 8, "Other Dimensionality Reduction Techniques"). And the loading table of the first component, which looks like a factor-analysis table – section 3.

3. PCA is not factor analysis – and why SPSS makes you think it is

For the social researcher this is the most important section of the page. Hastie and colleagues show that PCA can be read as a latent-variable model – each observed variable is a linear combination of p uncorrelated variables – "but this is not very satisfying", because every orthogonal rotation gives a different decomposition, "and hence it is impossible to identify any particular latent variables as unique underlying sources". "The classical factor analysis model, developed primarily by researchers in psychometrics, alleviates these problems to some extent": q < p latent factors, a loading matrix, and uncorrelated "disturbances" ε – "the idea is that the latent variables are common sources of variation amongst the observed variables, and account for their correlation structure, while the uncorrelated disturbances are unique to each variable and pick up the remaining unaccounted variation" (Hastie et al., 2009, pp. 558-559). That is the whole difference, and Bishop puts it in one sentence: factor analysis is probabilistic PCA in which the measurement noise of each variable is different – "the model explains the observed covariance structure of the data by representing the independent variance associated with each coordinate in the matrix Ψ and capturing the covariance between variables in the matrix W"; the columns of W are "the factor loadings", and the diagonal elements of Ψ – the uniquenesses (Bishop, 2006, pp. 583-584). Hence the implication: "factor analysis models the correlation structure of the variables rather than the covariance structure. This is an important distinction between factor analysis and PCA" (Hastie et al., 2009, p. 560). PCA explains all the variance of the items – the same items the researcher checks with Cronbach's alpha – including the measurement error of each item; factor analysis explains only what the items share. The community that lives on questionnaires reached the same conclusion from the other direction: "PCA is not a true method of factor analysis and there is disagreement among statistical theorists about when it should be used, if at all"; "component analysis is only a data reduction method"; "principal components analysis does not discriminate between shared and unique variance" – and yet "PCA is the default extraction method in many popular statistical software packages, including SPSS and SAS" (Costello & Osborne, 2005, p. 2). The researcher who clicks Analyze, Dimension Reduction, Factor in SPSS and touches nothing gets components, calls them factors and reports "a factor analysis". Modeler too: the node is called PCA/Factor, and it "predefines the PCA steps… which makes the process easier but also reduces our degree of control" (Truong, 2026, p. 442). And Pallant, whose book is how most researchers in Israel learned SPSS, says so frankly: in PCA "all the variance in the variables is used", in factor analysis "only the shared variance is analyzed"; "although PCA technically yields components, many authors use the term 'factor' to refer to the output of both PCA and FA. So don't assume, if you see the term 'factor' when you are reading journal articles, that the author has used FA" – and she herself chooses to demonstrate PCA, and gives the rule of Tabachnick and Fidell: "If you are interested in a theoretical solution uncontaminated by unique and error variability… FA is your choice. If, on the other hand, you simply want an empirical summary of the data set, PCA is the better choice" (Tabachnick & Fidell, 2013, p. 640, as cited in Pallant, 2020, chapter 15, opening). That is the whole rule in one sentence: a scale that measures a construct – FA; reducing variables before a regression – PCA. And Field explains why this is so, through communality: to extract factors you need to know how much of the variance is common, and you know that only after extraction; "there are two solutions. The first is to assume that all of the variance is common variance by assuming that the communality of every variable is 1… this procedure is PCA" – and so "PCA assumes that there is no measurement error", and "the fact that PCA assumes no measurement error upsets a lot of people who use factor analysis" (Field, 2024, Sections 18.4.2, 18.3.2). He cites Cliff: supporters of factor analysis "insist that components analysis is at best a common factor analysis with some error added and at worst an unrecognizable hodgepodge of things from which nothing can be determined" – and admits: "some argue that when PCA is used it should not be described as a factor analysis (oops!)" (section 18.4.3). And the other side: according to the review by Guadagnoli and Velicer, "the solutions generated from PCA differ little from those derived from factor analytic techniques" – with 30 or more variables and communalities above .7 "different solutions are unlikely", but "with fewer than 20 variables and any low communalities (below .4) differences can occur" (ibid.) – and that is exactly the case of a short questionnaire with weak items. The two SPSS books actually demonstrate the two choices: Pallant – PCA; Field, in the sixth edition – principal axis factoring (section 18.6.1). And the Modeler textbook has the same distinction, in Backhaus's wording: the factors of factor analysis "can be interpreted as 'the cause of the shared variance'", whereas the components of PCA "can be described as 'the general description [not the cause!] of the shared variance'" – and "PCA is used much more than PFA" (Wendler & Gröttrup, 2021, pp. 549, 552).

Both at once. The other side comes from the same page of Hastie and colleagues: since in the factor model too every rotation of the loadings gives an equivalent solution, "this leaves a certain subjectivity in the use of factor analysis, since the user can search for rotated versions of the factors that are more easily interpretable. This aspect has left many analysts skeptical of factor analysis, and may account for its lack of popularity in contemporary statistics" (p. 560). Psychometrics answers that rotation is exactly the tool: latent factors are defined not by the mathematics but by the interpretation, and rotation looks for a readable "simple structure". Both readings are right, and the practical conclusion is one: report which model (components or factors), which rotation, and why. And on rotation itself: if the factors are correlated – and psychological constructs are almost always correlated – an oblique rotation (oblimin, promax) "should theoretically render a more accurate, and perhaps more reproducible, solution"; and "if the factors are truly uncorrelated, orthogonal and oblique rotation produce nearly identical results" (Costello & Osborne, 2005, p. 4) – that is, there is nothing to lose with oblique; and Pallant: "I always recommend starting with an oblique rotation to check the degree of correlation between your factors" (Pallant, 2020, Chapter 15, "Step 3: Rotate and interpret the factors"). Varimax as a default is an assumption, not a choice. And before everything – is there anything to extract at all: Pallant requires correlations of .3 and above in the matrix, a KMO of at least .6 and a significant Bartlett's test (ibid., "Step 1") – all three in the SPSS output, and all three check only that there is a shared structure, not which one.

How many factors – the default rule is the worst. SPSS and Modeler keep every factor whose eigenvalue is greater than 1 (Kaiser). "There is broad consensus in the literature that this is among the least accurate methods for selecting the number of factors to retain"; in the simulations of Costello and Osborne "36% of our samples retained too many factors using this criterion" (pp. 2-3). Truong experiences this in real time: Modeler extracted five components for him "based on the condition that eigenvalues are greater than 1", and he asks "is this the best decision?", discovers that "SPSS Modeler does not provide a scree plot in the output of the PCA/Factor node", draws one in Excel – and admits that "identifying the exact elbow in this plot is somewhat challenging" (Truong, 2026, pp. 445-446). That is the flaw of the scree plot: it is a judgment by eye. The rule this page recommends is parallel analysis (parallel analysis; Horn, 1965): simulate random data of the same size – the same N, the same number of items – compute their eigenvalues, and keep only factors whose eigenvalue is greater than what noise gives. This is exactly the logic of the test set from the sister pages, applied to the only question possible: not "is the factor right" but "is it more than noise". Pallant summarizes the literature: parallel analysis "has been shown to be the most accurate, with both Kaiser's criterion and Cattell's scree test tending to overestimate the number of components", and "if you intend to publish… in psychology or education, you will need to use and report the results of parallel analysis" (Pallant, 2020, Chapter 15, "Parallel analysis"). In Modeler the situation is worse: Wendler and Gröttrup list the matrix adequacy measures – Bartlett, the inverse matrix, the anti-image, KMO – and conclude: "unfortunately, SPSS Modeler does not offer this measure nor any of the other statistics or measures described above"; Kaiser's rule, by contrast, "is implemented and used in SPSS Modeler", and a scree plot – "SPSS Modeler, unfortunately, does not" (Wendler & Gröttrup, 2021, pp. 551-552). Pallant's example is the evidence: the 20 PANAS items on 435 respondents – Kaiser keeps four components (eigenvalues 6.25, 3.40, 1.22, 1.16), the scree shows a break after the second, and parallel analysis, with criterion values of 1.40, 1.33, 1.27 and 1.22 from 100 random data sets, keeps two – exactly the two factors the scale was built on (ibid., "Interpretation of Output from Factor Analysis: Part 1", steps 2-4). The SPSS default was wrong by two. Field adds when Kaiser is accurate after all: "when the number of variables is less than 30 and the resulting communalities (after extraction) are all greater than .7; it can also be accurate when the sample size exceeds 250 and the average communality is .6 or greater" (Field, 2024, Section 18.4.5) – and summarizes the simulation of Auerswald and Moshagen (2019): "if your factors are highly correlated, all methods will be bad"; "with eight or more items and a sample of 200+, parallel analysis… will do a pretty decent job"; and the only time Kaiser is preferable is "when you have a small number of items (e.g. four) and correlated factors, and even then only in large samples and with the PCA method" (ibid.). In R, parallel analysis is one line (fa.parallel, Section 8); the SPSS menus do not have it, and the two books get around this in different ways – Pallant refers to a small separate program, Watkins's Monte Carlo PCA for Parallel Analysis, from the book's website, which prints the random eigenvalues for comparison by hand (ibid., step 4); Field runs R's psych from within SPSS syntax (BEGIN PROGRAM R, a ready-made file on the book's website; Field, 2024, SPSS Tip 18.1) – which is, in effect, the R line of section 8. And finally, N: factor analysis "is a 'large sample' procedure"; in the simulation of Costello and Osborne, at a ratio of two participants per item only 10% of the samples reproduced the true structure, at a ratio of 20:1 – 70% (pp. 4-7). Pallant, following Tabachnick and Fidell, settles for 150 participants and a 5:1 ratio, and also cites Nunnally's 10:1 (1978) (Pallant, 2020, Chapter 15, "Step 1"). And Field challenges the ratio itself: on 10-15 per item, "I have heard this rule mentioned on many occasions, but its empirical basis is unclear"; "what matters is the overall sample size, the factor loadings and the communalities" – Comrey and Lee rate 100 as a poor sample, 300 as good and 1,000 as excellent; a factor with four or more loadings above .6 "is reliable regardless of sample size"; and when communalities are low (well below .5) and the factors are many, over 500; "what is clear from this work is that a sample of 300 or more will probably provide a stable factor solution" (Field, 2024, Section 18.5.3). The combined reading: the ratios are rules of thumb, the number that recurs in all the sources is 300 – and what allows going below it is strong items, not few participants. A 30-item questionnaire on 90 participants will produce factors; they will not replicate in the next sample.

4. Clusters: the methods and what each one assumes

"Cluster analysis, also called data segmentation… is all about grouping or segmenting a collection of objects into subsets or 'clusters', such that those within each cluster are more closely related to one another than objects assigned to different clusters" (Hastie et al., 2009, p. 501). Géron: "it is the task of identifying similar instances and assigning them to clusters"; "just like in classification, each instance gets assigned to a group. However, unlike classification, clustering is an unsupervised task" (Géron, 2022, Chapter 9, "Clustering Algorithms"). And the word everything depends on is "similar": Gutierrez lists four questions – how "close" is defined, how to group, how to visualize, and "how to interpret the groupings… which may be hard to believe or may have arisen simply from noise" – and rules that "defining closeness is the most important step in a clustering algorithm" (Gutierrez, 2015, Chapter 8, "Clustering"). The distance – Euclidean, Manhattan, cosine for texts, Jaccard for lists – is a decision of the researcher, not of the algorithm, and Truong devotes to it the third of the eight steps of his analysis (Truong, 2026, p. 462). And before it, as with PCA, standardization: "it is important to scale the input features… before you run k-means, or the clusters may be very stretched and k-means will perform poorly" (Géron, 2022, Chapter 9, "Limits of k-means").

k-means (k-means) is the starting point, and Géron explains the idea through two symmetries: if we knew the cluster centers, we would assign each observation to the nearest one; if we knew the assignments, we would compute each center as a mean; we have neither – "start by placing the centroids randomly… then label the instances, update the centroids, label the instances, update the centroids, and so on until the centroids stop moving" (Chapter 9, "The k-means algorithm"). Bishop shows that these two steps are the E and M steps of the EM algorithm (Bishop, 2006, pp. 424-425). Two traps: the algorithm "is guaranteed to converge, but it may not converge to the right solution" – it depends on initialization, so it is run several times from different starts and the solution with the lowest inertia, the sum of squared distances from the centers, is kept (Géron, chapter 9, "Centroid initialization methods"; in R – nstart, Gutierrez, chapter 8, "K-Means Clustering"; and k-means++ initialization spreads the centers out in advance); and "k-means does not behave very well when the clusters have varying sizes, different densities, or nonspherical shapes" – in Géron's example it cuts a quarter off an elliptical cluster and attaches it to the neighbor, "even though its inertia is lower" (Chapter 9, "Limits of k-means"). And the quietest assumption: every observation belongs to one cluster. Bishop: "there may be data points that lie roughly midway between cluster centres. In such cases, it is not clear that the hard assignment to the nearest cluster is the most appropriate" (p. 428) – and the alternative is a Gaussian mixture (Gaussian mixture): each cluster is a multivariate normal distribution of its own, "with a different ellipsoidal shape, size, density, and orientation", and the model gives each observation a probability of belonging to each cluster – "a soft assignment, based on the posterior probabilities" (Bishop, 2006, p. 443; Géron, 2022, Chapter 9, "Gaussian Mixtures"). k-means is the limit of this model when the variance of all clusters is identical and tends to zero (Bishop, p. 443). For the researcher: if there are participants "between" the types – and there always are – the membership probability is the finding, not the label.

Hierarchical clustering (hierarchical clustering) does not require k in advance: "instead, they require the user to specify a measure of dissimilarity between (disjoint) groups of observations" – and then, in the agglomerative approach, start with each observation as a cluster and at each step merge the two closest, until one cluster remains; the result is a dendrogram, a tree in which the height of each node is the dissimilarity between its two daughters, and a horizontal cut of it gives a partition (Hastie et al., 2009, pp. 520-521). What "close" means between groups – the linkage: single, complete, average, or Ward (minimizing within-cluster variance) – changes the tree. And the warning of Hastie and colleagues, worth quoting in full: "Such a dendrogram is often viewed as a graphical summary of the data itself, rather than a description of the results of the algorithm. However, such interpretations should be treated with caution. First, different hierarchical methods…, as well as small changes in the data, can lead to quite different dendrograms. Also, such a summary will be valid only to the extent that the pairwise observation dissimilarities possess the hierarchical structure produced by the algorithm. Hierarchical methods impose hierarchical structure whether or not such structure actually exists in the data" (pp. 521-522). DBSCAN defines clusters "as continuous regions of high density": an observation with enough neighbors within a radius ε is a "core", neighbors of cores join the cluster, and anything that is not a core and has no core in its neighborhood "is considered an anomaly" – and it "works well if all the clusters are well separated by low-density regions" (Géron, 2022, Chapter 9, "DBSCAN"); there is no k, there is ε, and there are outliers as a by-product. And TwoStep in SPSS and Modeler: one pass over the data that compresses them into sub-clusters, then hierarchical clustering on those – intended for large data, and with "automatic determination of the number of clusters based on statistical criteria" (Truong, 2026, pp. 470, 473) – which is BIC, that is, section 5.

5. How many clusters – and whether at all

Hastie and colleagues distinguish between two situations. When the goal is segmentation, "K is usually defined as part of the problem" – a company with K sales people will divide its customers into K; "however, cluster analysis is often used to provide a descriptive statistic for ascertaining the extent to which the observations… fall into natural distinct groupings. Here the number of such groups K* is unknown" and must be estimated from the data (Hastie et al., 2009, p. 518). And the problem: the within-cluster dissimilarity, W_k, "generally decreases with increasing K. This is the case even when the criterion is evaluated on an independent test set, since a large number of cluster centers will tend to fill the feature space densely and thus will be close to all data points. Thus cross-validation techniques, so useful for model selection in supervised learning, cannot be utilized in this context" (p. 519). That is why there are no "performance metrics" here: all that remains is heuristics, and the page recommends three, in ascending order of validity.

The elbow. Géron: the inertia for k=3 is 653, for k=5 – 212, for k=8 – 119; "the inertia is not a good performance metric when trying to choose k because it keeps getting lower as we increase k". Plot it as a function of k and look for "an inflexion point called the elbow" – "this technique… is rather coarse" (Géron, 2022, Chapter 9, "Finding the optimal number of clusters"); for Hastie and colleagues this is the "kink" in W_k, "and as with other aspects of clustering procedures, this approach is somewhat heuristic" (p. 519). The silhouette (silhouette; Rousseeuw, 1987): for each observation, (b − a) / max(a, b), where a is the mean distance to the members of its own cluster and b the mean distance to the next closest cluster; "a coefficient close to +1 means that the instance is well inside its own cluster and far from other clusters… close to 0 means that it is close to a cluster boundary… and close to −1 means that the instance may have been assigned to the wrong cluster"; the silhouette score is the mean over all observations, and the silhouette diagram – a "knife shape" for each cluster – also shows clusters most of whose members are below the mean (chapter 9, ibid.). In Géron's example the silhouette confirms k=4 but shows that k=5 "is quite good as well, and much better than 6 or 7. This was not visible when comparing inertias" – and he chooses 5, for clusters of similar sizes: the silhouette helps, judgment remains. The gap statistic (gap statistic; Tibshirani, Walther & Hastie, 2001) is the only one of the three that also answers "and whether at all": it "compares the curve log W_k to the curve obtained from data uniformly distributed over a rectangle containing the data", and chooses the k at which the gap between the two is largest; "essentially this is an automatic way of locating the aforementioned 'kink'… It also works reasonably well when the data fall into a single cluster, and in that case will tend to estimate the optimal number of clusters to be one. This is the scenario where most other competing methods fail" (Hastie et al., 2009, p. 519). For Gaussian mixtures, where inertia and the silhouette "are not reliable when the clusters are not spherical or have different sizes", Géron moves to BIC and AIC – information criteria that penalize parameters (Chapter 9, "Selecting the Number of Clusters") – and that is what TwoStep does behind "automatically".

And in the calculator: the cluster analysis page runs the two-stage method on quantitative variables – Ward, followed by k-means starting from Ward's cluster centers – and prints for each k the three criteria of this section: W_k (the elbow), the mean silhouette and the gap statistic (as in clusGap, with R's generator). From there – a dendrogram, the cluster profile, a silhouette plot and a file with the cluster column for validation as in section 6. And at a click – stability under resampling: whether the partition, and each cluster in it, recurs in bootstrap samples – the validation without an external criterion of section 6.

6. The cluster as a hypothesis: what to do after it has been found

The rule that has no substitute: the algorithm always returns clusters. k-means on two completely uniform variables, with no structure at all, will return three clusters of reasonable sizes and with a respectable silhouette – section 8 shows this in numbers. Truong lists among the method's drawbacks the sensitivity to initialization, to outliers and to dimensionality, and the interpretation: "the clusters themselves may lack clear labels or explanations" (Truong, 2026, p. 462) – and his last two steps are profiling (profiling: the means and frequencies of each cluster) and validation, "ensuring that the clustering results are meaningful, reliable and appropriate for the data" (p. 462). How do you validate something that has no metric? In three ways, from the weakest to the strongest, and all three on data the cluster was not built from.

Stability. A cluster that exists in the data will recur: with another algorithm (k-means versus Ward versus a Gaussian mixture – the adjusted Rand index, ARI, between the two partitions), in another half of the sample (cluster one half, assign the second half to the centers found, and compare with its own independent partition), and with another initialization. A cluster that changes when the linkage is switched or ten participants are removed is a product of the algorithm, not of the population – this is the warning of Hastie and colleagues about the dendrogram (section 4), and it applies to every method. External variables. If the clusters are real types, they will also differ on a variable that took no part in building them – an outcome measured later, a background variable that was left out. That difference is the only evidence social research has that the cluster "means something", and it is tested with the usual tools (chi-square, ANOVA) – on one condition: the variable really was left out. Whoever clusters on five variables one of which is the outcome, and then reports that the clusters "predict" it, is reporting the definition. And what the cluster is not. It is a "descriptive statistic" (Hastie et al., 2009, p. 518) of this sample, with this distance and these variables; the name the researcher gives it – "the detached", "the engaged" – is an interpretation, exactly like the name of a factor. The tools map brings Drucker and Alberti's critique of topic modeling, which is described as if the topics "are already in the text rather than produced as an act of reading and interpretation" (ibid., section 1) – it applies, word for word, to clusters of participants. And if the cluster is later used as a variable in a supervised model – as a feature, or as a label that reduces labeling work: Géron shows that labeling 50 "representative images", one at the center of each cluster, instead of 50 random ones, raises accuracy from 74.8% to 84.9% (Chapter 9, "Using Clustering for Semi-Supervised Learning") – all the rules of leakage apply to it: the cluster is computed on the training set only, as a step in the pipeline.

7. And what it is called in the menu: SPSS Statistics and SPSS Modeler

SPSS Statistics. Pallant's procedure, step by step (Pallant, 2020, Chapter 15, "Procedure for Factor Analysis: Part 1"): Analyze, Dimension Reduction, Factor; move the items to Variables; under the Descriptives button check Initial Solution, Coefficients and KMO and Bartlett's test of sphericity; under the Extraction button – "make sure Principal components is shown, or choose one of the other extraction techniques (e.g., Maximum likelihood)" – this is the drop-down list that section 3 is about, and for factor analysis choose Principal axis factoring or Maximum likelihood in it; check Scree plot, and in the Extract area – Based on Eigenvalue (Kaiser's rule, the default) or Fixed number of factors, where you enter the number the parallel analysis gave; under the Rotation button – Direct Oblimin (the default is None); under the Options button – Sorted by size and Suppress small coefficients with .3. The syntax this produces is a FACTOR command with /EXTRACTION PC, /CRITERIA MINEIGEN(1) and /ROTATION OBLIMIN – the three lines that reveal what was done, and so they are what is attached to the report. And what is reported, according to Pallant: the extraction method, the criteria for the number of factors – "and this should include parallel analysis" – the rotation method, the total variance explained, the eigenvalues before and after rotation, a full table of loadings "showing all values (not just those above .3)", and with oblimin rotation – the pattern matrix and the structure matrix in full, with the correlations between the factors (ibid., "Presenting the Results from Factor Analysis"). Field, in the same windows, chooses Principal axis factoring, analyzes the correlation matrix ("a useful default because it takes the standardized form… if variables have been measured using different scales this will not affect the solution"), and recommends to "run a primary analysis with eigenvalues greater than 1, select a scree plot and compare… ideally also run a parallel analysis. If the different criteria give different results, examine the communalities and decide for yourself" (Field, 2024, Section 18.6.1, SPSS Tip 18.2). Reporting according to Field: the table of rotated loadings for all items "with values above a criterion (I would personally choose .40) highlighted", the percentage of variance for each factor and perhaps also the eigenvalue, the data adequacy measures – KMO with Kaiser and Rice's labels (.90 "marvelous", .80 "meritorious", .70 "middling", .60 "mediocre", .50 "miserable", and below .5 – collect more data or rethink the variables; section 18.5.2) – "and ideally, deposit the correlation matrix in an open science repository such as the OSF so that someone can reproduce the analysis" (section 18.8); his example: principal axis, oblimin, KMO .92, four factors in a parallel analysis against the 95th percentile of 20 random data sets. And his concluding sentence, which deserves to be stuck to the screen: "very many decisions are left to the researcher who uses factor analysis, and so researcher degrees of freedom… come into play quite a bit: try to make informed and unbiased decisions, and resist the temptation of decisions that give you the results you want to get" (section 18.7.6). Clusters – Analyze, Classify (here already without Pallant, who does not cover clusters; the paths are general information about the software): Hierarchical Cluster (with the dendrogram under Plots and the choice of distance and linkage under Method), K-Means Cluster (the number of clusters is required; Iterate and Classify), and TwoStep Cluster (automatic number by BIC or AIC). Standardization – Descriptives, Save standardized values, or Standardize inside the hierarchical window – is a separate step, and the researcher who forgets it gets clusters of the variable with the largest range. SPSS does not compute the silhouette or the gap statistic; TwoStep shows a "silhouette measure of cohesion and separation" in the Model Viewer output, and for the rest – section 8.

SPSS Modeler. The same thing in nodes (the stream – the machine-learning page, section 10), and this time from IBM's own documentation. The PCA/Factor node in the Modeling group offers "two similar but distinct approaches", and puts section 3 in its own words: "PCA focuses on all variance, including both shared and unique variance… Factor analysis focuses on shared variance only. Variance that is unique to specific fields is not considered in estimating the model" (IBM Corporation, 2016a, p. 182). The default Extraction Method is Principal Components; the alternatives – Unweighted / Generalized Least Squares, Maximum Likelihood, Principal Axis Factoring and more – are in Expert mode, and so is Extract factors: Eigenvalues over (Kaiser's rule) or Maximum number, whose "default is five factors/components"; and Rotation, whose default is "No rotation" (p. 183). That is, the same three SPSS Statistics defaults, under other names – and without a scree plot: Truong, whom the node gave five components, drew it in Excel (Truong, 2026, pp. 445-446). For clusters there are three nodes: K-Means – "the number of clusters to specify… the default is 5", and a warning worth reading twice: "the resulting model depends to a certain extent on the order of the training data. Reordering the data and rebuilding the model may lead to a different final cluster model" (IBM Corporation, 2016a, p. 227) – a single run, without the nstart of section 8, and Wendler and Gröttrup explain why: the first center is "the values of the first record in the dataset", and the next ones – the records farthest from the centers already set; the numeric variables are first standardized, internally, to a 0-1 range (Wendler & Gröttrup, 2021, pp. 676-677); TwoStep – standardizes numeric fields by default, can exclude outliers to a "noise" cluster, and chooses the number of clusters automatically "based on the change in the Bayesian Information Criterion (BIC) as more clusters are added" and then by the largest change in the minimum distance between clusters; the distance – log-likelihood (numeric and categorical together; assumes normal and independent continuous variables) or Euclidean (pp. 229-230) – and it too, according to Wendler and Gröttrup, "depends on the order of the objects in the dataset" because of the pre-clustering (Wendler & Gröttrup, 2021, p. 651); and Auto Cluster, which runs TwoStep, K-Means and Kohonen, ranks by silhouette (the default; "models with a negative silhouette can safely be discarded"), by number of clusters or by their size – and allows "evaluation fields": "although no target is used, you can specify one or more evaluation fields to compare the models. The usefulness of a cluster model can be evaluated by measuring how well (or badly) the clusters differentiate these fields" (pp. 70-71) – this is the external variable of section 6, built into the software. And what is missing: the Analysis node, which compares predictions with the truth, "is useful only for supervised models… unsupervised models such as clustering algorithms have no true results to serve as a basis for comparison" (IBM Corporation, 2016b, p. 276) – IBM says section 1 in one sentence. And the Modeler textbook says sections 5-6: "based on practical experience, we believe that the decision of how many clusters to determine should be made first", with the silhouette as a measure of fit (p. 724); "no less, the number of clusters must be determined on the basis of practical knowledge" (p. 750); and "the clustering methods provide no help in finding an appropriate description for each cluster. The researcher must work out for themselves how best to describe each cluster" (Wendler & Gröttrup, 2021, p. 650). Parallel analysis, the gap statistic and stability across algorithms – are one line of R.

8. And how to do it in R

The following script ran as is on the page's sample data file – DATA_unsup_example_eng.xlsx (and in Hebrew _heb.xlsx; both in qa/ml/): 300 respondents in a community survey, 12 Likert items (belonging, trust, safety – four items each), four behavior variables (volunteering hours, events, neighbors known, years in the community), two uniform "noise" variables, and an outcome measured later – leaving. The data are artificial, and were generated from three factors and three profiles – so that we know the answer and can see which criterion finds it; the true profile is not in the file. Packages: readxl, psych (parallel analysis, factors), cluster (silhouette, gap statistic), mclust – mixtures and the Rand index (Scrucca et al., 2016). The path – replace it. The comments are in English, because the block is displayed left to right.

library(readxl); library(psych); library(cluster); library(mclust)
d <- read_excel("C:/Rdata/DATA_unsup_example_eng.xlsx")
items <- d[, c("belong1","belong2","belong3","belong4","trust1","trust2","trust3","trust4",
               "safe1","safe2","safe3","safe4")]
beh   <- d[, c("volunteer_h","events","neighbours","years")]

# --- Step 1: PCA on the 12 items - standardised (scale = TRUE), because units differ ------
pca <- prcomp(items, scale. = TRUE)
ev  <- pca$sdev^2                                   # eigenvalues of the correlation matrix
round(rbind(eigenvalue = ev, proportion = ev / sum(ev), cumulative = cumsum(ev) / sum(ev)), 3)
cat("Kaiser (eigenvalue > 1) keeps:", sum(ev > 1), "components\n")

# --- Step 2: how many? parallel analysis (Horn) for components AND factors ----------------
set.seed(20260922)
pa <- fa.parallel(items, fa = "both", n.iter = 100, plot = FALSE)
cat("parallel analysis: components =", pa$ncomp, " factors =", pa$nfact, "\n")

# --- Step 3: PCA is not factor analysis: same items, two models ---------------------------
pc3 <- principal(items, nfactors = 3, rotate = "oblimin")   # components: total variance
fa3 <- fa(items, nfactors = 3, fm = "ml", rotate = "oblimin") # common factors: shared variance
print(pc3$loadings, cutoff = .30)
print(fa3$loadings, cutoff = .30)
cat("factor correlations (EFA):\n"); print(round(fa3$Phi, 2))
cat("communalities - PCA vs EFA (first 4 items):\n")
print(round(rbind(PCA = pc3$communality[1:4], EFA = fa3$communality[1:4]), 2))

# --- Step 4: k-means on the 4 behaviour variables - scaled first ---------------------------
X <- scale(beh)
set.seed(20260922)
km <- lapply(2:6, function(k) kmeans(X, centers = k, nstart = 25, iter.max = 100))
inertia <- sapply(km, function(m) m$tot.withinss)              # the "elbow" curve
sil     <- sapply(km, function(m) mean(silhouette(m$cluster, dist(X))[, 3]))
gap     <- clusGap(X, FUN = function(x, k) kmeans(x, k, nstart = 25), K.max = 6, B = 100)
round(cbind(k = 2:6, inertia = inertia, silhouette = sil), 3)
print(gap, method = "Tibs2001SEmax")

# --- Step 5: the chosen solution - and what it looks like in the original units ---------
k3 <- km[[2]]                                                   # k = 3
cat("sizes:", k3$size, "\n")
round(aggregate(beh, by = list(cluster = k3$cluster), FUN = mean), 1)

# --- Step 6: does another algorithm agree? hierarchical (Ward) vs k-means ----------------
hc  <- hclust(dist(X), method = "ward.D2")
hc3 <- cutree(hc, k = 3)
table(kmeans = k3$cluster, ward = hc3)
cat("adjusted Rand index, k-means vs Ward:", round(adjustedRandIndex(k3$cluster, hc3), 3), "\n")

# --- Step 7: stability - split-half: cluster each half, predict the other by nearest centre
set.seed(20260922)
h  <- sample(rep(c(TRUE, FALSE), length.out = nrow(X)))
kA <- kmeans(X[h, ], 3, nstart = 25); kB <- kmeans(X[!h, ], 3, nstart = 25)
nearest <- function(x, centres) apply(x, 1, function(r) which.min(colSums((t(centres) - r)^2)))
predB <- nearest(X[!h, ], kA$centers)                          # A's rule applied to B
cat("split-half ARI (A's centres on B vs B's own clusters):",
    round(adjustedRandIndex(predB, kB$cluster), 3), "\n")

# --- Step 8: the control - two pure-noise variables get clustered too ---------------------
Z <- scale(d[, c("noise_a", "noise_b")])
set.seed(20260922)
kz <- kmeans(Z, 3, nstart = 25)
cat("noise, k = 3: sizes", kz$size, " silhouette",
    round(mean(silhouette(kz$cluster, dist(Z))[, 3]), 3), "\n")
gz <- clusGap(Z, FUN = function(x, k) kmeans(x, k, nstart = 25), K.max = 6, B = 100)
cat("noise, gap statistic chooses k =", maxSE(gz$Tab[, "gap"], gz$Tab[, "SE.sim"], method = "Tibs2001SEmax"), "\n")

# --- Step 9: model-based alternative - Gaussian mixture, number of components by BIC ------
mc <- Mclust(beh, G = 1:6, verbose = FALSE)
cat("Mclust: model", mc$modelName, "with G =", mc$G, "components; ARI vs k-means",
    round(adjustedRandIndex(mc$classification, k3$cluster), 3), "\n")

# --- Step 10: the cluster is a hypothesis, not a finding: an outcome that was NOT clustered on
tab <- table(cluster = k3$cluster, moved_out = d$moved_out)
round(prop.table(tab, 1), 2)
chisq.test(tab)

What came out (R 4.3.3, psych 2.4.1, cluster 2.1.6, mclust 6.0.1; the full output – qa/ml/unsup_out.txt). The items: the eigenvalues of the 12 items – 2.88, 2.12, 1.80, then 0.85 and below; the first three explain 56.7% of the variance. Kaiser's rule keeps three, and so does parallel analysis – three components and three factors; here they agree, because the structure is clean, and in a real questionnaire they will not always agree. The same items in both models, both with oblimin rotation: each item loads on its own factor in both – but the component loadings are higher (0.80, 0.79, 0.68, 0.75 for belonging, against 0.74, 0.71, 0.55, 0.63), and the communalities of the four belonging items are 0.65, 0.63, 0.48, 0.55 in PCA against 0.56, 0.51, 0.32, 0.39 in factor analysis; the three components "explain" 56.7% and the three factors 43.0%. The difference is the unique variance of the items – measurement error – which PCA puts into the components and factor analysis leaves out; this is section 3 in numbers. The factor correlations .17, .09, .26 – not zero, hence oblique. The behavior: on the four standardized variables, the inertia falls 498, 274, 245, 227, 200 for k from 2 to 6 – an elbow at 3; the silhouette .47, .48, .44, .44, .37 – a maximum at 3, but 2 and 4 are close; the gap statistic, with 100 simulations – 3. The cluster sizes are 71, 120 and 109, and their means in the original units: 15 volunteering hours, 22 events, 25 neighbors and 20 years in one cluster; 1.8, 3, 6 and 4.5 in the second; 8, 10, 14 and 12.5 in the third – engaged, detached, in-between; the names are ours, not the algorithm's. Stability: Ward agrees with k-means on 298 of 300 participants (ARI .98); a partition learned on one half and applied to the other agrees with that half's own independent partition at ARI .96; a Gaussian mixture chooses three components by BIC (model VEI) and agrees with k-means completely (ARI 1). The noise: on two uniform variables with no structure at all, k-means returns three clusters of sizes 108, 104 and 88 with a silhouette of .39 – a number that in a real questionnaire would be reported; the gap statistic chooses k=1. That is the whole argument. And the hypothesis: leaving, which took no part in the clustering, differs between the clusters – 7%, 30% and 21% (χ²(2) = 14.0, p < .001); this is evidence that the clusters mean something, and this is the level at which they are reported: a difference on an external variable, not "three types in the community".

Common mistakes

Reporting "factor analysis" for principal components. The SPSS default – "oops!" (Field). The components contain the measurement error, the communalities are inflated, and there is no way to speak of a latent structure (section 3). For factor analysis – Principal axis or Maximum likelihood, oblique rotation.

Letting the eigenvalue of 1 decide. "Among the least accurate methods", and yet the default in both programs; in Pallant's PANAS example it gives four components for a two-factor scale. Parallel analysis – and if it is not available, at least a scree plot with reasoned judgment (Table 2).

Clustering without standardizing. A variable with a 0-100 range alongside variables with a 1-5 range: the clusters belong to the large variable (Géron, chapter 9; Gutierrez, chapter 8). In SPSS standardization is a separate step.

Choosing k by inertia – or by habit. Inertia always decreases; "three clusters" is a habit. Silhouette and gap statistic, and an answer to "one" as well (section 5).

Calling the dendrogram "the structure of the data". It is a product of the algorithm; another linkage, ten fewer participants – another tree. "Hierarchical methods impose hierarchical structure whether or not such structure actually exists" (Hastie and colleagues, p. 522).

Validating the cluster on the variable that built it. "The clusters differ significantly in volunteering" – when volunteering is one of the clustering variables – is a report of the definition. Validate only on a variable that was left out (section 6).

Giving names and forgetting that they are yours. "The detached" is an interpretation of means, not a property the algorithm discovered; and the observations in the middle – which a Gaussian mixture would have given 50/50 – disappear into the label.

Interpreting distances in t-SNE. The map is meant for viewing; the size of a cluster and its distance from others in the picture depend on the parameters and are not a finding (section 2).

Checklist – before reporting components, factors or clusters

References

Bishop, C. M. (2006). Pattern recognition and machine learning. Springer.

Costello, A. B., & Osborne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, and Evaluation, 10(7). https://doi.org/10.7275/jyj1-4868

Field, A. (2024). Discovering statistics using IBM SPSS statistics (6th ed.). SAGE.

Géron, A. (2022). Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems (3rd ed.). O'Reilly Media.

Gutierrez, D. D. (2015). Machine learning and data science: An introduction to statistical learning methods with R. Technics Publications.

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. https://doi.org/10.1007/978-0-387-84858-7

IBM Corporation. (2016a). IBM SPSS Modeler 18.0 modeling nodes. IBM. https://public.dhe.ibm.com/software/analytics/spss/documentation/modeler/18.0/en/ModelerModelingNodes.pdf

IBM Corporation. (2016b). IBM SPSS Modeler 18.0 source, process, and output nodes. IBM. https://public.dhe.ibm.com/software/analytics/spss/documentation/modeler/18.0/en/ModelerSPOnodes.pdf

Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179-185. https://doi.org/10.1007/BF02289447

Pallant, J. (2020). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (7th ed.). Routledge. https://doi.org/10.4324/9781003117452

Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53-65. https://doi.org/10.1016/0377-0427(87)90125-7

Scrucca, L., Fop, M., Murphy, T. B., & Raftery, A. E. (2016). mclust 5: Clustering, classification and density estimation using Gaussian finite mixture models. The R Journal, 8(1), 289-317. https://doi.org/10.32614/RJ-2016-021

Tibshirani, R., Walther, G., & Hastie, T. (2001). Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(2), 411-423. https://doi.org/10.1111/1467-9868.00293

Truong, D. (2026). Demystifying AI: Data science and machine learning using IBM SPSS Modeler. Chapman and Hall/CRC. https://doi.org/10.1201/9781003468196

Wendler, T., & Gröttrup, S. (2021). Data mining with SPSS Modeler: Theory, exercises and solutions (2nd ed.). Springer. https://doi.org/10.1007/978-3-030-54338-9