Sample and sampling

Guide

On this page

Every statistical conclusion passes from the sample – the people, texts or localities that were measured – to the population that really interests us. This step depends on two questions that are decided before the first questionnaire: whom the sample represents, that is, how it was selected, and how large it is. The page defines population, sample, parameter, statistic and sampling error (Gravetter & Wallnau, 2017); explains what exactly "random sampling" requires and why the membership list of the yacht club is not a sample of the city; surveys the probability and non-probability sampling methods following Krippendorff (2019, chapter 6) – written about texts, but valid for people and documents; presents the standard error and the central limit theorem – the reason a sample of a few hundred is enough for a population of millions; and ends with the question of sample size, which is a question of power (see research question and hypotheses), and with those who did not respond (see missing values). Version 3 added the recruitment chapter by Chandler (2023) from the Cambridge handbook of research methods in the social and behavioral sciences (Nichols & Edlund, 2023): what lies between the sample you planned and the sample you will get – when a non-probability sample is enough, how many to invite in order to reach n, and what the evidence says about advance notices, reminders and incentives.

1. Population, sample, parameter, statistic – and sampling error

Research begins with a question about a particular group. The entire group the researcher wants to study is the population – all the individuals of interest to the study; it can be enormous (all the women in the world) or narrow (women who are heads of state), and it does not have to consist of people – companies, parts in a factory, articles. The researcher is the one who defines it, and she must always identify it explicitly (Gravetter & Wallnau, 2017, p. 3). Since populations are too large to measure in their entirety, a sample is selected – a group from the population intended to represent it, and always identified by the population from which it was selected. A value that describes a population is a parameter; a value that describes a sample is a statistic; every parameter has a corresponding statistic, and most research deals with the relationship between them (ibid., p. 4).

And here is the fundamental problem that all of inferential statistics is meant to solve: sampling error – the natural gap, created by chance, between a sample statistic and a population parameter (Gravetter & Wallnau, 2017, pp. 5-7). Their example: a population of 1,000 students, two samples of five – the mean age in one sample is 19.8, in the other 20.4, and in the population 21.3. No sample is identical to the population, and the samples differ from each other – and they are only two of hundreds of possible samples. Or: divide a class in two along an imaginary line down the middle of the room and compute the mean age for each side – there will almost certainly be a difference, and there is no mysterious force pulling older people to the right side; the unpredictable, unsystematic differences from sample to sample are sampling error. The "margin of error" in election polls – "plus or minus 4 points" – is exactly this. All statistical tests ask one question: is the difference we saw larger than what sampling error alone would produce?

2. What a "random sample" is – and why it is not "whoever happened to be around"

"Random" is a technical term, not a description of a lack of planning. A simple random sample requires that every individual in the population have an equal chance of being selected; the statistical formulas also require a second condition – that the probability stay constant from one selection to the next, that is, independent of who has already been selected – and this is an "independent random sample", which most statistical applications assume (Gravetter & Wallnau, 2017, pp. 161-162). The first condition is what ensures there is no bias in selection: in a population of N individuals, each has probability 1/N. And so, in their examples, you do not get a random sample of the city's residents from the membership list of the yacht club, nor a sample of students from your own psychology course. To draw a sample you need a sampling frame – a numbered list of all the individuals in the population – and a drawing mechanism that gives each individual the same probability (Krippendorff, 2019, p. 114). And in Krippendorff's words (p. 119): a random sample is an admission that we have no idea what the population looks like and where the information is – and when we do have an idea, other methods exploit it.

3. Sampling methods – what each one allows you to infer

Krippendorff (2019, chapter 6) distinguishes between three probability methods – designed to ensure that every unit has the same chance of being included, which is the basis for statistical generalization – and methods that exploit knowledge of the population's structure or give up on representation. Simple random: numbering all the units and drawing (p. 114). Systematic: every k-th unit on the list, from a randomly drawn starting point; convenient for publications and recurring events, but biased when the fixed interval coincides with a natural "rhythm" in the list – seasonality, cyclicality: every seventh day in a daily newspaper will always give the same day of the week (and the Tuesday science section will appear in the whole sample or in none of it), and every June between 1932 and 1942 in the wedding announcements of the New York Times found not a single synagogue wedding – because June fell in the period in which Jewish tradition prohibits weddings (p. 115). Stratified: identify subpopulations (strata) to which each unit belongs exclusively, and sample randomly or systematically within each stratum separately – so all strata are represented, according to their relative size or any definition set in advance; the example: the "typical week" of American television, constructed by drawing one program out of 52 for each broadcast slot (pp. 115-116). Varying probability: when units are not equal in the information they carry – newspapers by circulation – each is given its own probability (p. 116). Cluster: when the units cannot be listed but groups of them can – localities, schools, issues – sample clusters and measure all the units within them; convenient and common, but the probability of a unit being included depends on the size of its cluster, the variance is inflated and the sampling error is not controlled – the sample represents the clusters, not necessarily the units within them (pp. 116-117). Snowball: start from initial units and repeatedly apply to them a criterion that adds units – citations in the literature, ties in a social network – up to a natural limit; the initial sample matters, and when the snowball explodes a stopping criterion is needed (pp. 117-118). Relevance (purposive): select all the units that contribute to the answer, in a conceptual hierarchy that narrows down the population – not probabilistic, and the result is not a "representative sample" but the population of relevant units; only afterwards, if it is still too large, sample from it (pp. 118-120). Census: all the units of their kind – the writings of one author, all the articles on an event – there is nothing to sample (p. 120). Convenience: an available body of data that is known not to include the whole population; the choice of who gets in remains in the hands of circumstances beyond the researcher's control, and sometimes in the hands of those who have an interest – diaries are written to impress, election speeches omit what would be costly, and few eyewitnesses survived (pp. 120-121). A convenience sample is not invalid – but it obliges the researcher to think about what brought these data, and not others, into her hands.

When a non-probability sample is enough. Chandler (2023) arranges the methods on a single continuum of cost and rigor: a census (the whole population – practical when it is small: employees of an organization, students of an institution), a probability sample (every individual can be identified, and each has a known, nonzero probability of being included – "the gold standard", because it approximates the census result at a fraction of the cost, and its biases can be corrected statistically and its uncertainty quantified), and a non-probability sample – anything that does not meet the condition, intentionally or not. His important point: a non-probability sample is not necessarily more biased – but whether it is biased, and by how much, cannot be known. This uncertainty is tolerable in four situations: when the sample covers a large part of the population and the response rate is high; when the variables studied are not related to the chance of being included in the sample; when the question is about a relationship between variables and not a point estimate – relationships are much less sensitive to population differences than means and percentages; and when testing a theory rather than describing the world – the truth of a theory is determined by its ability to predict what will happen under given conditions, not by how common those conditions are in reality (Chandler, 2023). The site holds both: per Table 1 – a convenience sample is reported as a convenience sample, and generalizing from it to the population is an argument; and for this argument Chandler provides the words – "a relationship, not an estimate" and "theory, not description" – and the limit: whoever needs to know how much (the percentage of supporters, the mean in the population) needs a random draw.

The composition of the sample. Even without a random draw, and certainly with one, you can decide in advance who will be in the sample (Chandler, 2023): a purposive sample – people with certain characteristics, or quotas – or a stratified probability sample. The composition follows from the research question: for estimating an effect in the population – a sample that represents it on the characteristics that affect the size of the effect; for comparing groups – groups of equal size, because the comparison is statistically more efficient (so in a survey of localities, compare stratum with stratum and not according to their size on the map); for a relationship between continuous variables – oversampling of those with extreme scores, which increases the power to detect the relationship and makes it easier to identify extreme cases that affect the result. And at the formative stage of research – a critical case: first approach those among whom the phenomenon is most likely, in order to know whether it exists at all before investing in a representative sample.

4. The standard error and the central limit theorem – why a few hundred are enough

If we took every possible sample of size n from a population and computed a mean for each, we would get a distribution of sample means – the sampling distribution. This cannot be done in practice, but the central limit theorem says exactly what it looks like: for any population with mean μ and standard deviation σ, the distribution of sample means for samples of size n is centered on μ, its standard deviation is σ/√n, and it approaches a normal distribution as n increases – "almost perfectly normal" already at n = 30, regardless of the shape of the population (Gravetter & Wallnau, 2017, p. 199). This standard deviation of the sample means is the standard error: how much distance can reasonably be expected between a sample mean and the population mean. It does two things – it describes how much sample means differ from one another, and it measures how well a single mean represents the population (ibid., pp. 200-201). Two things determine its size: the standard deviation of the population, and the sample size – the larger n is, the closer the mean is to μ, "the law of large numbers" (ibid., p. 201). Field (2024, section 2.8) adds that everything said about the mean is true of any parameter – a correlation, a regression coefficient: every estimate has a sampling distribution and a standard error, and the confidence interval and the test are built from it.

Two practical conclusions. The first – the √n: to halve the standard error you need to quadruple the sample. Going from 100 to 400 participants gains a lot; from 4,000 to 16,000 – very little relative to the cost. That is why a national survey of a few hundred to a thousand-odd respondents is enough for a population of millions – if it is random. The second – what this number does not do: it reduces sampling error, not bias. A convenience sample of 10,000 selected non-randomly is a large biased sample; its standard error is small and its confidence interval narrow, around the wrong value. The size of the sample and its representativeness are two separate questions, and only the second decides to whom the conclusion applies.

5. How many participants are needed – a question of power, not of percentage

There is no single number. The right question is: how many participants are required to have a reasonable chance of detecting an effect of the size that interests us – and that is a question of power, which is detailed in research question and hypotheses: from the expected effect size, the significance level, and the desired power (.80), compute n with a free tool such as G*Power, which Pallant (2020, introduction to Part Five) also recommends, quoting Stevens: in a large sample (100 or more) "power is not an issue", and in small groups (n = 20) a non-significant result may reflect insufficient power. A few rules of thumb from the literature the site cites, all secondary citations: in multiple regression – about 15 participants per predictor (Stevens, 1996), or N > 50 + 8m where m is the number of predictors (Tabachnick & Fidell, 2013) – both as cited in Pallant (2020, chapter 13); and more if the outcome is skewed. And in content analysis, Krippendorff (2019, pp. 121-122) adds one solid principle: when the units that would change the answer are rare, the sample must be much larger – to "catch" a phenomenon whose probability is 1 in 1,000 with 95% confidence you need 2,995 units (his Table 6.1); in a close election a larger sample is needed than when the gap is wide. He also suggests an empirical approach: compare samples of different sizes with a complete population and see from which size the addition no longer changes the answer.

And what does not determine it: a percentage of the population. The standard error depends on n, not on the ratio n/N (except when the sample is a substantial part of the population); 400 random respondents represent a city of 100,000 just as well as they represent a country of 10 million. And also: Pallant (2020, chapter 1) reminds us to plan pessimistically – people do not show up, drop out and do not complete the questionnaire properly – and to make sure that an experiment has enough participants in each group, preferably equal.

The recruitment arithmetic. Chandler (2023) turns "plan pessimistically" into numbers: the power calculation gives the n of the analysis file – the starting point, not the target. A survey that needs 100 participants in the analysis needs at least 112 complete questionnaires, because some will be discarded in data cleaning (see the standard template): about 2% of respondents skip any given question, and about 10% of responses – more in an online survey – are unusable. And 112 complete questionnaires require inviting several hundred people, because many will not see the invitation or will refuse: 80%-90% of those who start an online survey with an incentive complete it, but the completion rate among those invited is around 40%-50% on average – about 60% face to face, about 50% by mail, about 30% online, about 20% by telephone – and the rates decline from year to year, "and all these estimates may be optimistic" (Chandler, 2023; the estimates are his summary of the American literature, and vary between populations). Therefore a good sampling plan draws a small initial sample plus replicate samples (replicates) that are used as needed: this way you do not recruit too many, not only those easy to find, and do not get stuck with an underpowered sample. The estimate of the response rate is checked again as soon as recruitment begins.

6. Who did not respond – the sample you planned versus the sample you got

The only random sample that ever exists is the one that was planned. The sample actually measured is those who agreed, were found, and answered to the end. If those who did not respond differ systematically from those who did – young people who do not answer the phone, busy people who do not fill in a long questionnaire, those for whom the question is too personal – the sampling was random and the sample no longer is. This is exactly the question of the missingness mechanism in missing values, at the respondent level instead of the item level: nonresponse that depends on variables recorded for everyone (age, locality, gender) can be corrected by weighting and by controlling for them; nonresponse that depends on what was not measured, or on the answer itself – cannot, and it is impossible to prove that it is not like that. What can be done: document the response rate and the composition of those who responded against the frame, and compare – with chi-square – the respondents with the population on the known variables. Chandler (2023) adds the caveat and the exception: a high response rate usually reduces nonresponse bias – but not always (Groves & Peytcheva, 2008, as cited in Chandler); and in an intercept or volunteer sample, where it is not known how many people saw the call, there is no response rate at all – there is at most a cooperation rate (how many of those approached agreed), according to the AAPOR definitions (2016, as cited in Chandler).

7. Recruitment – how to reach the sample you planned

"A strong recruitment plan is a key factor in the success of research with human participants", Chandler (2023) opens – and research without such a plan risks recruiting too few, or the wrong kind, and discovering this only in the analysis. The plan follows from five questions: which population; which inference (description or theory); how much precision; how fast; and with what resources – "fast, good or cheap: pick two". Where to recruit from. From a list (a sampling frame: names, addresses, telephone numbers, sometimes demographics – often a list of an organization to which access must be obtained): every probability sample begins with one; its advantages – contact details, linking each participant to the study materials (only eligible people, only once), a response rate that can be computed, and comparison of those who responded with those who did not – but understand how it was created and who is missing from it (coverage error), and be prepared to clean it: many email addresses are not valid. Without a list: intercept (intercept) – recruiting from a stream of activity: mall visitors, website visitors, an ad on a social network targeted at a group; efficient for populations that gather in one place, but the sampling unit is the event (a visit) and not the person – frequent shoppers are overrepresented; volunteers – ads, posts, mailing lists: easy, and volunteers are more conscientious, but they have selected themselves by interest in the study; network (snowball – in Table 1): each participant refers others – for populations that are hard to locate, or those who will not identify themselves before a peer confirms the study is legitimate. And buying access: research companies with panels – probability-based (expensive) or not; before buying, ask for the answers to ESOMAR-36, the standard questionnaire for describing a sample provider. And combining: a result that recurs in different samples of different types is itself evidence for generalization (Chandler, 2023).

The response rate – four factors and what the evidence says. Whether they knew there was a study (about 80% screen calls from an unknown number, 75% of emails are not opened); what the expected benefit is; what the perceived cost is (time, complicated instructions, sensitive questions); and who the population is. Chandler presents the averages of meta-analyses, most of them on mail and internet surveys – the numbers, as he presents them, are relative changes in the response rate and not percentage points (unless stated otherwise): length – shorter gets more responses, and 20 minutes is probably the limit for most people; open-ended questions and sensitive questions lower response, an "interesting" questionnaire doubles it, and opening with an open-ended question lowers it by 5%; mode – an online survey gets about 11% less response than non-electronic modes, and internet users are younger, more educated and earn more; combining modes – sequentially, not as a choice (offering a choice up front lowers response): first the cheap one (online), then mail or telephone for those who did not respond; advance notice (a letter that tells about the survey, its purpose, and what about privacy) – +50% in mail surveys, and also by telephone and online; a letter by mail signals legitimacy; reminders – +35% by mail, +30% online, with diminishing returns; about half of those who respond to an online survey respond on the first day and almost all the rest within a week, so space them out; different wording for each reminder – +30% compared with the same wording; monetary incentive – +87% by mail, +20% online, with diminishing returns (the first dollar +5 points, above 10 dollars – about a point per dollar); an incentive in advance, unconditional – +61% compared with the same amount after completion, by mail and by telephone (reciprocity), less online; non-monetary incentives – only +15% and regardless of their value, and sharing the results – almost nothing; lotteries – popular because they are cheap, but "rarely" improve response; personal address by name – +10% by mail, +25% online; a handwritten signature and address – more than 25%; and in email – a direct subject line that says what the study is, not "clickbait", and sometimes the first question inside the invitation. What the invitation contains: the purpose and content of the study, framed around what is relevant to the recipient; the incentive; the length (especially if short); a deadline. Claims of benefit to society or to science – almost no effect (Chandler, 2023; the averages – from meta-analyses he cites, mainly Edwards et al., 2009, and presented here as he summarizes them).

Who responds less, and who fakes. Men, young people, those with low (and very high) income, the less educated and the unmarried respond less to surveys, depending on the mode – and this directs the recruitment effort toward those less likely to respond when representativeness is wanted, and toward those likely to respond when you want to save. When participants have to meet a condition, screening requires care: in a study that recruited through ads, 14% admitted to inventing a medical condition in order to be eligible for a paid study – and in a rare population the fakers can be the majority. The safeguard: recruit first of all where eligible people are common, do not reveal the eligibility criteria before measuring them, and preferably the respondent should not know at all that they were screened – a short study with the screening questions, and the eligible are routed onward (Chandler, 2023).

Pilot, monitoring and documentation. The recruitment procedure is piloted like the questionnaire itself – a "dress rehearsal" that exposes ambiguity in the protocol, misunderstandings in the team, and wrong assumptions about cost and response rate. And during data collection, monitor three indicators: the response or completion rate, how similar those who responded are to the population, and the cost per completion (all expenses divided by the number of complete questionnaires) – and adjust: channels, the wording of the invitation, an additional approach or a larger incentive for groups that respond less. And document the plan, the effort and what worked and did not work – both in the preregistration and as supplementary material for the article – because without it the study population cannot be understood or the study replicated (Chandler, 2023). The ethical aspect of incentives – coercion versus undue influence – is on the research ethics page.

8. How to report

In the method section, under "Participants" or "Sample": the population as defined; the sampling frame (which list, from when); the sampling method by name and in detail (systematic – k and the start; stratified – the strata and the allocation; cluster – what the cluster is and how many; convenience – who, where, how they were recruited); the planned sample size and how it was determined (the power calculation, the assumed effect size); the response rate; the actual N; and the comparison of respondents with the population on the known variables. And in the discussion – to whom the conclusion applies. A convenience sample is reported as a convenience sample, without "representative" and without "random" – the word "random" is reserved for a random draw. The minimum according to Chandler (2023): whether there was a sampling frame; how the participants were selected; the steps taken to recruit them; response or cooperation rates – and how they were computed, numerator and denominator, because there are several ways (the standard definitions – AAPOR, 2016, as cited in Chandler); the exclusion criteria in screening; and without a frame – the target population, the time and place of recruitment that may affect the composition, and the limits on generalization that follow from them.

Common mistakes

A "random sample" of whoever happened to be around. Random means a draw from a list with an equal chance for everyone. Students in a course, users who clicked, visitors to the community center – a convenience sample, and it is reported as such.

Thinking that size corrects bias. 10,000 respondents selected non-randomly are a large biased sample, with a narrow confidence interval around the wrong value. Size reduces sampling error; only a random draw deals with bias.

Computing n as a percentage of the population. The standard error depends on n, not on n/N. 400 are enough for a city and for a country alike.

Systematic sampling with an interval that coincides with a cycle. Every seventh day, every even-numbered call – the interval locks the sample onto a single subgroup. Check the list for cyclicality before setting k.

Sampling clusters and analyzing as if the individuals were independent. 20 residents from each of ten localities are not 200 independent observations – the dependence within the locality shrinks the standard error. A multilevel model, or at least reporting the structure.

Forgetting who did not respond. A response rate of 30% means that the random sample has become a sample of those who agreed. Report the rate and compare respondents with the frame.

Setting n after data collection. "We'll collect until it's significant" destroys the meaning of p. n is set in advance from a power calculation, and reported even if it was not reached.

Writing "representative" without a random draw. A relevance or convenience sample can be correct and useful – but generalizing from it is an argument that must be justified, not a result of statistics.

Inviting exactly n. The power calculation is the analysis file; invite several times as many, and draw replicate samples (Chandler, 2023).

A prize draw as an incentive, and sharing the results as a lure. Almost no effect; a small monetary incentive in advance, advance notice and reminders – yes (Chandler, 2023).

Publishing the eligibility criteria in the recruitment ad. Whoever wants to take part will fit themselves to them; the screening questions come before revealing what is being screened for (Chandler, 2023).

Checklist – before recruiting

References

Chandler, J. (2023). Participant recruitment. In A. L. Nichols & J. E. Edlund (Eds.), The Cambridge handbook of research methods and statistics for the social and behavioral sciences: Vol. 1. Building a program of research (pp. 179-201). Cambridge University Press. https://doi.org/10.1017/9781009010054.010

Field, A. (2024). Discovering statistics using IBM SPSS statistics (6th ed.). SAGE.

Gravetter, F. J., & Wallnau, L. B. (2017). Statistics for the behavioral sciences (10th ed.). Cengage Learning.

Krippendorff, K. (2019). Content analysis: An introduction to its methodology (4th ed.). SAGE.

Pallant, J. (2020). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (7th ed.). Routledge.

Stevens (1996), Tabachnick and Fidell (2013) and Cohen (1988) – as cited in Pallant (2020); Edwards and colleagues (2009), Göritz (2006, 2014), Mercer and colleagues (2015), Groves and Peytcheva (2008), Devine and colleagues (2013), Simons and colleagues (2017) and AAPOR (2016) – as cited in Chandler (2023). Per the site rule, the reference list includes only sources that were read in the original; Chandler (2023) – by the chapter as a whole, without page numbers in the body of the text.