Research question and hypotheses

Guide

On this page

Most problems in statistical analysis are not born in the analysis – they are born earlier, when the research question did not become a falsifiable hypothesis, and when decisions that should have been made before data collection were made after it. The page follows Field's path (2024, chapters 1-3): observation → theory → hypothesis → prediction → data, and then the null and alternative hypotheses, the direction of the test, the significance level, the sample size – and what has become clear in the last decade about what happens when these are decided after the fact. It ends with what p says and what it does not, and with effect size – the number that should accompany every p.

1. The research process – and why the hypothesis is not the prediction

Research begins with an observation that calls for explanation – anecdotal ("I noticed that...") or data-based – and with the question "why". To check whether the observation itself is correct and not biased, at least one variable that quantifies it is needed first; then turn to relevant theories, derive an explanation from them (a hypothesis), and from the hypothesis – predictions. Predictions need data: identify what can be measured, collect, analyze. The analysis supports the hypothesis, or generates a new one, or revises the theory – so data collection and theory are linked in both directions (Field, 2024, Sections 1.2-1.3).

The distinction that is easiest to miss – and Field (2024, section 1.4) devotes an explicit correction to it – is between a hypothesis and a prediction. A hypothesis is an explanation, a statement about a mechanism, and it cannot be observed. A prediction is what follows from it and moves it into the observable realm: it is what makes it possible to collect data that bear on the plausibility of the hypothesis. An example we will return to on this page: observation – in localities with an active community association, residents report a higher sense of belonging. Theory – social capital: networks of acquaintance and trust produce belonging and collective action. Hypothesis – participation in community activity strengthens the sense of belonging to the locality. Prediction – residents who participate in community activity will report higher belonging than residents who do not participate, even when comparing residents with the same length of residence in the locality. Now there is something to measure: participation (hours per month, or yes/no) and belonging (a multi-item scale – see reliability and validity).

Note that the same observation is also consistent with a reverse hypothesis – those who already feel they belong are the ones who join the activity – and with a third hypothesis: length of residence in the locality produces both belonging and participation. A good prediction distinguishes between the hypotheses: if the first hypothesis is true, those who joined the activity this year should show an increase in belonging compared with last year; if the second is true – belonging preceded joining. What the competing hypotheses require is a research design that can separate them, and that is a topic for the page variables, levels of measurement and research designs. The main point here: a hypothesis that explains every possible result is not a hypothesis.

2. Theory, hypothesis, prediction – and what makes a statement scientific

Field (2024, section 1.4) distinguishes the three by breadth and status: a theory explains a broad set of phenomena by means of a small set of well-established principles; a hypothesis explains a narrower phenomenon, and has not yet been tested; a prediction is the observable statement derived from the hypothesis. A statement is scientific when it can be verified or refuted against empirical evidence. "The Beatles were the most influential band ever" is not scientific – "influence" probably cannot be meaningfully quantified – but "The Beatles were the best-selling band ever" is, because sales data can be collected and checked. Non-scientific statements can sometimes be rewritten as scientific ones, and a good theory produces predictions that are such statements (Field mentions Popper here). And in the language of social research: "The program strengthened the community" is not a prediction; "A year after the program, the proportion of residents who know at least five neighbors by name will rise" – is.

The other side of the same coin is falsification: data that contradict the hypothesis. Field illustrates this with a long story about reality-show contestants and a theory of his that failed: one prediction from it was refuted, findings of other researchers did not fit it, and a competing theory – which explained both the original observation and the new findings – won. The test of a theory is not that it sounds right; it is that it explains the data, that its predictions are confirmed in new data, and that it also explains the findings of others. A theory that cannot fail on any data explains nothing.

3. From the prediction to the statistical hypothesis

The prediction breaks down into two statistical hypotheses (Field, 2024, Section 2.10.2). The alternative hypothesis (H₁) is the prediction from the theory – usually that there is an effect: participants will report higher belonging. The null hypothesis (H₀) is its opposite – there is no effect: participants and non-participants will report the same belonging. The null hypothesis is useful because it provides a baseline: how likely would it have been to get the data we got – or more extreme data – if there were no effect at all? That is what p measures, and no more. Field warns against the language of "accepting" and "rejecting" hypotheses: p-values do not provide evidence for such black-and-white decisions, and even when the alternative is more plausible than the null, there may be other models – which we did not test – that fit the data better. We speak of relative plausibility, not of truth.

Direction. A directional hypothesis also states the direction of the effect ("participants will report higher belonging"); a non-directional one states only that there is an effect ("participation is related to belonging"). A statistical test of a directional hypothesis is one-tailed; of a non-directional one – two-tailed (Field, 2024, Section 2.10.6). The difference is not technical: in a one-tailed test a significant result in the opposite direction does not count – so it is justified only when an opposite result would lead to the same action as a non-significant result (a drug that is worse than the existing one will not be approved, exactly like a drug that is no better than it), or when an opposite result is theoretically meaningless. Field cites two surveys of ecology journals: in one "all uses of one-tailed tests appeared invalid", and in the other only one of 17 articles was justified. And the temptation is obvious: a two-tailed p = .06 becomes a one-tailed p = .03 – but the rules cannot be changed after the data have been collected. The site default, as in Field: two-tailed.

4. What is decided before collecting data – and what it costs to decide afterwards

A statistical test works only if the hypotheses and criteria were set before data collection (Field, 2024, Section 2.10.3). Field's parable: betting on the World Cup. You choose a team and get odds before the tournament; you cannot switch teams halfway, and the bookmaker cannot change the odds after the final. Researchers "cheat" when they formulate hypotheses after the data have been collected, change them midway, or choose the significance level after seeing the result – and every such change increases the chance of a significant result, and also the chance that no one will be able to replicate it. Four decisions belong to this stage:

The significance level (α). The long-run error rate you are willing to tolerate – the probability of declaring an effect that does not exist (a Type I error). Field stresses that this is supposed to be a meaningful decision based on the research field and on what is being tested, and admits that in practice everyone uses .05 "without thinking about what it means or why"; Fisher himself never said that .05 is a magic number (Field, 2024, Sections 2.10.1, 2.10.3; Box 3.1). The rule: decide in advance, and justify any departure from the convention.

The sample size – in advance. A p-value is a long-run probability: how often we would get such a statistic if we repeated the same sampling process an infinite number of times. Therefore you must collect the number of participants you planned: there is a way to compute p for repeated samples of 80, but no way to know what p is for samples of 74 "because the semester ended and we couldn't find more participants" (Field, 2024, Section 2.10.3). Stopping early or continuing until the result is significant – two sides of the same mistake. How many participants are needed? That is the question of power: the probability that the test will find an effect that exists in the population, 1 − β, where β is the probability of missing it (a Type II error). The convention, following Cohen (1988, 1992, as cited in Field, 2024, Sections 2.10.7 and 2.10.9), is power of .80 – an 80% chance of detecting a real effect, that is, missing one in every five. Power depends on the effect size, on α and on the sample size; when three are known, the fourth is computed. The useful application, according to Field, is before the study: estimate the expected effect size from previous studies, set α = .05 and power .80, and get the number of participants required – with a free tool such as G*Power or the pwr package in R. Computing power after the study, he adds, never made sense to him: if the result is not significant – there was not enough power; if it is significant – there was.

The hypotheses and the analysis plan. "Researcher degrees of freedom" (Simmons et al., 2011, as cited in Field, 2024, Section 3.3.2) are all the decisions a researcher makes along the way: which model, what to do with extreme values, which control variables, which measures, when to stop collecting. Field cites a study in which 29 teams received the same data and the same question – whether soccer referees give more red cards to dark-skinned players – 20 found a significant effect and nine did not, across a wide range of models. When these decisions are chosen according to what brings p below .05 – trying several analyses and reporting one, adding or removing variables according to their effect on p, measuring ten outcomes and reporting the significant one – that is p-hacking; and when an article presents a hypothesis formulated after the results as if it had been formulated before them – that is HARKing, "hypothesizing after the results are known" (Field, 2024, Section 3.3.3). The remedy science has adopted is preregistration (preregistration): public posting of the rationale, hypotheses, design and data processing and analysis plan before data collection – in a registered report in a journal, which is reviewed before there are results and is published whatever they are, or simply in an open repository such as the Open Science Framework (Field, 2024, Section 3.6). Deviating from the plan is allowed – it is just made transparent. And in a study without formal registration: a dated planning document, with the hypotheses, tests and sample, before the first questionnaire.

5. What p says – and what it does not

p is the probability of getting the result we got, or a more extreme one, assuming the null hypothesis is true. Field (2024, section 3.2.1) opens with three common mistakes in what is inferred from a significant result. "Significant" is not "important": p depends on the sample size, so tiny, unimportant effects will be significant if we collect enough data, and large, important effects will be missed in a small sample. Non-significant is not "the null hypothesis is true": a non-significant result says only that the effect was not large enough to be found in this sample; it does not say that its size is zero. An effect of .000000001 in the population is not zero, and in a large enough sample it too "will be found"; therefore a non-significant result must not be interpreted as "no difference" or "no relationship". And significant is not "the null hypothesis is false": probabilistic inference does not permit this conclusion – Field illustrates it with a syllogism about the band Iron Maiden: "If a person plays the guitar, it is unlikely that they are a member of Iron Maiden; this person is a member of the band; therefore it is unlikely that they play the guitar" – and the conclusion is false, because half of the band's members play the guitar. Significance provides no evidence about either of the two hypotheses.

The big practical problem is all-or-nothing thinking (Field, 2024, Section 3.2.2): p = .0499 is significant and p = .0501 is not, and opposite conclusions are reached from numbers that differ by .0002 and reflect the same effect size. Field presents ten studies of the same fictitious drug: four significant and six not – and according to the NHST cookbook, "the evidence is contradictory". Then look at the confidence intervals: all ten differences are positive, the confidence intervals overlap considerably, and most of each interval is above zero – consistent evidence that the effect in the population is probably positive. The confidence interval shows the consistency that p hides. In the chapter on "common sense" (section 3.5) he presents the six principles of the American Statistical Association (Wasserstein & American Statistical Association, 2016, as cited in Field): p can indicate how incompatible the data are with the null hypothesis – and its exact value is used, not which side of the threshold it falls on; p is not the probability that the hypothesis is true, nor the probability that the data were produced by chance; scientific conclusions and policy decisions are not based only on crossing a threshold; p-values must not be reported selectively, and the number of hypotheses tested is reported; significance is not practical importance; and p alone is not a good measure of evidence for a model or a hypothesis.

6. Effect size – the number that accompanies every p

The solution to the "significant is not important" criticism is to measure the effect size: the magnitude of the effect and its direction (Field, 2024, Section 3.7). There are two types. Raw – in the original units of measurement: a difference of 2 minutes between groups, a regression coefficient b that says how many belonging points each hour of participation adds. Field (following Baguley) sees it as the most useful way to quantify an effect, because it is directly understandable – provided the units are meaningful. Standardized – in units of standard deviations, in order to compare studies that measured different variables: Cohen's d for a difference between means, r for a relationship, the odds ratio for binary outcomes. Its price: intuition. What is "half a standard deviation" in the real world? And in the absence of direct understanding, people cling to yardsticks.

Cohen's yardsticks (1988, 1992, as cited in Field, 2024, Sections 3.7.3-3.7.4) – d of .2 (small), .5 (medium), .8 (large); r of .10 (1% of the variance), .30 (9%), .50 (25%) – are widespread, but Field reminds us that Cohen himself admitted that they encourage lazy thinking and ignore the context of the effect: the measuring instrument, the norms in the field. He quotes Lenth: when interpreting an effect size you are not buying T-shirts – "I'll take the Metallica effect in size M". Evaluate an effect in the context of the research question: r = .30 between participation and belonging is one finding in a survey with dozens of background variables, and another when it is the only one found. A further advantage of effect size (section 3.7.6): it has no decision rule. It is interpreted on a continuum, so it does not break between .0499 and .0501, and it is less sensitive to stopping data collection early or late – although it is estimated better in a large sample.

7. Wording and reporting

In the research proposal: the research question in one sentence; the hypothesis as an explanation; the predictions as observable statements, directional or not, each with the variables that measure it, the test that will check it and the effect size; the significance level; the planned sample size and how it was determined (the estimated effect size, the power). What was not formulated here is not a prediction – it is an exploratory finding, and it is reported as such.

In the results chapter (Field, 2024, Sections 2.11 and 3.9): an exact p-value – p = .023, not p < .05 – because the computer calculates it, and it gives the reader more information than "below an arbitrary threshold"; the only exception is p < .001, below which the digits are not interesting. A confidence interval in square brackets, with its level: M = 30, 95% CI [20, 40]. And an effect size alongside every p, because p depends on the sample size: "Belonging was higher among participants, t(148) = 2.31, p = .022, d = 0.38, 95% CI [0.05, 0.71]". A non-significant result is reported in the same way, with an exact p and an effect size – not "no difference was found". On the site, as in APA (2020, section 6.36), coefficients that cannot exceed 1 are written without a leading zero (p = .023, r = .30); Field mentions the rule and chooses not to follow it in his book – the report sentences produced by the site's calculators follow it.

Common mistakes

Phrasing the hypothesis as a prediction. "A difference will be found between the groups" is not an explanation of anything. The hypothesis says why; the prediction says what we will see. Without the "why" there is no theory to revise when the prediction fails.

A hypothesis that every result confirms. If an increase in belonging supports the theory, and so does a decrease ("initial resistance"), and so does no change ("a delayed effect") – there is no prediction here. A prediction is written so that a specific result would refute it.

Choosing one-tailed after seeing p = .06. One-tailed is justified only when an opposite result would lead to the same action as a non-significant result, and only when decided in advance. Otherwise – two-tailed, and the result is not significant.

"We'll collect a bit more and see." Continuing to collect until the result is significant, or stopping when it is significant, destroys the meaning of p. The sample size is set in advance from a power calculation and kept; if it was not reached, report that.

Computing power after the study to explain a non-significant result. This is circular: a non-significant result will always look "underpowered". Power is computed before, to determine the sample.

Reporting the significant and keeping quiet about the rest. Twenty tests, one significant – at the .05 threshold that is exactly what is expected by chance. Report how many hypotheses were tested, and a finding that was not predicted is presented as exploratory, to be tested in a new sample.

Reading "non-significant" as "no relationship". A non-significant result is not evidence for zero; it is evidence that this sample was not enough. The confidence interval says which effect sizes are still plausible.

Relying on the size yardstick instead of the context. "A medium effect" is not an interpretation. The question is what d = 0.4 means in the original units, and what it is worth in this study – for someone planning a community program, two points on a 7-point belonging scale are a different finding than for a theorist.

Checklist – before the first questionnaire

References

American Psychological Association. (2020). Publication manual of the American Psychological Association (7th ed.). https://doi.org/10.1037/0000165-000

Field, A. (2024). Discovering statistics using IBM SPSS statistics (6th ed.). SAGE.