Hypotheses, significance and choosing a test

Writing a hypothesis that scores

An alternative hypothesis must be testable and must contain the operationalised variables. "People remember better in the morning" is not enough; "participants recall significantly more words from a 20-word list when tested at 9am than at 9pm" is. A directional (one-tailed) hypothesis states which way the difference goes and is justified when previous research points one way; a non-directional (two-tailed) hypothesis is used when it does not. The null hypothesis states there is no significant difference or relationship, and any difference found is due to chance.

What significance actually means

The $p$ value is the probability of getting these results if the null hypothesis were true. Psychology conventionally uses $p \le 0{,}05$: a $5\%$ chance of a fluke. A stricter level such as $p\le0{,}01$ is used when a false claim would be costly, for example in medical research. Significance is not the same as importance — a tiny difference can be significant with a large sample.

Type I and Type II errors

A Type I error rejects a true null hypothesis — claiming an effect that is not there. A Type II error retains a false null — missing a real effect. Making the significance level stricter reduces Type I errors and increases Type II errors. They trade off; you cannot cut both.

Choosing a test — three questions

1) Difference or relationship? 2) If difference: same participants (related) or different participants (independent)? 3) What level of measurement? Nominal difference data give chi-square; ordinal or better with a related design give Wilcoxon; with an independent design give Mann-Whitney; a relationship with ordinal data gives Spearman. Answering the three questions is faster and safer than memorising the grid.

Reading the result

Compare the calculated value with the critical value for your $N$, your significance level and one or two tails. Whether you need the calculated value to be larger or smaller depends on the test, so the table's instruction must be read, not assumed.