Market Research, Sampling and the Reliability of Findings

Primary and secondary research. Primary research is collected first-hand for the specific question being asked, through surveys, interviews, focus groups, observation or experiments. It is current, precisely targeted and unavailable to competitors, but expensive, slow and easily distorted if badly designed. Secondary research uses data that already exists — industry reports, government statistics, internal sales records, trade press. It is cheap and fast and often provides a long time series, but it was gathered for somebody else's purpose, so its definitions may not match the question, it may be out of date, and competitors can buy it too, so it confers no advantage.

The order matters. Secondary research should come first, to establish what is already known, before money is spent on primary research to fill the remaining gap. Recommending an immediate survey without checking existing data is a costed decision made without checking the price, and it is marked down as such.

Qualitative and quantitative. Quantitative research produces numbers that can be aggregated and compared, answering how many and how much. Qualitative research produces depth of understanding, answering why. They are complementary rather than rivals: a focus group generates the hypotheses that a survey then tests at scale, and a survey identifies the pattern whose explanation only interviews can supply.

Sampling methods, in descending order of reliability. Random sampling gives every member of the population an equal chance of selection; it is the most reliable but needs a complete list of the population and costs the most. Stratified sampling divides the population into groups and samples randomly within each, so the sample mirrors the population's structure — but it requires knowing that structure in advance. Quota sampling fills a target number in each group without randomness, so the interviewer's own choices introduce bias. Convenience sampling asks whoever is available: cheapest, and least reliable.

Four sources of bias, each with its own name. Sample bias: the wrong people were asked. Non-response bias: those who refuse differ systematically from those who reply, so a low response rate makes a survey not merely smaller but skewed — this is the most dangerous and least often named. Leading questions: the wording suggests its own answer. Social desirability bias: respondents report what sounds good rather than what they actually do, which is why stated intentions to buy consistently overstate actual purchases.

Judging a finding. Before acting on research, ask four questions: who was asked and were they the right people; how many replied out of how many were approached; how were the questions worded; and does the conclusion follow from the data or from the researcher's hope. A finding that survives all four is worth acting on; one that fails any of them should trigger better research, not a decision.