Foundations of Market Research: Evidence and Validity

You have probably heard some version of this reassurance: the data has been cleaned, the sample is representative, the fieldwork was rigorous. For decades that was usually enough. Quality meant effort — somebody had to read a questionnaire, think about it, type an answer. The checks worked because faking that effort was hard.

It is no longer hard. A language model can fill out a survey in seconds, pass an attention check that stops a distracted human, and produce open-ended answers that read better than most real ones. A synthetic panel can mimic a demographic slice of a country without sampling a single person. A vendor can ship results by Friday that took a field team six weeks last year. None of this is automatically wrong, some of it is genuinely useful, but all of it breaks the old compact where "this number was expensive to produce" was a rough proxy for "this number is trustworthy."

That is what this chapter is about. Not whether AI is good or bad for research, but the harder question underneath: when the cost of producing a number drops to near zero, what makes one number worth betting on and another worth ignoring? What does "valid" even mean for evidence that came from a simulation? How do you trace where a finding originated when the pipeline has six steps and three vendors? How do you tell whether a fast result traded away the thing that made it useful?

Why this matters to you

If you commission research, buy data, or make decisions on the basis of either, the ground has shifted under you. The old quality signals — slow turnaround, expensive fieldwork, large sample sizes — no longer reliably separate the good from the bad. A study that took three days and cost a fraction of what you used to pay might be excellent or might be theatre, and the deliverable looks the same either way. You need a different filter, one built around provenance, disclosure and holdout testing rather than price and prestige. This chapter gives you that filter.

What you'll find inside

The pieces here work through the new trust problem from several angles. You will see why a number can now pass every traditional check and still be worthless, and what the industry's authenticity crisis looks like from the inside. You will learn what validity means when the evidence is synthetic, and why benchmarks and evals, borrowed from machine learning, are becoming the language of research quality. You will meet provenance — the chain of custody a finding needs before you can trust it — and the question of what "representative" means now that a sample can be assembled or generated rather than drawn. You will look at the machine version of the say-do gap, where a model's stated preference diverges from real market behaviour, and at the exchange rate between speed and truth that every commissioning decision now involves. The chapter closes with triangulation, the principle that no single method deserves your full confidence, rebuilt for a stack where the methods themselves have changed.

The honest note

The characteristic caveat for this chapter, and for the whole corpus that follows, is this: the authenticity problem is market research's own replication-crisis moment. The replication crisis taught psychology that famous findings could fail to hold up; the authenticity crisis is teaching research that familiar quality markers can fail to mean what they used to. The response that worked last time works again — holdouts, pre-registration, open disclosure, and a stubborn willingness to report when the answer is "we don't know" — but only if the profession actually commits to it rather than assuming the problem is someone else's. The discipline that answered the first crisis answers this one. Whether the industry chooses to apply it is the open question.

The nine pieces in this chapter

1. "When is a number worth betting on?" — decision-grade evidence
2. "Is anyone real out there?" — the authenticity crisis
3. "What makes a simulated answer true?" — validity for synthetic evidence
4. "A score is only as honest as the test behind it" — benchmarks and evals
5. "A number can now come from nowhere" — provenance
6. "A representative sample is now a claim, not a draw" — representation after panels
7. "Models say; markets do" — the say-do gap, machine edition
8. "The 48-hour insight has an exchange rate" — speed vs truth
9. "One method is an opinion" — triangulation, rebuilt