Synthetic respondents vs. real people: What still counts as evidence in market research
·13 min read
Synthetic respondents are useful for generating hypotheses and pretesting a questionnaire. They are not a substitute for real people when the decision behind the study is expensive to reverse. The same AI that powers them now also shows up uninvited, as fluent fake answers inside human samples, so a study is only as good as your checks on who answered.
That was the shared view of a September 2026 panel hosted by itracks, “The advantage of human truth,” with Rasto Ivanic
(co-founder and CEO, GroupSolver), Leon Bourner
(Vice President, itracks) and Don Carli
(Founder and President, Nima Hunter). This guide turns that conversation into practical steps for insights professionals and the research buyers who sign off on their studies.
Synthetic respondents are AI-generated stand-ins for survey or interview participants. A large language model is prompted, or trained on existing data, to answer research questions the way a defined person or segment would.
They come in a few common forms:
Persona panels: a model answers as a described person, such as “a 35-year-old suburban parent who shops at Costco.”
Digital twins: a model built from one real person’s past answers, meant to predict how that individual would respond to new questions.
Segment-based simulators: synthetic interviews built on commercial segmentation data. Don Carli uses one built on Claritas segments for early exploration.
Uninvited AI respondents: bots, or real people pasting chatbot output, answering surveys that were meant for humans.
The first three are a deliberate method choice. The fourth is contamination. They belong in the same conversation because they fail in the same way: each produces an answer that reads well with no lived experience behind it.
Why synthetic respondents fall short on decisions that cost real money
The case against relying on synthetic data for consequential research rests on three findings the panel kept returning to.
1. They compress the range of human answers
Rasto has compared humans and synthetic respondents side by side for close to two years. The synthetic answers are plausible, he said, “but you see, without fail, the same behavior.” Humans surprise you with details from their own lives that no model would produce unless told to make something up.
Independent research points the same way. A Columbia Business School mega-study compared real people with digital twins trained on each person’s answers to more than 500 questions, across 19 preregistered studies and 164 outcomes. The twins correlated only weakly with their humans (about 0.2 on average) and their answers were less spread out.
That narrowing has a cost. As Don put it, the tails are where the risk lives: the minority views, the dissatisfied customers, the people who already left.
2. They only know what their training data knows
A synthetic respondent has no idea what happened yesterday. Rasto’s example: a candidate’s standing with voters in Saskatchewan can shift overnight because of a local event, and a model trained months earlier will miss it entirely.
New categories hit the same wall. Research from Harvard Business School on using large language models for market research found that fine-tuning a model on earlier survey data improved its willingness-to-pay estimates within a familiar product category, while the gains did not carry over to unrelated categories. Rasto’s rule of thumb follows from this: the further a question sits from what the model has already seen, the more a human answer will differ from an AI one.
3. The cost of being wrong scales with the decision
Rasto grades generic AI output on research tasks at “C+, B-minus.” That can be fine for a low-stakes direction check. It looks very different when the finding feeds a pricing change or a product launch.
Don framed the math for clients this way:
“
A compromised fifth of a $50,000 study is a problem. A compromised fifth of a $5 million product launch is a disaster.
In Qualtrics’ 2026 Market Research Trends report, 15% of researchers said they were already using AI agents, and 78% expected agents to run more than half of research projects end to end within three years. An agent can design, field and report a study today. Whether the output is worth acting on still comes down to who answered.
Where synthetic respondents earn a place in research
None of the panelists dismissed synthetic data outright. Don calls it a hypothesis engine. In his view it holds up for design work:
Pretesting a survey instrument before fielding it
Screening early concepts for direction
Trimming a long list of issues so real respondents face a lighter cognitive load
Giving no-budget clients a first look that surfaces a tension worth testing with people
Rasto also uses synthetic respondents internally, in a different role: GroupSolver runs its own bots against its surveys to see whether its fraud checks catch them.
He also challenges teams that build synthetic panels from their own CRM or survey data: skip the personas. Point your AI tool at that data and ask your question directly, and you will likely get a better answer. The persona layer puts a costume on data you already own.
Fluent fraud: AI-generated survey responses inside human samples
Synthetic respondents you chose to use are one problem. AI-generated survey responses you never asked for are a bigger one, because they arrive inside samples you believe are human.
Leon described how the challenge has moved. A fraudulent participant used to look fraudulent: inconsistent answers, speeding, poor-quality open ends. AI now helps people pass screeners with plausible answers in the right vocabulary.
“
The industry has gotten really good at checking whether someone can answer the questions, but increasingly, we need to ask whether we're dealing with the right person in the first place.
Don summed up the shift in two sentences: “The fraud used to be lazy. Now bad answers are fluent, and they’re on topic, and they’re polite.” Most detection methods were built for the lazy version, which is why a pass rate can look healthy while the data underneath it is compromised.
Real people, Rasto noted, give messy answers: typos, shortcuts, one lazy open end followed by a thoughtful one. AI is consistent in ways people are not.
The scale is real. GroupSolver publishes monthly data quality reports
, and as of October 2026 the three-month rolling average removal rate is 59%. That is the average share of respondents removed for bot activity, fraud or low quality in the last three months, across studies whose audiences and sample suppliers vary.
When contamination slips through, nothing in the report flags it. If a fifth of your sample was never a buyer, Don noted, your segment sizes, price sensitivity and feature priorities are all off. “The chart still looks like a chart.”
How to keep synthetic and fraudulent answers out of consequential research
1. Size the decision before you size the study
Start with the decision and what it would cost to get wrong. Don’s test: a client who needs an answer by tomorrow usually has an answer looking for confirmation.
2. Match verification to the stakes, and build it in early
Leon’s principle is proportionality: the more consequential the research, the more verification you can justify. In qualitative work, a short video step at recruitment goes a long way (a bot is unlikely to struggle with a frog in its throat, he joked). Building checks in up front also protects speed. His team recruited, verified and ran a swing-voter study in a small Canadian community within five days by automating the operational work and keeping the judgment about who belongs with people.
3. Layer your fraud checks
No single question catches every fake respondent. Rasto’s team combines browser fingerprinting, latency checks (a respondent who claims to be in Saskatchewan but responds like someone in Malaysia), and questions designed around how AI behaves. One favorite asks for the zip code of an Oregon town with a population of zero. A real person has no reason to know it; a model asked for a zip code will give one. Another check turns a respondent’s own open-ended answer into its opposite and asks later whether they agree. Bots tend to agree. Rasto relies on at least two or three such signals together, and looks at them as a whole:
“
There's not one question. You have to look at this holistically, and then you start catching, and you never catch 100%.
4. Build proof of life and why-probes into the instrument
Don designs for what models handle poorly: a reference to something recent, questions about anything new (a category, a price point, a shifting market), probes into the tails, and then why. “A person can tell you the thing you didn’t ask about,” Don said. “A model gives you the most probable answer to the question you did ask.”
5. Check what people declare against what they do
Don recruits and qualifies respondents through GroupSolver, then moves them into an itracks bulletin board where he can probe over several days and return to anyone whose answers did not add up. Pricing and trade-off exercises run inside the same board with the same verified people, and he layers in intent data and the client’s transaction data. “When the observed and the discussed evidence agree, I’ll sign off,” he said. “When they don’t, the disagreement is usually the finding.”
6. Read 50 verbatims before you look at the chart
Don’s quick checks: read the open ends yourself before trusting any summary. Look for specifics such as brand names, prices, places and dates; fluent answers with none of them are a warning sign. Compare the variation to the last human sample you trust. If the new one is suspiciously tight, find out why.
7. Ask your providers what they do about data quality
Rasto’s advice to research buyers is direct: ask every vendor how they verify that you are hearing from humans. The industry does not ask this enough, and the answer tells you a lot about what you are paying for.
Where humans still have to sit in the research process
AI can now design a study, field it and draft the report. Rasto named three places where the work gets worse without a person:
Defining the problem. A chatbot-written brief is competent and generic; a researcher who talks to the client adds nuance and better experimental design.
Designing the instrument. Take your own survey as a participant and notice how the flow feels.
Interpreting the results. Rasto never hands over findings without checking them against the client’s objectives himself.
That division of labor is the thinking behind Agatha, GroupSolver’s AI research analyst
, now in early access. Agatha handles cleaning and statistics and keeps its analysis inside the actual study data, which frees the researcher to form and test hypotheses.
Don put the accountability in one line:
“
The vendor carries the contract. The researcher carries the judgment. The client carries the decision.
Common mistakes in AI-assisted research
Treating “human in the loop” as an accept button
An agentic system can present so many outputs that the reviewer ends up approving outputs without evaluating them. Don compared it to a Skinner box. The fix is to give the reviewer specific things to challenge: whether the right people were asked, and whether the themes simply confirm the brief. As Don put it, “Unsurprising is a tell.” For a broader framework, see our principles for trusting AI in market research
.
Cutting sample size, then asking subgroup questions
Clients trim the sample to save money, then ask how one group compared with another, and the smaller sample cannot support those crosstabs. Decide which comparisons matter before fielding.
Presenting simulated findings as customer voice
Once a synthetic finding lands in a deck, readers cannot tell it apart from a human one. Label simulated results clearly. When a client proceeds against the evidence, Don documents the risk in writing.
Key takeaways
Synthetic respondents work as a hypothesis engine for pretesting instruments and screening early concepts. They are not a basis for decisions that are costly to reverse.
Synthetic answers cluster around the average. Independent research on digital twins found weak correlation with real people and less variation than human answers.
Models miss what is new: recent events, new categories and niche audiences.
AI-generated survey responses are now fluent and polite, and older detection methods were built to catch lazy fraud.
Layered fraud checks, proof-of-life questions and why-probes catch more than any single screener question.
Keep people on problem definition, instrument design and interpretation. Let AI take the operational work.
FAQ
Synthetic respondents is the broad term for any AI that answers research questions in place of a person, often from a persona description. A digital twin is a narrower type, built from one real individual's past answers to predict how that person would respond. Large-scale testing has found twins only modestly more accurate than generic AI personas, with weak correlation to the real people behind them.
They can help with an early screen, such as narrowing ten rough concepts to three worth testing. The final read on which concept to launch should come from verified humans, especially for a new category or a niche audience. Synthetic answers cluster around the average and miss the strong reactions at the edges, which is often where a concept succeeds or fails.
No single signal is reliable. Look for open ends that are fluent but contain no brand names, prices, places or dates; answers that agree with each other too neatly; and respondents who never contradict themselves. Behind the scenes, combine device fingerprinting, location and latency checks, and questions designed around known AI tendencies, such as the urge to answer every question.
Every extra hurdle screens out some legitimate participants along with the fraudulent ones. If the process feels like an interrogation, real respondents stop taking part, and the pool that remains skews toward people who are in it only for the incentive. Proportional checks, scaled to the stakes of the study and as unobtrusive as possible, protect quality on both sides.
Before the study is commissioned. By the time results look odd, the budget is already spent. Ask how the vendor verifies participants, what share of respondents it typically removes, and whether it publishes its methodology. A vendor that cannot answer clearly is asking you to trust a process you cannot see, and the cost of that trust grows with the size of the decision.
Bringing human truth back to the center of your research
Synthetic respondents have a real place in research: drafting hypotheses, pretesting instruments and trimming what you ask real people. The decisions that shape pricing, positioning and launches still need verified human voices, gathered with layered checks and read by a researcher who asks why. AI can carry the operational load. The judgment about who is answering, and what their answers mean, stays with you. For the itracks perspective on the same conversation, read their recap of the session
.
