Solutions
Agatha Brand Perception Brand Equity Customer Segmentation Concept Testing Product Development Pricing Research Employee Experience All solutionsIndustries
Consumer Packaged Goods Food & Beverage Retail Healthcare Financial Services Technology All industriesPlatform
How it Works
See how Agatha turns survey data into decision-ready insights in minutes.
Features
Agatha AI
NEW · Your autonomous research analyst.
LiveSlides™
Live presentations that update as data comes in.
Solver Interface
The AI-powered respondent experience.
Libraries
Reusable question templates for faster study design.
QuantQual
Quant scale and qual depth — in one study.
MCP Server
Coming soon · Your studies, native in every AI tool.
AI Open-End™
Open-ended responses, quantified at scale.
Segments
Reusable audiences across the whole study.
Respondent Manager
Data-quality control for every study.
IdeaCloud™
Open-end verbatims, visualised by support strength.
HaloGraph™
Drill from themes to verbatims in one sunburst chart.
AugmentIQ™
Coming soon · Test your survey on AI respondents before launch.
Focus Groups
Live discussions and async bulletin boards.
Multilanguage
23 languages. One study. One dashboard.
Learn
Case Studies Real research, real results Blog Insights on market research FAQ Common questions answeredData Quality
Data Quality Reports Monthly evidence of research integrity Data Quality Our commitment to research integrity
AI bots built on large language models (LLMs) can now pass many of the quality control (QC) questions survey platforms use to screen out fake respondents, but layered QC questions still stop almost all of them. In GroupSolver’s 2026 tests, 99.8% of 1,266 ChatGPT-based bots failed at least one of three randomly drawn QC questions.
Our mantra on survey data quality is simple: keep working on it. The enemy is motivated, and there is real money at stake for them to get through our defenses. If you ever needed proof that you can never take your boots off and hang your hat by the fire, here it is.
We regularly test our own QC questions against AI bots to make sure they still work. This article covers four waves we ran in 2026 with ChatGPT-based bots, and what they taught us.
Short on time? Jump straight to the specific section:
The lesson for anyone choosing a survey platform: no single QC question is safe for long. Data quality depends on testing QC questions continuously, rotating them, layering several per respondent, and testing against more than one AI model.
A quality control (QC) question, also called an attention check or trap question, is a survey question that is easy for a genuine respondent and hard for anyone who is not paying attention or not answering honestly, human or bot. A good QC question has to clear two bars at once:
The hard part is stopping bots and fraudsters without screening out genuine respondents. As LLMs get better at communicating the way humans do, writing questions that clear both bars gets harder every month.
A survey answer from an AI bot looks like a real answer. It has no typos and stays on topic. Unlike older survey fraud, it arrives fast and does not give itself away with gibberish or straight-lining.
That makes quality control questions the front line of survey data quality, alongside the other data quality measures every survey needs in 2026. The problem is that LLMs improve every few months. A QC question that worked in March can be solved by October.
So the useful question for a researcher is how a platform knows its QC questions still work. Our answer is to test them against AI bots on a schedule and publish what we learn.
We send AI bots through a real survey on the GroupSolver platform, exactly as a human panelist would take it. Every bot sees three QC questions, one drawn at random from each of three pools. A bot that answers any of them wrong is terminated, the same way a human would be.
The QC questions fall into five types. We describe them only by type, so we don’t hand bot builders and survey fraud farms the answers:
The question set grows as we add new checks. This article covers four waves run with ChatGPT-based bots in 2026:
| Wave | Bots tested | QC questions in rotation |
|---|---|---|
| February 2026 | 300 | 6 |
| March 2026 | 322 | 7 |
| May 2026 | 322 | 8 |
| November 2026 | 322 | 11 |
We measure each QC question by its catch rate: the share of bots shown that question that answered it wrong. A catch rate of 100% means the question stopped every bot that saw it. We run the same test with bots built on other LLMs, including Anthropic’s models, and refer to those results where they differ.
Only 3 of 1,266 ChatGPT-based bots answered all three of their QC questions correctly, and all three came in November.
| Wave | Bots that passed all three QC questions |
|---|---|
| February 2026 | 0 of 300 |
| March 2026 | 0 of 322 |
| May 2026 | 0 of 322 |
| November 2026 | 3 of 322 (0.9%) |
This holds even though some individual QC questions were easy for bots. In November, four questions caught no bots at all. Layering is what makes the difference: an AI bot has to beat every question it sees, and each question it sees is chosen at random. One weak question rarely lets a bot through.
Two or three QC questions per survey catch the vast majority of bots and of inattentive human respondents too. Adding device fingerprinting and other behavioral metrics cuts the odds of a bad respondent getting through even further.
The practical takeaway: a platform that relies on one attention check per survey is far more exposed than one that layers several and rotates them.
From February to May 2026, bot performance barely moved: every QC question tested in those waves stayed within about 7 points of its earlier catch rate. November broke that pattern.
Four QC questions caught no bots in November 2026
Catch rate by QC question, ChatGPT-based bots: earlier 2026 waves vs November
* QC4 changed format in November, so its November result is not directly comparable.
Source: GroupSolver QC tests with ChatGPT-based bots, February–November 2026, 1,266 bots
In November, QC2 and QC3, two cultural questions, went from catching about half and nearly all bots to catching none. QC1, a math question, had been easy for bots since March. The new QC9 caught no ChatGPT-based bots either. Meanwhile, QC5 through QC8, QC10, and QC11 still caught 90% or more. They are holding well, for now.
The drop came in a single wave. A platform that set its QC questions once and moved on would not have noticed until its data was already contaminated. To stay ahead, we keep inventing new ways to detect fraud. As Alex Tenai, data analyst at GroupSolver, puts it:
If you don't catch the fraud, the fraud will catch up to you.
Three changes in AI bot behavior explain most of the November results.
Bots stopped disagreeing with each other. From February to May, bots spread their answers across many options, so most QC questions caught some bots and missed others. By November, nearly every bot gave the same answer to the same question. That makes each QC question all-or-nothing: it either stops every bot or none. A platform that isn’t testing its questions can lose one completely without noticing.
Bots answer what a human would admit not knowing. Some of our best QC questions have an honest right answer: “I don’t know.” A real person usually can’t answer them from memory. In November, bots confidently picked a specific answer instead, and in one case it was even factually correct. Knowing too much is what got them caught. QC7, a language question, and QC6, a cultural question, each caught 97% of bots.
Bots don’t see images the way people do. Our visual QC questions ask what a person perceives. Bots tend to answer with the technical fact, or say they can’t tell. In November, QC8, QC10, and QC11 caught 99% to 100% of bots.
The pattern is clear: QC questions that test raw knowledge or reasoning are losing ground, because that is exactly what LLMs are built to do. QC questions that test honesty and human perception are holding up.
A QC question that stops one LLM can be easy for another. That is why we run this test with bots built on several models, not just one. Here is one example:
| QC question | ChatGPT-based bots | Bots built on Anthropic’s models |
|---|---|---|
| QC9 (visual, new in November) | Caught none (0%) | Caught all (100%) |
| QC7 (language) | Failed it at a high rate (85% to 97% caught) | Passed it easily |
If we had tested only against ChatGPT, we would have dropped QC9 as useless. If we had tested only against Anthropic’s models, we would have undervalued QC7. Real survey fraud uses whatever model is cheapest or most convenient, so a QC strategy tuned to one model leaves gaps against the others.
QC questions are one of our four layers of quality control. Over the three months to October 2026, GroupSolver removed 57% of survey completes for fraud or low quality (October 2026 data quality report), compared with an industry B2C average of 33% reported in the Global Data Quality Benchmarking Report, H1 2026.
In October 2026, two U.S. consumer studies closed on our platform with identical panels and safeguards. Their audiences and incidence rates were similar. The big difference was how long each stayed in field.
| Study 1 | Study 2 | |
|---|---|---|
| Time in field | 6 days (October 2026) | 78 days (August 1 to October 17, 2026) |
| Entrants | 8,872 | 10,296 |
| Incidence rate | 16% | 18% |
| Median length of interview | 13m 9s | 21m 30s |
| Average cost per interview | $5.38 | $6.30 |
| Low-quality responses removed | 774 | 3,119 |
| Clean completes delivered | 1,207 | 1,250 |
| Bad-data removal rate | 39% | 71% |
The study that stayed in field for 78 days had nearly double the removal rate. Both delivered about the same number of clean completes, because the safeguards caught what came in. Time in field was not the only difference: Study 2’s median interview also ran about eight minutes longer.
The lesson: a long field period appears to attract far more fraud, and a layered quality process is what keeps the delivered data clean. Long field periods are not the only risk. In a five-market study for a global technology brand, nearly 70% of completed responses were removed for low quality or fraud, and 6,400 verified respondents remained.
If data quality matters to your research, these questions separate platforms that test their defenses from platforms that assume them:
The sections above show how GroupSolver answers each of them, with data.
Here is what all of this means to us. We still trust the survey data that passes through our checks. AI bots are getting better, and fraudsters are learning to play the game better, but layered QC questions, device fingerprinting, behavioral checks, and human review are still catching nearly all of them.
It also means the work is never finished. Every few months, a question that used to work stops working, and the next model brings new tricks. So we keep our boots laced and our hats on, and we head back out into the elements to find new ways to keep our data clean.
GroupSolver is an intelligent survey and analytics platform that combines quantitative and qualitative research. Its AI Open-End™ technology lets respondents answer open-ended questions in their own words and then rate each other’s answers, turning qualitative input into measurable data. GroupSolver offers a DIY platform and full-service research, with fraud protection built into every study.
Book a 30-minute demo with our research team. No deck, no pitch, just the platform answering your questions.
Book a 30-min demoNo credit card · we reply within 24h