A Call for Shared Responsibility in Market Research
The conversation about survey data quality usually centers on what panel suppliers can do to reduce the alarming rates of respondent fraud and prevailing low data quality. I don’t want to take the supply side of the equation completely off the hook, because suppliers are, after all, the gate keepers of the sample that gets into our doors. However, it would be unfair to not ask in the same breath that the demand side of the equation does their part too. I am talking about the collective us on the demand side: market researchers, insights professionals and our end customers who sponsor the research we do.
Ultimately, both market researchers and the end clients bear their share of responsibility for the disastrous state of panel data quality in our industry by helping drive the cost of research down to the lowest common denominator while shading their eyes from the consequences that a $1 CPI inevitably leads to.
My point, however, is not to preach and point fingers. It is more productive to focus on specific actions we must take to help our panel suppliers win the fight for better data quality. To that end, I am asking my industry colleagues for three things that I am convinced will make a real dent in solving this problem. None of these three asks is trivial, but they are 100% doable. Here they are ordered from easiest to the most difficult to affect change:
Let’s write better surveys.
This may be the easiest request I am making today, and I am yet to see an end-client or a market researcher who would disagree with making surveys better. Yet somehow, I still come across way too many surveys that make me want to go do my taxes instead.
Let’s use language that people understand.
Let’s put ourselves on the other side of the survey – how well could we understand our own questions? People today don’t have the patience to read long sentences with big words. How can we trust their answers if they can’t make it to the end of the question or matrix grid? Let’s make surveys sound less like McKinsey PowerPoint presentations and more like WhatsApp chats with friends.Let’s include more open-ended questions: Besides gaining a deeper insight into our consumers’ minds, open ends give us the most direct opportunity to understand who is sitting behind the keyboard, if there is anyone sitting there at all. Even if our survey is heavily quant focused, let’s find a reason to include a couple of open ends and give ourselves a chance to differentiate between an engaged human respondent and everyone else.Let’s make surveys shorter: In research, we ask important, intense questions. They take effort and clear thinking to answer thoroughly. Let’s not ignore the fact that even the biggest survey lovers will get tired after 10-15 minutes and their answer quality will decrease despite their best intentions. Next time, when we write a survey, let’s remember how it used to feel in college when we had to focus on linear algebra for 45 minutes straight. We were not at our best mental capacity by the time the bell rang.
Let’s talk to our clients about data quality and report back bad responses.
Data quality has simply not been an explicit enough focus in market research, although we need to applaud those of us who have stayed on it. I understand why it is easy to deprioritize taking data quality measures: setting stricter criteria will extend fielding time and possibly increase the CPI (cost per complete). Running respondents through extended trap questions and quality prescreens will extend the LOI (length of interview) and will make for a miserable user experience. Combing through collected data looking for fraud patterns adds more time to the analysis and removing suspicious respondents may mean we have to go back into the field and potentially miss the project deadline. It is tempting to just trust that our panel partner’s quality assurance processes work and only qualified respondents pass into our survey.
Sadly, turning a blind eye will likely mean that anywhere between 20% and 80% of your sample is unreliable or downright fraudulent. For example, a random sample of our projects this March showed that we removed an average of 39% of respondents due to quality after they had passed any supplier-level checks. The only way we help our industry rid itself of the data quality issue is to be good citizens and be transparent with data quality data. Let’s encourage our clients to ask to see data quality reports and let’s always send back respondents who do not pass the muster.
Let’s commit to paying for higher quality data.
A common complaint by panel suppliers is that they have little margin to invest into better data quality while the industry is depressing the cost per complete. It is hard to disagree with that. Ask yourself: what qualified human would answer questions for 20 minutes just to get $1 for their effort? Depending on the state they live in, the minimum wage in the US is between $7.50 and $17.50. You do the math.
Sure, rewards can be more than just monetary incentive and respondents may take survey during otherwise unproductive time and may not demand the same wage they earn at work (e.g., watching TV, commuting, working on something else), but even considering that and glossing over what divided attention does to data quality, that low level of incentive doesn’t pass my red face test. We need to create a functioning marketplace that reflects that good quality survey data costs more. Until we establish our willingness to pay for quality, it will be disingenuous to ask panel suppliers to get serious about data quality issues and deprioritize pushing volume.
To that end, I am convinced that our ultimate customers will pay for better quality data if we can show them the difference between making decisions based on good data vs. bad data. Ultimately for them, the cost of sample is only a minor part of their market research project budgets and that doesn’t even account for the opportunity cost of making a wrong strategic decision based on poor survey data.
The matter of trust.
Our entire industry is built on the trust and reputation we carry as professionals, and the last thing our industry can afford is losing the trust of our clients in the data and insights we provide. To me, poor data quality is an existential threat to all of us working in customer insights. Not AI-generated insights, not synthetic panels, not any other innovative technology that is yet to come out of stealth more.
The good news is that data quality is largely in our hands, if we only choose to do the right things to tackle it head on. Putting our head in the sand or waiting for others to address the issue will not do. It is us – the market researchers, consultants, insights professionals – who have the ear and the trust of our clients today. We need to take that privilege, do the right thing and force the issue toward resolution. Otherwise, as flawed as it is today, AI generated market research may start looking like a more appealing and trustworthy alternative to traditional (human) market research that our livelihoods depend on.
