Decay in quality of synthetic panel data over time is a key driver synthetic panel economics 

Are surveys that require humans to answer researchers’ questions still necessary? Are they worth the expense, or can we now rely on GenAI technology to replace them with cheaper and quicker synthetic panels? These are the questions we’re grappling with as we watch the market research industry evolve with the rise of synthetic panel data. Is their expense justified by the value of answers they provide? Or is GenAI technology at the point where traditional survey research can be replaced by cheaper and quicker querying of synthetic panels?  

Ultimately, we believe that the decision to invest in synthetic panels will come down to the economics of a reliable synthetic (a.k.a. Gen AI) panel, which depends on a few key drivers: 

1. Accuracy of initial panel training 
2. Decay of panel’s accuracy as the world around it changes over time 
3. The number of ad-hoc surveys the synthetic panel replaces 

Our most recent paper (email us at info@groupsolver.com for the full report!) explores these questions in some detail and offers a few rules of thumb for researchers who may be considering such an investment. Here, we summarize our key theses. 

Why Synthetic Respondents Matter 

Imagine having a virtual panel of respondents at your fingertips, ready to answer your questions any time of the day. That’s the promise of synthetic panels. They simulate data creation that would otherwise require expensive surveys and interviews, allowing you to ask follow-up questions and test hypotheses dynamically.  

However, constructing well-trained synthetic panels is not trivial: it requires significant investment in recruiting and compensating human subject-matter experts to train the model, not to mention researchers’ time and skill needed to define the questions on which the panel is to be trained. 

Training Accuracy and Its Temporal Decay 

Think of a synthetic panel as an asset that delivers accurate predictions repeatedly. Like any asset, it loses value over time as the world changes. For a synthetic panel to remain valuable, it must start with high accuracy and retain it as long as possible. 

To illustrate this concept, consider the potential accuracy path of a synthetic panel. Figure 1 shows a synthetic panel in purple and a human panel in yellow. The data from both panels overlap reasonably, giving researchers acceptably similar (accurate) answers.

As the world changes, and humans update their beliefs and preferences accordingly, a synthetic panel that was trained before the world started to change, will continue to provide results consistent with its initial run. However, as we see in Figure 2, actual human answers drift away from their initial position to reflect their updated mindsets. Eventually, the overlap between the synthetic and human response decreases and the synthetic panel no longer accurately predicts real human behavior. 

The Importance of Temporal Decay of Synthetic Panels 

This concept of temporal decay is crucial for understanding the longevity and return on investment (ROI) of synthetic panels. It is influenced by several factors: 

• Rate of Change in Consumer Preferences: Topics with rapidly changing preferences, like pop culture, will experience faster decay compared to more stable topics such as medical diagnoses. 

• Frequency of Retraining: Regular updates and retraining of the synthetic panel can mitigate the effects of temporal decay, ensuring the panel remains relevant and accurate. 

• Initial Training Quality: The better the initial training accuracy, the longer the panel can remain useful before significant decay sets in. 

Understanding and managing temporal decay is essential for maximizing the return on investment in synthetic panels. Researchers who pursue synthetic panels must strive to understand the speed of panel’s temporal decay and consider the costs of continuous retraining to maintain high accuracy over time. 

Asset vs. Expense: A Foundation of Synthetic Panel Economics 

When thinking of synthetic panels as an asset, it’s essential to understand that they represent a long-term investment. Like any asset, they require an initial outlay of resources but can provide ongoing value over time. Compare that to traditional surveys, which are in essence a series of expenses incurred each time new data is needed. Here are the factors that impact the balance of economics between an asset and series of expenses: 

1. Initial Investment vs. Recurring Costs: Building a synthetic panel involves a significant upfront cost, but it can be used repeatedly without incurring the same level of expense. Traditional surveys require recurring costs for each new survey. Their relative size is the key consideration. 
2. Longevity vs. Maintenance Costs: Synthetic panels need periodic retraining to maintain their value, similar to maintaining a piece of machinery. Traditional surveys do not have this issue because each time we survey human brains, they come to answer our questions with already “updated” preferences. 
3. Agility: Synthetic panels offer quick insights, enabling faster decision-making. Traditional surveys can be time-consuming, potentially delaying critical business decisions. 

Rule of Thumb Example 

Estimating the useful shelf life of a synthetic panel is challenging without an accurate and practical temporal decay function estimate. However, we can use some commonsense assumptions to get in the ballpark.  

To construct a reasonable general population synthetic panel, including recruiting fees, honoraria, platform licenses, and data scientist fees, you can expect to spend around $35,000 to $45,000. 

In contrast, conducting ad-hoc surveys can be less expensive initially, with a robust study sample costing about $5,000. Since the ROI for a synthetic panel depends on its frequency of use, the panel must replace approximately 7-9 surveys to pay for itself. 

Of course, the more surveys a synthetic panel can replace, the better its economics. However, we must recoup the investment before the panel becomes obsolete. For instance, if a brand conducts a survey once per year, it would take 7-9 years to recoup that investment. Will the panel be still relevant so many years after its initial training? Perhaps in industries that see little change. But in our fast-moving consumer world of today, most panels would be well past their due date by year 2 or 3. 

Conclusions 

Trust in a synthetic panel’s accuracy is essential for its value proposition. Researchers must understand the decay curve of their panels to know when they should stop relying on them for critical business decisions. Further empirical research is needed to establish practical accuracy decay benchmarks for market research practitioners. Until then, we can refer to simple rules of thumb to guide our decision making. The future of market research is being shaped before our eyes, and synthetic panels are set to play a significant role. It’s crucial for the industry to stay ahead of this exciting technology, both technically and economically.