Survey Respondents Say They'll Buy. Will They Actually?
You've probably seen this one. The concept that won the survey turns out to be the one that sells worst after launch.
The usual explanation is a bad sample or a badly worded question. Most of the time, that's not it.
A survey records what people say, and people are surprisingly bad at predicting what they'll actually do.
The share who say "I'd buy it" and the share who actually buy often don't line up. Marketing research has confirmed this for decades.
So why do companies keep fielding surveys every season? Because until now there was no other way to see a reaction before launching for real.
This post is about that other way. In our last post, we listed consumer surveys as one of the three ways companies set prices today, and noted that responses and actual purchase behavior diverge. This time we'll look at how simulation handles the survey itself.
Surveys go wrong because of the question, not the sample.
Think about a time a survey result diverged from reality. It usually fits one of three patterns.
Asking about something people have never used
A pre-launch concept, a plan that doesn't exist yet, a package they're seeing for the first time.
Respondents can only answer from imagination, and imagination is generous. A product on a concept board always looks better than the same product on a shelf.Asking about a decision far in the future
"Will you still be using this service a year from now?" People answer by projecting today's budget and mood straight ahead.
That a competitor will launch, prices will change, and their own circumstances will shift doesn't enter the calculation at the moment they answer.When the asker has a stake
Satisfaction asked by the brand manager, repurchase intent asked by the account rep: a polite yes creeps in.
Respondents aren't lying. In that setting, that's simply the natural answer.
None of these goes away by polishing the questionnaire. They're not a sampling-design problem. They're the limit of asking people about their own future.
On top of that come two practical pains.
Time and cost
A study takes weeks, sometimes months, from design to results, and changing a single condition means paying for another round. By the time results land, the market has often moved.The sample
The people you most need to hear from are the hardest to recruit: chronic patients, high-net-worth investors, specialized professionals, B2B buyers. When recruiting fails, a less representative panel stands in, and the reliability of the results drops with it.
The most accurate answer is to launch for real and run the campaign for real. But that's exactly the cost a survey was meant to avoid.
So you need a way to see the outcome before you ask.
A Social World Model changes who you ask.
Dalpha's Social World Model solves this by changing the respondent.
Instead of asking people "what would you do?", it puts the same survey to a model trained on what people actually did: purchases and churn, reviews and clicks, and past survey answers paired with what those respondents went on to do.
One distinction matters here. If you've ever told ChatGPT "answer as a woman in her 40s with an office job," this isn't that.
A persona given only demographics drifts to the average, folds when you push back, and has no grounding for why it answered that way.
A Social World Model learns, per individual, how attitudes connect to behavior. Two women in their 40s with different purchase histories, different reviews, and different exposure to public opinion answer differently, and the model carries that structure. So the output isn't a single average. It's a distribution by segment.
Here's how it runs.
Build the respondent pool. The starting point is what the company already has: past survey questions and answers, respondent profiles and metadata. Dalpha adds social, review, and community sentiment data it collects itself, plus demographics, to correct the pool's representativeness.
Run the survey. Enter the questions along with the stimulus: concept copy, creative, message wording, price tier, exactly what a real respondent would be shown.
Get the response distribution. Multiple-choice items come back as distributions by question and by segment. Open-ended items come back as the distribution of recurring reasons. You see why they picked concept 2, not just that they did.
A survey stops being a one-off report and becomes an asset you can query again.
A respondent pool outlives the report. A traditional survey ends the moment the deck is delivered, but a simulation lets you do three more things afterward.
Ask the same pool again with the conditions changed.
Try a different price tier, narrow the target, expand to three message versions.
After a competitor launches or interest rates move, you can put the question to the same pool again in short order.Ask groups that were impossible to recruit.
Chronic patients, high-net-worth investors, policy staff: people no incentive brings into a panel. Their reactions are built from the behavioral traces they've left.
Spend the research budget differently.
Instead of fielding all five concepts to a real panel, you narrow to two in simulation and send only those two to real research.
The budget shrinks, and the real panel concentrates on the questions with the most uncertainty.
Where does it get used first?
Look at the surveys you're running now and ask which ones fall into the three traps above. From where Dalpha sits, three places come first.
CPG : pre-launch concept and package testing
This is the work of product planning and consumer insights teams that screen concepts every season. Expose five candidates to the same respondent pool at once, get purchase-intent distributions by segment with the reasons behind them within a day, and send only the top two to a real panel study.
This is the survey most exposed to the first trap: intent for something no one has used yet.Healthcare, pharma, finance : research on hard-to-reach groups
The groups that matter most to the outcome, like chronic patients or people with a specific treatment history, are the slowest and most expensive to recruit. Simulation lets you compare reactions across combinations of reminder cadence, education content, and benefit design before recruiting anyone.
Finance sits in the same spot. Regulation makes experiments on real customers hard to run, so simulating reactions to a product change or a fee adjustment is one of the few ways to test in advance.Research and consulting firms : client report delivery
The same logic applies to firms that run surveys for others. Run the client's questions through a simulated panel first to sharpen the study design and shorten the report cycle.
When market conditions change after delivery, re-run the same pool for a follow-up report.
What these three have in common is clear. The research budget line already exists, the same question repeats on a schedule, and the people who need to be asked rarely show up.
Real surveys aren't going away.
Simulation doesn't replace every survey. It's weak on stimuli with no behavioral trace at all.
If nothing like the product has ever existed, there's no behavior for the model to learn from.
And to be candid, confidence varies by item. Some items land very close to observed results. Others show wider error.
That's why Dalpha fixes the order of validation. First, we replay a study the client actually ran last year, with the results hidden.
We check how closely the distribution matches question by question and where it diverges.
Start by getting the known answers right, then move on to questions nobody has the answer to yet.
Seen this way, simulation isn't a tool for eliminating surveys. It's the step before them, the one that decides where real fieldwork goes.
The real panel goes to the items the simulation flagged as most uncertain, and to the decisions where a miss is most expensive.
In short : real research goes only where the uncertainty is.
The real limit of surveys isn't that they're slow and expensive. It's that what people say and what they do are different.
A Social World Model puts the same survey to a simulated respondent pool trained on what people actually did, instead of asking people.
Real research doesn't disappear. It concentrates on the questions with the most uncertainty.
If you're spending survey and focus-group budget on the same questions every season or every quarter, running those questions through a simulation first is a place to start.
Frequently Asked Questions (FAQ)
Q. How can I trust a simulated respondent's answer?
We validate first by backtesting: replaying a survey you actually ran, with the results hidden, and checking how closely the response distribution matches question by question. Even so, the best results come from treating the output as evidence with reasons attached, not as a verdict, and making the final call internally. Items that come back with low confidence can be confirmed with real fieldwork.
Q. So can we stop running surveys altogether?
No. Simulation is the step before real research, the one that decides where to look. Narrowing candidates, sharpening questions, and flagging the least certain items is the simulation's job. Real research on those items is still needed. What changes is how often and how widely you survey, not whether you do.
Q. Can we start without much past survey data?
Past questions and answers speed up the backtest, but you can start without them. The respondent pool can be built from social, review, and community sentiment data plus demographics. The first step is checking together what data you have and what's missing.
