Surveys are slop because they ask people to describe their own behaviour, and people cannot do this accurately. The instrument and the thing it claims to measure are structurally incompatible. A survey lives entirely in the slow, rational, narrating part of the mind. Behaviour is driven by the fast, emotional, automatic part. When you ask a person why they bought something, what they believe, or what they will do next, you are not collecting data. You are collecting a story. And the story is wrong in a predictable direction.

This is not an argument that surveys never work. Surveys are useful when people can report facts they know firsthand and have no reason to distort. They become slop when companies ask people to explain why they acted, predict what they will do, or describe identity-driven preference. In those cases, the survey captures narration rather than behaviour. Better survey design reduces the damage. It does not make self-report a clean substitute for behaviour. The stronger instrument is triangulation: what people do, what they say when no one is asking, and what independent records reveal.

What is wrong with surveys, exactly?

Surveys measure stated intention. Decisions are made by revealed behaviour. The gap between the two is one of the most replicated findings in behavioural science, and it runs in a single direction: people overstate.

Consider the most basic version. Across 77 studies and more than 24,000 hypothetical responses, the average gap between what people say they will pay and what they actually pay is roughly 21 percent (Schmidt and Bijmolt, Journal of the Academy of Marketing Science, 2020). In environmental and resource economics, subjects overstate their preferences by a factor of about three in hypothetical settings (List and Gallet, 2001). In one Dectech/Warwick study comparing stated and revealed preference against real UK supermarket sales data, revealed preference explained 49 percent of observed variance while the best stated-preference model explained 32 percent. That is a meaningful improvement in predictive accuracy from watching what people do rather than asking them — one study, not a universal law, but consistent with the broader pattern.

The consumer-facing version is starker. Thirty percent of consumers say they will buy ethical products. Three percent actually do. That is a ten-to-one gap between intention and behaviour. Blood donors told researchers they intended to give at a certain frequency; the intended frequency exceeded the actual frequency by 41 percent for men and 30 percent for women across the main study sample (de Corte, Cairns and Grieve, Health Economics, 2021).

The pattern is not noise. Noise is random and cancels out in large samples. This is bias, and bias compounds. A bigger sample of the wrong instrument does not get you closer to the truth. It gets you a more confident version of the wrong answer.

The error usually leans the same way

What makes the say-do gap so dangerous is not just its size but its tendency to lean in one direction. People do not always distort, and some contexts produce understatement, satisficing, or simple indifference. But in the cases that matter most to companies, the error leans predictably: toward the answer that flatters them, toward the socially approved choice, toward the version of themselves they would like to be.

This directional consistency is why averaging cannot save you. If errors were random, a thousand responses would cancel out into something close to truth. But when respondents lean the same way, the average simply encodes the shared lean with greater statistical confidence. You end up with a tight, narrow, beautifully precise distribution centred on the wrong number. The confidence interval shrinks. The error does not. This is a dangerous failure mode for a decision tool because it manufactures a sense of certainty about a result with little contact with reality.

Wearable abandonment is high: some reports find roughly a third of buyers stop using fitness trackers within six months, and more than half eventually abandon them. Online course data routinely show a large gap between purchase and actual learning behaviour, with many buyers never engaging with the material they paid for. And in a controlled experiment, people who believed they had taken a dietary supplement showed reduced desire to exercise, a stronger pull toward a buffet over a healthy meal, and walked less (Chiou et al., Psychological Science, 2011) — a licensing effect, not a claim about all supplement users. In each case the behaviour diverges from the belief, and a survey would only ever capture the belief.

Why can’t people describe their own behaviour?

People cannot describe their own behaviour because experience and explanation happen in two different systems of the mind, and only one of them can talk. You live your decisions in the fast, automatic, emotional system. You report them through the slow, rational, narrating system. The survey only ever reaches the narrator. The narrator was not in the room when the decision was made.

This is the crux of the argument, so it is worth sitting with. When a person answers a survey question, they are not retrieving a fact. They are constructing an explanation that feels coherent, makes them look reasonable, and matches what they believe a sensible person would say. None of those three goals is “be accurate.”

There is a name for the confidence trap underneath this. The Illusion of Explanatory Depth (Rozenblit and Keil, Cognitive Science, 2002, twelve studies at Yale) shows that people believe they understand complex things with far more precision and depth than they actually do. When asked to explain in detail, their confidence collapses (p < .001). The effect is strongest for exactly the kind of explanatory knowledge a survey demands: not “do you like this,” but “why did you choose this.” People feel they know why. They mostly do not.

There is also a deception trap that runs in two layers at once. The Paulhus Deception Scales (University of British Columbia) identify Self-Deceptive Enhancement, an unconscious tendency toward grandiosity and overconfidence that is a stable personality trait, and Impression Management, a conscious, strategic exaggeration that shifts with context. When a person fills out a survey, both run simultaneously. They are managing your impression of them and their own self-image at the same time, and they are fully aware of neither.

The “how do you buy software” problem

A simple illustration of the problem is to ask people how they make a decision they make constantly. Anecdotally, when buyers are asked how they buy software, they tend to point to the visible, recent, rational touchpoint: “we start with the website.”

They rarely start with the website. They start with a conversation they half-remember, a competitor a peer mentioned at dinner, a logo they saw three times in a month, a vague sense that a category exists. By the time they reach the website, the decision is often most of the way made, and the website’s job is to confirm a choice the limbic system already reached. But the website is the part they can see and narrate, so the website is the answer they give.

This is the problem in one sentence. The website is visible, recent, and rational, so it becomes the reported cause. The real drivers are invisible, diffuse, and emotional, so they fade from the report. Now build a multi-million-dollar go-to-market strategy on the survey answer. You have optimized for the part of the journey the buyer can describe and underweighted the part that actually moves them.

The decision sits even deeper than people admit. Harvard’s Gerald Zaltman has estimated that much purchase decision-making happens subconsciously — a figure often cited as high as 95 percent. That includes business-to-business decisions, the ones wrapped in requests for proposals, ROI spreadsheets, and procurement committees. The 47-page evaluation matrix is often less about how the decision gets made and more about how it gets justified after the limbic system has chosen. A Google/CEB/Motista study found B2B buyers can be more emotionally connected to their vendors than consumers are to many consumer brands. “Nobody ever got fired for buying IBM” was never about IBM’s quality. It was about career protection. The stated reason is best-in-class features. The real driver is “I need to be able to defend this if it fails.” A survey will faithfully record the first and rarely see the second.

How NPS became the most expensive survey in business

Net Promoter Score is the clearest example of survey weakness operating at the scale of the entire economy. It is among the most prevalent survey instruments in business, used by roughly two-thirds of the Fortune 1000, and its limits are documented in the same peer-reviewed literature that the companies relying on it rarely read.

NPS asks one question: How likely are you to recommend us on a scale of 0 to 10? The appeal is obvious. It is one number. It fits on a dashboard. It survives a board meeting. SAP paid $8 billion for Qualtrics in 2019, largely on the strength of survey infrastructure built around this kind of metric. The metric carries enormous institutional weight.

The academic record is unflattering. A longitudinal study using more than 15,500 interviews across 21 firms found that NPS was the best or second-best predictor of growth in only two of five industries (Keiningham et al., Journal of Marketing, 2007). Separately, Morgan and Rego (2006) found no evidence that NPS was superior to other loyalty metrics. Even Fred Reichheld, who introduced NPS in Harvard Business Review in 2003, conceded in his 2021 update that “self-reported scores and misinterpretations of the NPS framework have sown confusion and diminished its credibility.”

Then there is the phantom-score problem, which is structural rather than statistical. Rob Markey, who co-leads NPS work at Bain, has stated plainly that in any customer population, “the most likely responders are drawn from the ranks of Promoters. The least likely to respond are the Detractors.” Bain’s own worked example shows how, at a 20 percent response rate, a reported NPS of 50 can mask a true NPS of minus 22 — a 72-point swing. In practice, many NPS programs operate on low response rates; one B2B benchmark puts the average near 12 percent, with wide variation by company and audience. So the headline number is often built on the most satisfied minority, who self-selected into answering, measured with a method its own inventor says has been hollowed out by misuse. Companies set strategy by it anyway.

The seven layers of contamination

Seven common distortions can contaminate self-report — especially when the question asks people to explain motives, predict future behaviour, or perform virtue. Each is established in peer-reviewed research. Not every survey carries every distortion with the same intensity, but the more a survey leans on motivation and prediction, the more of them switch on at once, compounding.

The point of listing them is to show that survey error is not simply a flaw you can engineer away with better wording. It is the sum of distinct failure modes, several of which are usually present together.

1. Framing decides what can be discovered. Tversky and Kahneman (Science, 1981) demonstrated complete preference reversals from logically identical problems phrased differently. Every survey question is a frame. “How satisfied are you?” and “What problems have you had?” produce different realities from the same respondent.
   
2. The hypothesis can harden before the evidence arrives. Early hypothesis formation is a legitimate and useful research practice. But when the hypothesis sets before the data lands — often shaped by the client’s existing narrative — the research quietly becomes confirmation machinery, gathering support for a conclusion already reached rather than testing it.
   
3. Respondents perform for the researcher. Orne (American Psychologist, 1962) named demand characteristics “the totality of cues which convey an experimental hypothesis to the subject” and concluded that it is futile to imagine a study without them. Later work (Corneille and Lush, 2023) showed these cues do not merely change answers; they can change the respondent’s actual subjective experience.
   
4. Agreement inflates everything positive. Acquiescence bias can raise the estimated prevalence of a belief by up to 50 percent in agree/disagree formats (Hill and Roberts, 2023). Social desirability bias does the rest, pushing people to overreport virtues and underreport vices.
   
5. Non-response creates phantom data. Covered above with NPS, but it applies to nearly every voluntary survey. The people who answer are not a random slice of the people who matter.
   
6. Interpretation bends toward belief. Confirmation bias, “perhaps the best known and most widely accepted notion of inferential error” (Nickerson, Review of General Psychology, 1998), means even clean data gets read in favour of what the reader already thought.
   
7. Self-knowledge is hard from the inside. Argyris and Schon (1974) showed that the theory governing a person’s actions, their theory-in-use, differs from the theory they espouse, and that “few people are aware that the maps they use to take action are not the theories they explicitly espouse.” A survey can usually capture only the espoused theory. The real one tends to stay invisible to the person holding it.

Run a number through several multiplicative distortions and the output is not slightly off. It can be a different number wearing the costume of precision.

What does survey slop actually cost?

Survey slop is not a curiosity in measurement. It can produce named, dated, quantified business failures, because companies act on the bad data with real money.

New Coke: 4 million dollars of research that asked the wrong question

In 1985, Coca-Cola ran one of the most exhaustive market research programs in corporate history: taste tests with nearly 200,000 consumers at a cost of about $4 million. The new formula won 53 to 47 over the original Coke and beat Pepsi by six to eight points. In branded tests, the margin widened to 61:39. The CEO called it “the surest move we have ever made.”

Seventy-nine days after launch, the company brought back Coca-Cola Classic amid open revolt. By Coca-Cola’s own account, the research measured sensory preference but failed to capture the emotional bond consumers had with the brand. The taste test reached the rational palate. It never touched the emotional attachment and cultural identity that actually drove behaviour. Focus groups had whispered the warning. Management downweighted the soft signal in favour of the hard, quantified, and wrong one.

Tropicana: tens of millions destroyed in weeks

In 2009, Tropicana, then holding 30 to 35 percent of the US juice market and around $700 million in annual US sales, redesigned its packaging and backed it with a campaign reported at roughly $35 million. The iconic orange-with-a-straw vanished. Within about two months, sales fell 20 percent, roughly 30 million dollars in lost revenue, and the company reverted with a full-page “We hear you.” A later visual-attention analysis (not part of the original business case) found that the redesign drew only about 2.5 percent of attention to the logo, compared with 10.8 percent for the original. The widely repeated total cost of $50 million or more and the $35 million campaign figure circulate across many secondary sources rather than from a single primary audit. Whatever testing was done did not capture how people actually find their juice on a shelf, because shelf-finding is automatic behaviour, not a reportable opinion.

The 80/8 delusion

The gap between what companies believe and what customers experience has a number. In Bain’s 2005 “delivery gap” study of 362 firms, 80 percent believed they delivered a superior experience. Customers said only 8 percent of companies really did, a 72-point gap. This figure is now two decades old and has not been formally replicated at the same scale, so treat it as influential and directional rather than current. The current echo comes from a 2026 McKinsey survey of 1,257 executives, which found most are confident they understand what drives customer choice, “a confidence that may be misplaced as change accelerates.” Organizations that actually track market-level perception were found to be more than 2.5 times as likely to outperform peers.

The downstream cost: building strategy on a fault line

The deepest cost of survey slop is not the survey’s price. It is everything built on top of it. A flawed positioning assumption does not stay contained in a research report. It cascades into campaigns, product roadmaps, pricing, packaging, and board decks, each layer compounding the original error.

Consider the architecture of the waste, as a back-of-envelope estimate rather than a measured audit. Global advertising spend has surpassed a trillion dollars a year (WARC), and in one survey, marketers estimated they were wasting about 26 percent of their budgets (Rakuten Marketing). Extrapolating self-reported waste against total spend implies hundreds of billions of dollars annually directed at messages, audiences, and positions that do not work — an inference, not an audited figure. The research industry that exists specifically to prevent this waste is built largely on the same survey methodology that can produce it. The instrument meant to be the cure can share the disease.

This is what makes survey slop a structural problem rather than a line-item inefficiency. A company commissions a study to understand its market. The study, contaminated by several of the distortions above, returns a confident but distorted picture. The company allocates a campaign budget against that picture. The campaign underperforms. The company commissions another study to understand why, using the same method, and gets another distorted answer. The loop never closes because the feedback mechanism is broken in the same way as the original measurement. You cannot reliably debug a system using the instrument that introduced the bug.

The failures that make headlines, New Coke and Tropicana among them, are simply the cases where the gap between the survey’s picture and reality became too large to absorb quietly. The far higher cost is invisible: the thousands of campaigns that merely underperformed, the products that launched into indifference, the positioning that quietly failed, many traceable to a confident answer to a question the respondent could never honestly answer.

If surveys are so flawed, why do companies keep buying them?

Companies keep buying surveys because surveys are often not really hired to produce insight. They are hired to instill confidence, provide cover, and create the appearance of diligence. Once you see the real job, the persistence of a weak tool no longer seems a puzzle.

The foundational research on this is direct. Moorman, Zaltman, and Deshpande (Journal of Marketing Research, 1992, n=779) found that trust and the perceived quality of the interaction, rather than the findings themselves, contributed most to whether the research was actually used. Firms invest in research partly as an anxiety-management mechanism. Part of the deliverable is calm.

You can see the same pattern in how decisions actually get made:

• Political cover. “Nobody gets fired for hiring McKinsey.” The branded study can function as blame insurance. If the decision fails, the executive points to the research.
  
• The HiPPO effect. When data and opinion collide, the Highest Paid Person’s Opinion often wins. As Netscape’s Jim Barksdale put it, “If we have data, let’s look at data. If all we have are opinions, let’s go with mine.” Surveys are sometimes commissioned to ratify a view leadership already holds.
  
• Data theatre. Gartner (2022, n=377) found marketing analytics influenced only 53 percent of marketing decisions. One-third of decision-makers cherry-picked data to fit a conclusion already reached. Twenty-six percent did not review the analytics at all. Forrester has described most business-intelligence efforts as “data to insight to hoping for action.”

This connects to a deeper organizational truth. Chris Argyris, in his 1977 work on single- and double-loop learning, described a multi-billion-dollar product that five people knew was failing six years before it was killed. The information never reached the top because organizational norms and career protection filtered it into silence. A survey commissioned inside that same organization, by the same people, optimizing for the same self-protection, can produce the same filtered output. The survey is single-loop by design. It can measure performance against the existing plan. It struggles to ask whether the plan itself rests on a false assumption, because the people who would have to ask are often the people the survey is protecting.

What people say versus what their brains do

Peer-reviewed neuroscience shows that self-report can diverge from what people’s brains actually do — which means asking them is not the same as knowing. Neural response is evidence, not an oracle; behaviour remains the higher standard. But when self-report, behaviour, and neural response disagree, self-report should not get the tie-breaker by default.

• The Pepsi paradox. In blind taste tests, preference between Coke and Pepsi splits roughly evenly. When people know what they are drinking, the balance swings sharply toward Coke, and brain regions tied to memory and meaning activate (McClure et al., Neuron, 2004). People are not only tasting the drink. They are tasting their own associations. No survey question can separate the two, because the respondent cannot either.
  
• Price as pleasure. When shown the same wine at a higher price, people’s experienced pleasantness regions activate more strongly (Plassmann et al., PNAS, 2008). Price does not merely signal quality; it shapes the experience of enjoyment. Ask why they liked it, and they will describe the wine, not the price tag that moved them.
  
• A weaker but suggestive case. Some neuromarketing work — notably claims that cigarette warning labels can activate craving rather than fear — points the same way, but it comes from a less rigorous literature that has drawn methodological criticism. Treat it as illustrative, not as proof.

The throughline holds in the peer-reviewed cases. Self-report captures the conscious story. Behaviour and neural response often tell a different one. They diverge in ways the survey, by design, can never see.

What works instead of surveys?

The fix for survey slop is not a better survey. It is to stop treating self-report as the source of truth, and to use it as one weak signal beside behaviour, records, and unprompted market evidence. You stop relying on the narrator and start watching what people actually do.

The principle is straightforward. Most distortions described above share one root cause: the act of asking. The moment a person knows they are being measured, framing, demand characteristics, social desirability, self-deception, and impression management tend to switch on. Some questions still have to be asked — but the less weight you place on the asked answer, and the more you place on observed behaviour, the cleaner the picture.

That means triangulating unprompted voices from rooms where the company is not present:

• What customers say to each other when the brand is not listening — reviews, forums, and peer conversations written for other buyers, not for the company.
  
• What the record shows when no one is performing — regulatory filings, complaint databases, hiring patterns, and other behavioural traces that exist independent of any survey.
  
• What the market reveals through action — what people actually buy, search, abandon, return, and renew, rather than what they say they intend to do.

No single unprompted source is perfect; each carries its own selection effects. But triangulating across many independent, behavioural sources removes the one bias they would otherwise share: the bias introduced by being asked.

Why triangulation beats interrogation

The logic of triangulation is borrowed from navigation and from intelligence work, and it is worth making explicit. A single source of unprompted data has its own selection bias. Online reviews skew toward the extremes, the delighted and the furious. Regulatory filings capture only what is legally reportable. Search behaviour reflects intent but not satisfaction. Each source, taken alone, is partial.

But these biases do not point in the same direction. Review extremity has nothing to do with regulatory reporting thresholds, which have nothing to do with search-volume patterns. When a signal appears consistently across sources whose biases are unrelated and cannot coordinate, the probability that the signal is a bias artifact drops sharply. What survives across independent, uncoordinated, unprompted sources is more likely to be real than any answer a single survey could produce, because the one bias all those sources avoid is the bias of being asked.

This is the precise inversion of the survey model. A survey concentrates all its eggs in one contaminated basket: a single instrument, a single moment of asking, a single set of incentives acting on the respondent. Triangulation spreads the measurement across many baskets that fail in different ways, so their failures tend to cancel rather than compound. The survey averages many people through one biased lens. Triangulation averages many lenses, each catching the behaviour from a different, unguarded angle.

There is one more advantage that matters for high-stakes decisions: speed and independence. Traditional survey-based brand tracking arrives in waves, quarterly or annually, creating months of lag between the market moving and the data landing. Unprompted behavioural signals exist continuously and can be read on demand. And because they come from outside the company, they carry a credibility internal research cannot. An executive presenting a survey their own team designed is presenting a closed loop. An executive presenting what the market revealed when no one was asking is presenting evidence.

The deeper reframe is this. You cannot read the label from inside the jar. A company asking its own customers, through its own survey, interpreted by its own team, is running a closed loop in which every step shares the same incentives. Truth about perception is more reliable when it comes from outside that loop, from voices that were never prompted and evidence that was never staged.

Interrogating the assumption everyone inherited

The reason surveys persist is that the assumption underneath them is rarely questioned: that a person is a reliable witness to their own mind. It is worth asking the question directly, because so much rests on the unexamined answer.

Why do we assume self-report is accurate? Largely because it is convenient and familiar, not because it is always true. Asking is cheap, fast, and feels democratic. It produces clean numbers that slot into spreadsheets. It has been the default for so long that it has become invisible, treated as the natural way to understand people rather than as one method among many, each with specific limitations.

What would still be true if we removed that assumption? The behaviour would remain. Sales would still happen. Reviews would still be written. People would still abandon carts, renew subscriptions, recommend products to friends, and switch brands. All the evidence of what people actually want would continue to exist, recorded in their actions, entirely independent of whether anyone surveyed them. The survey is not the source of the truth. The behaviour is. The survey is a lossy, biased proxy that is easy to mistake for the thing itself.

What is the irreducible core of understanding a market? It is knowing what people do and carefully inferring why. Much of what a survey adds on top of that — the stated reasons, the rated intentions, the satisfaction scores — is data about the stories people tell, not data about behaviour. Useful sometimes. Decisive rarely. Once you see that the behaviour was always there to be observed, the question shifts from “how do we write a better survey” to “how much weight should the asking really carry.”

Frequently asked questions

Are all surveys useless?

No. Surveys are useful for narrow, factual questions where the respondent knows the answer directly and has no reason to distort it — for example, “which version of the software are you running?” They fail specifically when used to explain motivation, predict future behaviour, or measure emotional and identity-driven preference, which is much of what they are actually used for.

Isn’t a bigger sample size the answer?

No. A larger sample reduces random noise, but survey error is often systematic bias, not noise. Bias does not cancel out as the sample grows. A larger sample from a biased instrument yields a more confident wrong answer, not a more accurate one.

What is the difference between stated and revealed preference?

Stated preference is what a person says they want or will do, captured by asking. Revealed preference is what a person’s actual choices demonstrate they want, captured by observing behaviour. Across the research literature, revealed preference tends to predict real outcomes better than stated preference; one supermarket study found that revealed preference explained 49 percent of the variance, versus 32 percent for the best stated-preference model.

Why is NPS still so widely used if it is flawed?

Because it is simple, fits on a dashboard, and provides a single defensible number for board meetings and executive reviews. Its prevalence is largely a function of institutional convenience and risk reduction, not of demonstrated superiority over other loyalty metrics.

What should replace surveys for understanding customer perception?

Not a replacement so much as a re-weighting: treat self-report as one weak signal, and lean on unprompted behavioural evidence from independent sources — what customers say to each other, what regulatory and public records reveal, and what the market does through actual purchasing. Triangulating across multiple independent sources removes the shared bias that asking introduces.

Finally

Surveys are not useless. They become slop when companies use them to explain motivation, predict future behaviour, or measure emotional, identity-driven preference. For those jobs they reach the conscious narrator and miss the unconscious decider. They ask people to explain behaviour they cannot see, in a setting that rewards performance over truth, through several compounding distortions, and then dress the result in the language of precision.

The evidence is consistent. The say-do gap is one of the most replicated findings in behavioural science. NPS, the most prevalent business survey on earth, has repeatedly failed to prove the superiority it claims. New Coke and Tropicana are reminders that research can measure the wrong thing flawlessly. And the companies that keep buying surveys are often, on the evidence, buying confidence and cover as much as they are buying truth.

The alternative is not a cleverer questionnaire. It is to stop making self-report the source of truth and start triangulating: what people do, what they say when no one is asking, and what independent records reveal. Watch the behaviour. Read the voices that were never prompted. You cannot read the label from inside the jar — and the survey, for the questions that matter most, is just an expensive way to keep trying.

A note on sources and certainty: figures here are drawn from named studies wherever possible. Some widely cited business numbers (the Tropicana total cost, the ad-waste extrapolation, and the 2005 Bain 80/8 gap) circulate through secondary sources or are now dated, and are flagged as directional rather than audited. The argument does not rest on any single figure; it rests on the consistent direction of the evidence.