Breaking the tie with a price penalty
When evaluating how AI shopping assistants respond to seller copy, testing products at identical prices can easily hit a measurement ceiling. In controlled comparisons of headphones, GPT-4o selected the option featuring a strong social-proof claim—copy emphasizing approval from other buyers—in all 144 equal-price trials. Yet a product carrying a bland, neutral factual sentence also achieved a near-perfect win rate. Judged solely by selection frequency at parity, both messages appeared equally persuasive.
Introducing a price penalty for the cued product broke the tie immediately. A price increase of roughly 2% was enough to erase the advantage of the neutral sentence. By contrast, the strong social-proof claim maintained its edge until the price premium reached an estimated 12.9% (with a reported 95% confidence interval of 10.9–17.1%). Behind nearly identical parity win rates lay a roughly sixfold difference in how much extra cost the model would tolerate.
These findings reflect historical behavior from a rolling GPT-4o API alias rather than a locked model snapshot, meaning they capture the endpoint at the time of testing rather than a permanent baseline.
In this methodology, the threshold represents the tipping point where the cued product's pooled selection share drops to 50% across both presentation orders. It does not indicate total rejection; at that mark, the pricier item still wins half the time.
This highlights a core flaw in standard AI auditing. Once two prompts or product descriptions win nearly every time at equal prices, simple selection share runs out of room to differentiate them. Forcing the model to trade off its preference against a rising cost reveals how deeply that preference is actually anchored.
Untangling persuasion from price apathy
Using price tolerance as a yardstick introduces its own complication: the metric blends the pull of the marketing copy with the model's underlying sensitivity to cost. If a model barely registers price hikes, it will look wildly persuadable even when the copy itself exerts only a mild influence.
This dynamic surfaced clearly in cross-model testing. For several different cues, Nova Micro continued picking the promoted product even when its price was marked up by 110%—more than double the cost of the competing option. Rather than proving a 110% persuasion threshold, this meant the test never actually crossed the 50% tipping point. The researchers rightly flagged this as price insensitivity rather than extraordinary gullibility.
The inverse is equally true. A model programmed to heavily penalize even minor markups will post a very low price threshold, even if it overwhelmingly favors the cued item when prices are matched. Consequently, price tolerance cannot stand alone as a leaderboard metric for model manipulability; it has to be read alongside parity win rates and an independent baseline of how the model handles cost.
Evidence [c4]
A diagnostic tool, not a retail pricing strategy
The methodological takeaway here travels much better than the specific numbers. Because each experimental condition relied on just six product pairs, running repeated trials tightened the estimates for those specific items without creating a broad sample of the market. The authors note that their confidence intervals likely understate uncertainty across different products. Furthermore, simply rephrasing the initial shopping prompt shifted the numerical estimates and occasionally scrambled the ranking of the cues, showing that the setup captures broad differences in magnitude far better than exact percentages.
It is also critical not to confuse a synthetic binary choice with a real-world consumer journey. The experiment did not track multi-step browsing, review reading, or actual human checkout behavior. A 12.9% threshold in an API test is not a green light for merchants to hike retail prices by 12.9% expecting stable sales.
For teams auditing how AI agents handle brand messaging, the practical lesson is straightforward: equal win rates do not guarantee equal weight. When two messages max out a parity test, you have likely just hit the ceiling of your ruler.
The source and evidence
A Cue's Hidden Price: Auditing Seller Influence on AI Shoppers
Evidence behind this briefing
[c1] Price-equivalent (PE) is the smallest percentage price premium at which the cued product's selection share falls to one half under the chosen presentation policy. It is conditional on the products, cue comparison, model, prompt, and decoding configuration—not a general willingness-to-pay estimate.
Section 3.1, Formal definition · Read the study
[c2] For GPT-4o on headphones, strong social proof was selected in all 144 equal-price trials and had a PE of 12.9% (95% CI [10.9,17.1]). A neutral factual sentence also approached 100% equal-price selection but was offset by approximately a 2% premium. Thus, nearly identical recommendation rates concealed an approximately sixfold difference in price tolerance.
Section 4.1, opening two paragraphs · Read the study
[c4] Across the six task-engaged text models, raw PE was not a valid standalone ranking of cue susceptibility because it combined cue response with price sensitivity. Claude Sonnet's small authority PE reflected a steep price response, whereas Nova Micro continued selecting cued products through an extended +110% sweep, leaving those PEs unidentified rather than establishing exceptionally strong persuasion.
Section 4.4, discussion following Table 4 · Read the study
[c10] The GPT-4o paraphrase check used six cues, four task phrasings, and 12 trials per pair and price point. Social-proof intensity ordering survived, but exact percentages and fine rankings did not: strong social proof and gain/loss framing exchanged rank, the neutral sentence reached 4.5%, and very strong social proof varied by roughly eight percentage points. The authors therefore support coarse magnitude bands and intensity ordering rather than phrasing-invariant numerical values.
Appendix J, Prompt-paraphrase robustness; Table 19 · Read the study
[c11] The effective product-level sample is only six pairs per condition, so percentile cluster-bootstrap intervals can understate uncertainty. GPT-4o’s headphone strong-social-proof PE varied from about 10% to beyond 26% across pairs despite a pooled 12.9% estimate. Scarcity remained below the stated 3% resolution threshold in every leave-one-pair-out check, but the authors identify the limited number of product pairs as the main design limitation.
Appendix F, Effective sample size: opening, Pair-level spread, and Leave-one-pair-out (scarcity) · Read the study