Reworded Buying Questions Produced Different AI Brand Lists

“Which CRM would suit a small consultancy?” and “Recommend a CRM for a small consulting firm” appear to ask the same buying question. Adding “with data hosted in the EU” changes the requirement.

An AI visibility audit should not treat those two kinds of variation as equivalent. The first tests whether wording changes the answer while intent stays similar. The second asks whether the assistant responds to a different customer constraint.

A study of commercial recommendations examined repeated prompts and paraphrases across approximately 12,000 runs on tested OpenAI and Anthropic model configurations. It found substantial variation even before the wording changed, with further differences across rephrased questions.

The result supports a practical question about a report: how much of the buying intent does its prompt set cover? It does not show that website quality is irrelevant or that every change in a fixed-prompt score is noise.

Measure ordinary variation first

The researchers repeated identical prompts within the same day. Recommendation-set similarity, measured with Jaccard overlap, ranged from 0.50 to 0.61 across the tested configurations.

Two AI judges extracted brands from each answer. A recommendation counted only when both judges identified it as recommended. The figures therefore describe sets produced by that conservative extraction method, which can miss recommendations.

Jaccard divides the number of shared items by the number of distinct items across both sets. It is not the percentage of each answer that agrees. For a hypothetical example, two lists with four brands each and three brands in common have five distinct brands between them. Their Jaccard score is three divided by five, or 0.60.

That metric describes set overlap, not whether the recommendation is correct, well-supported or suitable for a buyer. Two different lists can both contain reasonable options. A stable list can repeatedly recommend an unsuitable one.

The unchanged-prompt result supplies a baseline for this study's comparisons. It is not a universal stability ceiling for AI products, and a single-day sample does not establish how recommendations behave over months.

Preserve intent before testing wording

The study distinguished cosmetic paraphrases, such as synonym substitutions and structural rewrites, from changes that add or shift constraints. Its reported overlap was about 0.288 for the cosmetic comparisons and 0.135 for constraint-adding comparisons.

The lower cosmetic overlap shows that similar questions can produce materially different recommendation sets in this setup. The constraint-adding result needs a different interpretation: a changed region or a more specific requirement may legitimately call for different suppliers.

The study also compared providers. Its point estimates suggested that some wording changes produced more divergence than switching between the tested providers. That is an interesting comparison within the experiment, not a general law that wording always matters more than model choice.

Content quality was not experimentally varied. The paper cannot establish that prompt wording matters more than the quality of a business's information, even if the wording differences were large.

Keep a stable panel and test its coverage

Fixed prompts have a valid use: they let you compare answers to the same questions over time. If you change the wording every week, you introduce another reason for the score to move.

But a fixed panel only measures that panel. To understand a wider buying intent, inspect plausible alternative phrasings as a separate layer of the evaluation. Keep the equivalent rewordings separate from questions with genuinely different requirements.

A useful prompt record includes the intended customer need, exact wording, constraints and reason for inclusion. Where possible, draw language from actual customer research or enquiries rather than generating a large list of near-duplicates and calling it representative.

There is no magic number of paraphrases established here. The right coverage depends on how varied the buying questions are and what decision the report supports. More prompts can still repeat the same narrow interpretation.

Read changes against both kinds of variation

When a brand's score moves, first check whether the product, mode, query panel or scoring method changed. Then compare the movement with ordinary rerun variation and with the spread across equivalent phrasings.

Save complete answers and their sources, not only a count of brand names. A supplier can appear consistently while the answer misstates what it sells. It can also disappear from one wording while remaining visible for the customer requirement that matters most.

The useful distinction is between stability for a particular question and coverage of a broader intent. A report can provide the first without claiming the second. Asking it to show both makes its limits—and its value—much easier to judge.