AI search mode cuts publisher referrals without making search feel better

AI summaries are often sold as a cleaner search experience: fewer blue links, faster answers, more convenience. This paper tests that claim in a live Google Search setting, and the result is more complicated. When users were forced into AI Mode, they clicked out to publishers less often, spent longer in each search session, and rated Google as less trustworthy and less useful.

That matters because it changes what teams should optimize for. If you only look at in-product engagement, AI Mode can look like a win. But if you care about publisher traffic, user trust, or whether people stay with Google search at all, the picture looks worse.

The study’s anchor is simple: AI Mode reduced publisher click-through by 18.8 percentage points.

---

What the experiment changed in Google Search

This was a preregistered field experiment, not a lab task or a synthetic benchmark. The researchers recruited users from Prolific and a Northeastern work-study program, gave them three days in baseline Google Search, and then randomly assigned them to one of three conditions for seven days.

The three conditions were straightforward. In Current Search, Google behaved normally. In No AI Search, the experiment hid AI Overviews and redirected AI Mode searches back to general search. In AI Mode Search, all searches were routed into AI Mode.

That setup matters because it isolates the product change rather than the user’s preference for trying an AI feature. Baseline AI Mode usage was only 0.6% of searches before the intervention, so this was a forced adoption test, not a gradual rollout study.

The behavioral analysis covers N=1,100 users, with survey outcomes available for N=956. The authors estimate treatment effects with OLS, controlling for baseline outcome values and recruitment channel, and use randomization inference for p-values.

---

AI Mode kept people in search longer, but sent fewer of them to publishers

AI Mode made sessions longer and referrals weaker at the same time. The clearest behavioral effect is the drop in click-through to publishers: assignment to AI Mode Search reduced click-through rate by 18.8 percentage points, with a 95% confidence interval from -22.2 to -15.3. When AI Overviews and AI Mode were removed, click-through rose by 8.8 percentage points.

That is the key tradeoff for publisher-facing traffic. The paper also shows the effect isn’t just abstract “engagement” moving around. AI Mode reduced clicks to specific domains too, including news sites, Reddit, and Wikipedia categories reported in the results.

At the same time, AI Mode increased minutes per session by 0.43 minutes. So users spent more time inside Google, but a lot of that time did not translate into outbound referral traffic.

For search product teams, this is the part that matters operationally: AI Mode can improve time-on-product while simultaneously weakening the distribution channel that publishers depend on.

---

Trust and usefulness moved in the wrong direction

AI Mode did not just shift clicks. It also made the search experience feel worse to users on the measures the authors collected.

Trust in information on Google fell by 0.34 points on a 7-point scale. The same treatment also reduced perceived usefulness, satisfaction, and agency. In other words, people were not merely clicking less; they were also less confident that the system was giving them something reliable or useful.

The personalization story went the same way. Instead of making results feel more relevant, AI Mode decreased perceived personalization and relevance by 0.42 standard deviations. That cuts against the common assumption that generative search will feel more tailored and therefore more satisfying.

The paper also looked at a news-specific head-to-head comparison and did not find evidence that AI Mode was preferred there, or that it increased trust in that context. So the trust hit was not limited to a generic survey item; it showed up in a more concrete comparison as well.

---

Users did not just stay in Google less — they started looking elsewhere

One of the most important second-order effects is substitution. AI Mode did not simply compress user activity into Google Search. It pushed some of it outward.

The fraction of users searching on a competitor engine increased by 11.2 percentage points. The study also finds that intention to switch to Bing rose by 1.25 points on a 7-point scale. That is a stronger signal than a narrow in-product preference shift: some users seem to respond to AI Mode by changing where they search next.

There is another wrinkle here. Search sessions per day fell by 0.92, even as minutes per session rose. So the product may encourage fewer but longer search sessions, while also nudging users toward other engines when they do search.

That combination is hard to ignore if you are evaluating AI search as a retention strategy. More time per session is not the same thing as more loyalty.

---

What to do about it

If you build search products, this paper is a warning against narrow success metrics. AI Mode can look good on session duration or on-page engagement and still damage the things that matter downstream: publisher referrals, trust, usefulness, and competitive retention.

That means evaluation needs to be broader than “did users stay longer?” Teams should measure outbound clicks, trust, perceived relevance, and cross-engine substitution before treating AI summaries as an unqualified improvement. If a new search surface keeps users inside the product but makes them less confident and more willing to switch engines, the business story is mixed even if the interface looks cleaner.

For publishers, the practical takeaway is also blunt. Traffic from general search is not just being redistributed within Google; in this experiment, it is being pulled away from external sites altogether. That makes dependence on search referrals riskier if AI Mode or AI Overviews become the default experience.

The caveat is that this was a forced 7-day exposure with a sample recruited from Prolific and a university work-study program, so the exact magnitudes may shift in a broader rollout. But the direction is hard to miss: more AI in search did not make users trust search more.