Did Your Content Change Grow ChatGPT Referrals—or Did ChatGPT Grow?

Your ChatGPT referrals doubled after a content project. That is good news about the traffic count. It is not yet an estimate of what the project caused.

More people may be using ChatGPT. The product may be sending more visitors to websites. Your subject may have become more popular. If untreated pages also gained referrals, some of the growth may have arrived without the intervention.

A Glasp case study investigates this problem using first-party referral data. The authors compared a changed group of pages with the untreated remainder of the same domain. That is more informative than a before-and-after multiple, but the comparison still has important weaknesses.

What the comparison holds in common

Beginning in January 2026, Glasp applied a bundle of changes to its YouTube question-and-answer pages. The study compared ChatGPT referral traffic to those pages with traffic to other pages on glasp.co.

The groups shared a domain and much of the same analytics environment. They also experienced the broader changes affecting ChatGPT at the same time. Comparing their movement can help separate a sitewide or platform-wide increase from something particular to the changed pages.

The outcome was actual measured sessions, using a definition based on session sources containing “chatgpt.” That is different from a citation visibility score. It measures visits classified by that referral rule, not every person whose decision may have been influenced by an AI answer.

The intervention was not randomized. The changed pages and comparison pages had different content and intent, and they were not matched into otherwise equivalent groups. Sharing a domain does not make them interchangeable.

The adjusted estimate was still uncertain

The authors used weekly data across 47 weeks: 26 before the intervention and 21 afterward. Their models estimated an intervention-aligned level effect around 1.8–2.3 times, depending on the specification; one reported level estimate was 1.82 times.

That is an adjusted observational estimate, not a verified multiplier available from repeating the content bundle.

Two results are especially important when reading it. First, the treated-to-control relationship already had a significant trend before the intervention. The groups were not simply moving together until the change. Modeling that trend helps, but it makes the result depend on whether the model describes what would otherwise have happened.

Second, the placebo-in-time test returned p = 0.16. That test asks how unusual the observed change looks when compared with changes at alternative, pretend intervention dates. The result did not meet the conventional 0.05 threshold. It is not a 16% probability that the intervention had no effect, but it does weaken a confident causal story about the timing.

The paper therefore offers suggestive evidence, with a useful attempt to account for background growth. Calling the effect established would go further than these checks support.

Controls can introduce their own problems

The untreated group might also benefit if a change affects the domain more broadly. That kind of spillover could reduce the measured difference between groups. It is a possible bias, not proof that the reported estimate must be conservative overall.

Other differences can push the comparison in other directions. The pages may serve different demand, have different existing trends or change in ways unrelated to the recorded intervention. A mid-March bot-filtering change also altered session composition, creating a measurement issue within the study period.

And because the intervention bundled several tactics, the estimate cannot identify which component helped. A result for the package does not validate every item in the package.

What to ask about a referral-growth claim

Start with the raw counts and the dates. A large multiple from a tiny starting count can look more decisive than it is. Ask whether the tracking definition or filtering changed during the period.

Then inspect a contemporaneous comparison. Did untreated pages grow too? Did the groups have similar trends beforehand? Why were those pages chosen as controls, and could the intervention have affected them as well?

Keep the outcome precise. If the project promised visits, examine visits; if it promised qualified enquiries, examine those separately. A citation increase can be relevant without being evidence of either.

A control group is a way to improve the question, not a certificate of causality. The useful report shows how the estimate changes when its assumptions change—and leaves the uncertainty visible when the comparison cannot settle it.