What it means
A site owner may want different treatment for different uses of the same content. Content Signals provide a way to write those preferences into the published robots.txt response. The main named purposes distinguish conventional search results, content supplied to an AI model for an answer, and model training or fine-tuning.
Those purposes are not interchangeable. In Cloudflare's published policy, search covers an index and results with links or short excerpts; it does not include generated search summaries. AI input covers uses such as grounding an answer with retrieved material. AI training refers to training or fine-tuning models.
A declaration and a delivered response answer different questions. A server can publish a preference and still send the page to a request. Some tools implement these preferences, but this checker does not establish which outside systems enforce them. Decide the business policy first, then check what your site actually publishes.
Hypothetical example: separating search from training
A fictional publisher chooses Content-Signal: search=yes, ai-input=yes, ai-train=no. The line states different preferences for three uses. It does not prove that a model trained on the publisher's pages, that an assistant can retrieve them, or that a server refused any request.
| Declared signal | Intended distinction |
|---|---|
| search=yes | Permit the stated search use |
| ai-input=yes | Permit the stated AI-input use |
| ai-train=no | Decline the stated training use |
How BLURSOR measures it
In the report: Declared uses of content
BLURSOR extracts Content-Signal lines from the retrieved robots.txt text and records their comma-separated key/value declarations. The comparison view combines duplicate declaration text and retains a partial-file qualification when applicable. Ordinary comments explaining the policy are not themselves a declaration.
This parser records the text; it does not fully validate every value, resolve all possible declaration scopes, or test enforcement. New extensions can therefore appear in the report without their complete meaning being evaluated. If robots.txt was unavailable or cut short, the available evidence must retain that limit.
What to do next
- Agree which content uses the business intends to support before adding a declaration.
- Inspect the served robots.txt response, including any material added by the hosting or CDN provider.
- Review usage declarations alongside crawler-specific path rules and request logs when investigating a mismatch.
What it cannot tell you
- No declaration is not a measured refusal, and a yes or no value is not proof of actual downstream use.
- This measurement does not establish complete specification compliance or a service's enforcement behavior.
Sources and further reading
The definitions below describe the underlying concepts. The measurement section describes BLURSOR’s own checks.