Skip to content
Science-backed data on why AI says what it says
Reading & structure

Indexing instructions

Indexing instructions are page metadata or response headers that tell supporting search engines how a page may appear in search. A noindex instruction requests exclusion from the index. It differs from robots.txt, which controls crawling rather than search inclusion.

Checked against BLURSOR’s measurements ·

What it means

A business may deliberately keep a draft, internal search page, or temporary campaign page out of search results. A noindex instruction expresses that choice to supporting search engines. Public accessibility and search inclusion are different: a page can load normally while requesting not to be indexed.

These instructions can appear in HTML robots metadata or in an X-Robots-Tag response header supplied by the server. A crawler generally needs access to the response to discover them. Disallowing the URL in robots.txt can prevent that access, so the two controls should not be treated as interchangeable.

Illustrative example

Hypothetical example: a draft setting survives publication

Imagine a fictional business preparing a new public service page. During review, the editor enables a setting that produces noindex. The page is later linked from the main menu, but that setting remains. Customers can open the page while its response still asks supporting search engines to exclude it.

The useful next step is to confirm the owner's intent and find the setting or header responsible. Removing an intentional instruction from a private draft would be the wrong fix. Even after an unwanted instruction is removed, the checker cannot promise when a search service will revisit or index the page.

How BLURSOR measures it

In the report: Search indexing instructions

BLURSOR records recognized robots and supported bot-specific meta names in the selected raw response, together with its X-Robots-Tag header. It flags noindex or none found in those records. Inspect the recorded directive and its name: an instruction may be addressed to a particular crawler rather than every service.

The comparison distinguishes noindex detected from no noindex detected when usable evidence exists. The latter only describes the inspected response. The report may separately note noai or noimageai preferences; those are not the same instruction as noindex, and support should not be assumed across services.

What to do next

  1. Decide whether this exact page should be discoverable in search before changing any directive.
  2. If exclusion is accidental, ask the editor or developer to correct the responsible page setting or response header, then inspect the published response.
  3. For actual search inclusion, use the relevant search service's own inspection tools and keep crawling permissions consistent with your intent.

What it cannot tell you

  • No noindex detected is not an indexing confirmation, and a successful page request does not override an exclusion instruction.
  • This is a check of captured instructions, not an access-control mechanism, a full interpretation by every crawler, or a test of AI training use.

Sources and further reading

The definitions below describe the underlying concepts. The measurement section describes BLURSOR’s own checks.

Related explanations