What it means
Think of robots.txt as instructions published at the entrance to your website. A crawler that follows the Robots Exclusion Protocol reads the file for the relevant website address and chooses the instructions matching its name. The file is public and is not a password, firewall, or way to hide confidential URLs.
The useful business question is which crawler roles you want to permit. Search, model training, and a tool fetching a page for a person can serve different purposes. An allowed result describes the evaluated rule for a path. A disallowed result can reflect an intentional choice, rather than something that needs fixing.
Named crawler groups matter. When a specific group applies, the default User-agent: * group does not automatically supply extra restrictions. Within the applicable rules, the most specific matching path governs the decision. A file can therefore behave differently from a quick reading of its first few lines.
Hypothetical example: a restriction that does not carry over
A fictional retailer puts Disallow: /checkout under User-agent: *. It also creates a named ExampleBot group containing only Allow: /. The default checkout restriction does not automatically apply to that named group. This illustrates rule selection, not a finding about any real crawler or website.
How BLURSOR measures it
In the report: robots.txt · Published crawler rule · Specific crawler rules
BLURSOR requests robots.txt separately from the page, classifies the response, and evaluates usable rules for listed crawler roles and the checked path. The rows retain matching instructions and distinguish allowed, disallowed, not applicable, and unavailable evidence.
A partial result preserves usable rules alongside file errors. A capture cut short cannot establish the complete policy. The precedence observation identifies named groups that omit default disallow instructions; it does not decide whether the omission was intentional. A missing or valid empty file is different from a challenge or failed request.
What to do next
- Choose the crawler purposes you want to support before changing a rule merely to improve a report.
- Ask your developer to check the complete served file, named groups, and the exact paths involved.
- Use authentication for private pages and first-party request logs when you need evidence of actual crawler access.
What it cannot tell you
- Published permission does not establish crawling, indexing, citation, or a verified vendor identity.
- Partial, missing, and not-applicable evidence must not be counted as equivalent outcomes.
Sources and further reading
The definitions below describe the underlying concepts. The measurement section describes BLURSOR’s own checks.