Free · No signup · 10–20 seconds
Check whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can reach a public page and read its content. See the robots.txt rule and server response for each crawler, the raw HTML next to the rendered page, and the page’s indexing signals.
This tool exists because I had a problem with how major platforms tackle AI readiness checks. The common approach is "publish 10+ separate tools, try to catch search traffic for each 'checker' query".
I doubt that serves the user well. When people search for "AI Crawler Checker" what they really want to know is "Can AI see and read my page well?". Ideally, from one tool that covers the analysis from different angles: crawl access, content extraction, indexing signals, JS rendering, structured data, etc.
So I built this one — and I keep refining it. I hope you'll find it useful and have some fun checking your website against your competitors.
Alex Rostovtsev
Alex leads SEO and content operations at Elfsight and contributes to Beamtrace, an AI search visibility tracker. About Alex · alexros.tv
WebMCP enabled
This crawler checker uses WebMCP to let AI assistants run checks and read the results directly. Ask your AI to check a page or compare it with a competitor. The report stays here for you to review.
Try asking your assistant
Open the checker at blursor.ai/ai-readiness-audit, check my website, and explain the main findings.
Paste the prompt into your assistant. Include your website URL if it doesn’t already know it.
Assistants need WebMCP support for direct tool calls; assistants with browser control can also use the form.
Visit this checker in a browser with WebMCP support to discover check_page({ url }) and compare_pages({ url, competitor_url }). Both accept public HTTP or HTTPS page URLs and return structured findings, evidence and limitations alongside a visible report. Missing measurements remain unavailable. The check does not establish indexing, rankings or AI citations.
AI Crawler Checker
Usually takes 10–20 seconds.
AI crawlability FAQ
An AI Crawler Checker tests the technical signals that affect whether AI crawlers can access and read a web page. It can inspect crawler rules, server responses, raw HTML, indexing instructions, and content that depends on JavaScript.
BLURSOR reports each layer separately and shows the evidence behind its findings.
Yes. The checker is free, needs no signup and usually takes 10–20 seconds. It works with public URLs only.
You can also add a competitor’s URL to compare both pages side by side.
Paste the page’s public URL into the checker and run the test. Review whether the relevant crawlers are allowed by robots.txt, whether the server accepts their user-agent strings, and whether the page contains useful text without JavaScript.
Test the exact page you care about. A crawlable homepage does not guarantee that every page on the site is equally accessible.
Not necessarily. Googlebot and AI crawlers can have different robots.txt rules, server treatment, and rendering capabilities.
A page that is accessible to Googlebot may still block GPTBot, ClaudeBot, PerplexityBot, or another crawler. The reverse can also be true.
We read the published robots.txt rules for crawler roles associated with OpenAI, Anthropic, Perplexity, Common Crawl, Google, Apple, Meta, Amazon, ByteDance, DuckDuckGo, Mistral, Microsoft, and others.
For live delivery, we use a representative set: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot, Meta-ExternalAgent, and Amazonbot.
Google-Extended is a control token, not a standalone crawler. We evaluate other published roles against robots.txt, but do not send separate page requests for them.
Rendering capabilities vary. Some automated systems can process JavaScript, while others primarily use the HTML returned by the server.
BLURSOR reads the raw HTML and, when browser rendering is available, compares it with the finished page. A large difference can mean that some automated readers receive less content than a human visitor.
Common blockers include crawler-specific robots.txt rules, firewall or CDN bot protection, 403 and 429 responses, authentication, redirect problems, accidental noindex instructions, and content that only appears after JavaScript runs.
The checker identifies which of these signals it can observe and separates confirmed findings from estimates.
No. A passing result means BLURSOR did not find a technical access problem in the layers it tested. It does not prove that a vendor has crawled, indexed, cited, ranked, or recommended the page.
BLURSOR sends its probes from its own servers using published user-agent strings, not from OpenAI’s, Anthropic’s, Google’s, or Perplexity’s verified crawler IP ranges. A site that checks source IP addresses may treat a genuine crawler differently.
What the test covers
This tool tests the ability of an automated system to reach your page and retrieve useful content from it. A lot of factors change what an AI gets from a website: crawler-specific rules, server responses, indexing instructions, and JavaScript — we check those layers separately. The report shows which AI readiness problems you might have and the evidence behind them.
Two of these go beyond a robots.txt lookup. The raw-versus-rendered comparison shows what a crawler that does not run JavaScript receives. The competitor comparison runs both pages through the same checks, so you can see whether a gap is specific to your page.
BLURSOR reads the site’s robots.txt and evaluates the rule that applies to each supported crawler.
These rules can differ by user agent. A site might allow an AI search crawler while blocking a training crawler, or allow Googlebot while saying nothing about GPTBot or ClaudeBot. The report keeps those roles separate instead of treating every AI-related bot as the same thing.
A permissive robots.txt does not guarantee access. A server, CDN, or firewall can still refuse a request based on its user agent, IP address, rate, or other signals.
We request the page with a representative set of published crawler user-agent strings and record each response. Other roles are evaluated only against robots.txt, and the report keeps those evidence types separate.
A finished page in a browser can contain much more text than the HTML initially sent by the server.
BLURSOR extracts the readable text, headings, metadata, links, and structured data available in the raw HTML. When browser rendering is available, it compares that version with the fully rendered page so you can see what depends on JavaScript.
The crawlability test also checks signals that can limit how a page is used after it has been fetched. These include robots meta tags, X-Robots-Tag headers, noindex, AI preference signals, and canonical URLs.
Access and indexing are separate. A crawler may be able to retrieve a page that the page itself asks not to be indexed.
Add a competitor’s URL and both pages go through the same checks in one run.
The comparison lines up crawler rules, readable text, headings, structured data, indexing directives, and published discovery files for both pages, so you can see where they differ. Choose two pages with a similar purpose, such as two product pages.
Common fixes
If a crawler is disallowed, decide whether that rule is intentional. Search crawlers, user-directed fetchers, and training crawlers serve different purposes, so allowing one does not mean you must allow all of them.
If the page works in a browser but returns an error to a crawler probe, review your CDN, firewall, rate limits, and managed bot settings. These controls can override what robots.txt appears to allow.
If most of the page disappears without JavaScript, move the essential title, description, headings, and body copy into server-rendered or statically generated HTML. A crawler should not need to run your application before it can understand the page.
Review noindex, X-Robots-Tag, canonical URLs, and crawler-specific directives. These signals are useful when intentional and costly when left behind by mistake.
What comes after crawlability
Alex Rostovtsev · SEO & GEO audits
BLURSOR checks whether AI crawlers can reach your page and read its content. Here’s what you can do with the findings: