Surpasso
All free tools
AI search access

AI Crawler Access Checker

See whether search and training crawlers from OpenAI, Anthropic, Perplexity, and Google can access a page.

Free to use No sign-inSee the rule behind every result
Want Surpasso to keep doing this?Surpasso monitors crawler access rules and flags new blocks.
Track AI visibility

Check one website

Enter a domain or a specific page to test its published crawler rules.

01
Only public HTTP / HTTPS pages are read. Site content is not stored.
No Search Console connection is required; results come from public responses.

Ready to check

Results come from the site’s current public response. Anything that cannot be confirmed is marked Unknown.

About this tool

This check reads the site’s published robots.txt rules for named AI search and training crawlers. It shows the matching rule instead of turning a nuanced policy into a single unexplained score.

What you get

A clear, usable result

  • Whether each supported crawler has its own user-agent group or inherits the wildcard group.
  • The most specific Allow or Disallow path that applies to the page you entered.
  • Whether robots.txt was missing, readable, or unavailable, so uncertainty is not presented as permission.
How to read the result

Allowed

The published rules do not block that crawler on the tested path. This does not guarantee the crawler will visit, index, cite, or train on the page.

Blocked

A matching Disallow rule is more specific than any competing Allow rule. Review the exact group and path before changing it.

Unknown

The tool could not confirm a policy, usually because robots.txt could not be fetched. Retry later and inspect CDN or firewall logs before drawing a conclusion.

Limitations and sources
  • Robots.txt is a crawler directive, not an authentication or security control.
  • User-triggered fetch agents can follow different policies from autonomous search or training crawlers.
  • This check does not prove that a CDN, WAF, bot manager, login wall, or geographic rule permits the request.
OpenAI crawlers Anthropic web crawling Perplexity bots Google common crawlers
Questions about this tool

Can I allow AI search but block model training?

Often yes. Providers publish separate user agents for different purposes. Use provider-specific groups, keep a documented policy, and recheck after deployment.

Does blocking Google-Extended remove pages from Google Search?

Google documents Google-Extended as a control for certain generative AI uses, not a control for Google Search inclusion or ranking.