Surpasso
All free tools
Crawl control

Robots.txt Checker & Fixer

Validate crawl rules, find accidental blocks, discover sitemaps, and prepare a safer draft.

Free to use No sign-inFind risky rules before crawlers do
Want Surpasso to keep doing this?Surpasso monitors robots.txt and flags rules that block important pages.
Monitor this with Surpasso

Check a robots.txt file

Use a domain, or add a page path to test whether that path is allowed.

01
Only public HTTP / HTTPS pages are read. Site content is not stored.
No Search Console connection is required; results come from public responses.

Ready to check

Results come from the site’s current public response. Anything that cannot be confirmed is marked Unknown.

About this tool

Use this tool to inspect the live robots.txt file, test a specific page path, find malformed directives, discover sitemap declarations, and prepare a conservative draft without silently deleting existing rules.

What you get

A clear, usable result

  • HTTP availability and whether the response is a usable text robots file.
  • User-agent groups, Allow and Disallow directives, suspicious formatting, and site-wide blocks.
  • The rule that applies to the entered path, plus sitemap URLs published in the file.
How to read the result

Path allowed

The tested crawler is not disallowed on that path. It may still be unable to fetch the page for HTTP, network, authentication, or rendering reasons.

Path blocked

The displayed rule blocks the tested path. Confirm whether the block is intentional before editing a production file.

Draft available

The draft preserves readable existing rules and adds only safe missing basics. It is a review aid, not an automatic replacement.

Limitations and sources
  • Robots rules can prevent crawling but cannot keep a publicly linked URL secret.
  • Different crawlers may interpret unsupported directives differently; this tool focuses on standard rules.
  • An unavailable robots.txt is reported as unknown; the tool will not invent a replacement from an unreadable file.
Google robots.txt guide Robots Exclusion Protocol
Questions about this tool

Should I block admin and account pages?

You may reduce unnecessary crawling, but sensitive pages must still require real authentication. A robots rule is publicly visible and provides no access control.

Does robots.txt support wildcards?

Major search engines support common wildcard patterns, but standards and crawler behavior vary. Prefer the simplest path rule that expresses the intended policy.