Robots.txt tester
Validate crawl rules before you ship them.
In short
A robots.txt tester fetches the file at the root of a domain, parses each user-agent group, and reports which paths are allowed or disallowed for common crawlers along with syntax errors that cause rules to be ignored.
What the Robots.txt tester checks
- File is reachable and returns text/plain with a 200 status.
- User-agent groups parse and do not conflict.
- Sitemap directives are present and absolute.
- Critical paths are not accidentally disallowed.
How to read your results
- Disallow does not remove a URL from the index — it stops crawling. Use noindex or removal for de-indexing.
- A blank Disallow line allows everything for that group, which is a common accident when commenting rules out.
Limitations of this tool
- Only well-behaved crawlers honour robots.txt; it is not an access control mechanism.
Frequently asked questions
- Should I block query parameters in robots.txt?
- Usually no. Canonical tags and parameter handling are safer, because blocking prevents crawlers from ever seeing the canonical signal.
Category: SEO · Last updated 2026-08-08