robots.txt validator and tester
Fetch a site's robots.txt or paste your own. See every error by line, test any URL, and check which crawlers get in.
Results
Fetch a robots.txt or paste one to see its issues and which crawlers it lets in.
The validator reads robots.txt line by line, the way Google and other crawlers do. It flags lines crawlers ignore, rules that never match, rules that hide CSS and JavaScript from Google, and lines Google does not support, such as Noindex and Crawl-delay.
A missing robots.txt (HTTP 404) is fine: crawlers take it as permission to crawl everything. A server error (HTTP 5xx) is worse. Google treats it as “disallow everything” and stops crawling until the file loads again.
Content-Signal lines are valid. They state how your content may be used for search, AI answers and AI training, so the validator only checks that their values are yes or no.
Questions
How do I test my robots.txt?
Enter your domain, or paste the file. Then add paths under Test URLs and pick a crawler. Each path shows Allowed or Blocked and the rule that decided it.
Google's robots.txt tester is gone. What can I use instead?
Google retired the tester in Search Console in 2023. Its robots.txt report only shows the files Google fetched. This page tests your rules against any path and crawler, like the old tester did.
When Allow and Disallow both match, which one wins?
The rule with the longer path wins. If both are the same length, Allow wins. So Disallow: /blog with Allow: /blog/public lets crawlers into /blog/public and keeps them out of the rest of /blog.
Does Disallow keep a page out of Google?
No. Disallow stops crawling, not indexing. Google can still list a blocked URL if other pages link to it. To keep a page out, allow crawling and add a noindex meta tag or X-Robots-Tag header.
How do I block AI crawlers?
Give each crawler its own group, for example User-agent: GPTBot followed by Disallow: /. Training crawlers and AI search crawlers are separate bots, so block only the ones you mean to. Crawler coverage shows the result for each.
Is a server error on robots.txt worse than a 404?
Yes. A 404 means there are no rules, so crawlers may crawl everything. A 5xx means the file could not be read, so Google treats the whole site as disallowed. Fix server errors on robots.txt first.