[{"data":1,"prerenderedAt":96},["ShallowReactive",2],{"tool-robots-txt-validator-en":3},{"common":4,"copy":13},{"hubLink":5,"resultTitle":6,"fix":7,"fullTitle":8,"fullText":9,"fullCta":10,"faqTitle":11,"waiting":12},"All free tools","Results for {host}","How to fix it","See how AI reads this site","Run a free full scan of {host} to get both AI readiness scores and every fix.","Scan this site","Questions","Testing {host}. This can take several minutes, so keep this page open.",{"name":14,"title":15,"description":16,"h1":17,"lede":18,"empty":19,"labels":20,"about":21,"faq":25,"robots":44},"robots.txt validator","robots.txt validator and tester: check every rule","Validate robots.txt by URL or paste it. Find syntax errors and dead rules, test URLs, and see which search, AI and SEO crawlers it blocks.","robots.txt validator and tester","Fetch a site's robots.txt or paste your own. See every error by line, test any URL, and check which crawlers get in.","",{},[22,23,24],"The validator reads robots.txt line by line, the way Google and other crawlers do. It flags lines crawlers ignore, rules that never match, rules that hide CSS and JavaScript from Google, and lines Google does not support, such as Noindex and Crawl-delay.","A missing robots.txt (HTTP 404) is fine: crawlers take it as permission to crawl everything. A server error (HTTP 5xx) is worse. Google treats it as “disallow everything” and stops crawling until the file loads again.","Content-Signal lines are valid. They state how your content may be used for search, AI answers and AI training, so the validator only checks that their values are yes or no.",[26,29,32,35,38,41],{"q":27,"a":28},"How do I test my robots.txt?","Enter your domain, or paste the file. Then add paths under Test URLs and pick a crawler. Each path shows Allowed or Blocked and the rule that decided it.",{"q":30,"a":31},"Google's robots.txt tester is gone. What can I use instead?","Google retired the tester in Search Console in 2023. Its robots.txt report only shows the files Google fetched. This page tests your rules against any path and crawler, like the old tester did.",{"q":33,"a":34},"When Allow and Disallow both match, which one wins?","The rule with the longer path wins. If both are the same length, Allow wins. So Disallow: \u002Fblog with Allow: \u002Fblog\u002Fpublic lets crawlers into \u002Fblog\u002Fpublic and keeps them out of the rest of \u002Fblog.",{"q":36,"a":37},"Does Disallow keep a page out of Google?","No. Disallow stops crawling, not indexing. Google can still list a blocked URL if other pages link to it. To keep a page out, allow crawling and add a noindex meta tag or X-Robots-Tag header.",{"q":39,"a":40},"How do I block AI crawlers?","Give each crawler its own group, for example User-agent: GPTBot followed by Disallow: \u002F. Training crawlers and AI search crawlers are separate bots, so block only the ones you mean to. Crawler coverage shows the result for each.",{"q":42,"a":43},"Is a server error on robots.txt worse than a 404?","Yes. A 404 means there are no rules, so crawlers may crawl everything. A 5xx means the file could not be read, so Google treats the whole site as disallowed. Fix server errors on robots.txt first.",{"modeLegend":45,"modeFetch":46,"modePaste":47,"domainLabel":48,"domainPlaceholder":49,"fetch":50,"fetching":51,"pasteLabel":52,"pastePlaceholder":53,"pasteHint":54,"resultTitle":55,"empty":56,"pasted":57,"lines":58,"verdict":59,"counts":63,"severity":68,"line":72,"file":73,"noIssues":74,"sitemapsTitle":67,"testTitle":75,"testLabel":76,"crawlerLabel":77,"allCrawlers":78,"allowed":79,"blocked":80,"byRule":81,"noRule":82,"coverageTitle":83,"coverageFor":84,"groups":85,"blockedCount":89,"serverError":90,"html":91,"missing":92,"unread":93,"blockedTld":94,"fullScan":95},"Input","Fetch by URL","Paste robots.txt","Domain","example.com","Validate","Fetching...","robots.txt content","User-agent: *\nDisallow: \u002Fadmin\nSitemap: https:\u002F\u002Fexample.com\u002Fsitemap.xml","Checked in your browser as you type. Nothing is sent to SagBot.","Results","Fetch a robots.txt or paste one to see its issues and which crawlers it lets in.","Pasted text","Lines: {n}",{"errors":60,"warnings":61,"valid":62},"Errors found","Valid, with warnings to review","Valid",{"errors":64,"warnings":65,"groups":66,"sitemaps":67},"Errors","Warnings","Rule groups","Sitemaps",{"error":69,"warning":70,"note":71},"Error","Warning","Note","Line {n}","Whole file","No issues found.","Test URLs","Paths or URLs, one per line","Crawler","All crawlers (*)","Allowed","Blocked","{rule} (line {line})","No rule matched","Crawler coverage","Checked against {path}",{"search":86,"ai":87,"seo":88},"Search crawlers","AI crawlers","SEO crawlers","{n} of {total} blocked","The server answered HTTP {status}. Crawlers treat this as “disallow everything” until robots.txt loads again.","robots.txt returns an HTML page, not a text file. Crawlers find no rules in it.","No robots.txt (HTTP {status}). The site sets no rules, so every crawler may crawl everything.","Couldn't read robots.txt: {reason}","SagBot does not check domains on this TLD. You can still paste the file.","Run a full scan of {host}",1794386918817]