Serve a robots.txt that parses
Every crawler and AI agent asks for /robots.txt before anything else. A missing file is fine, but one that returns HTML from your app shell, or a 5xx, leaves bots guessing what they may fetch.
Return plain text with a 200, one User-agent group per rule set, and a Sitemap: line.
SagBot reads your /robots.txt: the status code, the content type and whether its rules parse.
Let AI search and assistant bots in
Blocking GPTBot to keep your content out of training is a policy choice. Blocking OAI-SearchBot, PerplexityBot or ChatGPT-User is usually an accident, and it means your site can’t be quoted or linked in their answers.
Check that a broad Disallow: / for * doesn’t catch the search and user agents you meant to allow.
SagBot reads the rules in your /robots.txt for each AI search and user agent it knows.
Point bots at a sitemap
A sitemap lists every page so crawlers don’t have to discover them link by link. Declare it in robots.txt so bots find it without guessing the path.
SagBot reads your sitemap (the one robots.txt declares, or /sitemap.xml) and how many URLs it lists.
Publish an llms.txt
/llms.txt is a short Markdown file: your site’s name, a one-paragraph summary, then links to the pages that matter, each with a line of context. It gives a model the map a human would get from your navigation.
SagBot reads your /llms.txt: whether it exists and is text rather than an HTML page.
Answer Accept: text/markdown with Markdown
Agents that fetch your pages pay per token. When a request says Accept: text/markdown, return the same page as Markdown and add Vary: Accept so caches keep the two versions apart.
SagBot reads your site’s response to a request with Accept: text/markdown: its content type and Vary header.
Send Link headers
A Link response header can point agents at your llms.txt, API catalog or Markdown version before they parse a byte of HTML, for example Link: </llms.txt>; rel="describedby"; type="text/markdown".
SagBot reads the Link header on your site’s response.
If you run bots, publish a key directory
Web Bot Auth lets sites verify that a request really comes from your bot: you sign requests and publish your public keys at /.well-known/http-message-signatures-directory. Skip it if you don’t operate a crawler or agent; SagBot shows it as advice, not a score.
SagBot reads /.well-known/http-message-signatures-directory on your site.
See which of these your site passes
Each failing check comes with a fix written for a coding agent: paste it into Claude Code, Cursor or Copilot.