· ReadyStack
Worked example
Real numbers from this tool, line by line.
Six lines in this robots.txt do nothing the author intended - for SEO consultants and publishers auditing a client site.
robots.txt
1 ERROR No crawler answers to this name - the real token has no hyphen
4 ERROR Retired token. Claude-Web and anthropic-ai were replaced by ClaudeBot in 2024
5 ERROR Disallow: * is not a valid path value
7 WARN Google-Extended is a control token, not a crawler
10 ERROR Google stopped honouring noindex in robots.txt on 1 September 2019
11 ERROR Sitemap must be an absolute URL including scheme and host.Those findings come from an eleven-line file whose every line reads like an instruction: User-agent: GPT-Bot, Disallow: *, Noindex: /private, Sitemap: /sitemap.xml. Not one of them is.
The names are wrong
There is no crawler called GPT-Bot. The real token is GPTBot, with no hyphen. Claude-Web was real once and was retired in 2024, when Anthropic consolidated on ClaudeBot. robots.txt ignores unknown user-agent tokens silently: no warning, no error, no log line. The file looks correct and blocks nothing.
Assistants produce these names because the crawler names changed after their training data was collected - which is why you cannot ask a chatbot to write the file that governs chatbots.
Training and citation are two different taps
Blocking a training crawler and blocking a citation crawler look identical in the file, and do opposite things to your business:
User-agent: GPTBot # training - your text feeds a future model
User-agent: OAI-SearchBot # citation - your link appears inside ChatGPT answers
User-agent: ChatGPT-User # a reader asked ChatGPT to open your pageGPTBot trains. OAI-SearchBot is how ChatGPT finds and cites you. Anthropic splits the same way: ClaudeBot trains, Claude-SearchBot cites, Claude-User fetches when a person asks.
So someone who pastes "block the AI bots" and disallows the whole family keeps the training exposure they were worried about on the crawlers they missed, and deletes themselves from the surfaces that were sending readers.
Google-Extended does not do what its name suggests
Google-Extended is not a crawler but a control token - you will never see it as a request in your logs. Google documents that it only opts content out of Gemini model training and grounding, that it does not affect inclusion in Google Search, and that it is not a ranking signal. AI Overviews are served from Googlebot. So blocking it does nothing about AI Overviews; the only way out is to block Googlebot, which removes you from Google Search entirely. Applebot-Extended is the same shape.
The wildcard group is not a base layer
User-agent: * is not inherited. A named group replaces it rather than merging with it, so adding a group for GPTBot makes that crawler ignore every rule under the wildcard. And Crawl-delay is not a fix: Googlebot ignores it, though Bing and several AI crawlers honour it.
Check it instead of trusting it
I built the check into the editor: AI Crawler Rules for robots.txt, a VS Code extension that reads the file line by line and says what each line actually controls. It carries nineteen rules, six of them hard errors, with every token taken from the vendors' own crawler documentation. There is a free web version too.
Checking the open file is free and complete: every rule runs, every finding is shown, nothing is hidden or watermarked. A licence is $29 once and adds a different job rather than more of the same one - scanning every robots.txt in a workspace, exporting the report for a client, failing CI when an error appears, and re-checking on save. A freelance technical SEO consultant bills roughly $100-150 an hour, and an AI-crawler robots.txt review is a one-to-two hour job per site.
Open your own robots.txt and search it for a hyphen in GPT-Bot, and for the word Claude-Web.
15 seconds — what it actually does
Questions people ask
What does the AI crawler robots.txt audit actually do?
It reads a robots.txt line by line and reports what each line really controls. It flags tokens that no crawler answers to, such as GPT-Bot or the retired Claude-Web, and separates training crawlers like GPTBot from citation crawlers like OAI-SearchBot, so you can see which lines cost you visibility in AI answers.
Who is this for?
SEO consultants, publishers and web teams who keep robots.txt in the repository. It suits anyone who wants a site trained on less but cited more, and agencies managing many sites who need one pass over every robots.txt they are responsible for rather than opening each file by hand.
Why not just ask a chatbot to write my robots.txt?
Assistants confidently produce tokens that do nothing, because the crawler names changed after their training data was collected. Claude-Web and anthropic-ai were retired in 2024, and GPT-Bot with a hyphen is not a real token. robots.txt ignores unknown names silently, so a wrong file looks correct and blocks nothing.
What is free and what needs a licence?
Checking the open file is free and complete: every rule runs, every finding is shown, and nothing is hidden or watermarked. A licence adds a different job, not more of the same one: scanning every robots.txt in a workspace, exporting a report for a client, failing CI on errors, adding your own rules, and re-checking on save.
What would this cost me otherwise?
A freelance technical SEO consultant bills roughly $100 to $150 an hour, and reviewing AI crawler access is a one-to-two hour job per site. The licence is $29 once,, and the free tier already finishes the job for a single file.
Why not just ask ChatGPT or another AI chat?
A general AI chat answers from training data with a cutoff date, cannot read your repository and names no rule version. robots.txt AI Crawler Audit - GPTBot, ClaudeBot, Google-Exte checks the file you open against 19 rules from a rule set dated 2026-09-20, and points at the exact line. For a filing, an audit or a client you need that dated result on your own files.
Ask about this tool
One question, answered by the person who built it. Your email only if you want the answer sent.