Which AI bots should you let into your website? A checklist

What are the three kinds of AI bot?

The three kinds are search bots, fetch bots and training bots, and each AI company runs its own version of each. The company's own documentation is the only reliable list, because names change and third-party lists go out of date.

Which bots decide whether you appear in AI answers?

The search bots and the fetch bots decide it, because they are the ones that read your pages for an answer. OpenAI says that a site which opts out of OAI-SearchBot will not appear in ChatGPT search answers, though it can still show up as a navigational link. Anthropic says that blocking Claude-SearchBot stops a site's content being indexed for search and may reduce its visibility, and that blocking Claude-User may reduce visibility in user-directed web search.

Perplexity describes PerplexityBot as the bot that surfaces and links websites in its search results, and recommends allowing it in robots.txt so your site can appear.

Google works differently. Google says its AI features need no special markup or files: a page must be indexed and able to show a snippet in Google Search. Googlebot is the bot that crawls for Search.

What about the training bots?

Training bots are your choice, and blocking them is separate from being found. OpenAI says GPTBot and OAI-SearchBot can be controlled independently, so you can disallow GPTBot and still allow the search bot.

Google-Extended needs care. Google's crawler page says it is a robots.txt token, not a separate crawler, and that it controls whether crawled content may be used to train future Gemini models and also whether it may be used for grounding in the Gemini apps. It says Google-Extended does not affect inclusion in Google Search. So blocking it is not only a training decision.

Why is robots.txt not enough?

robots.txt is a request, and two other things can override it. The first is the bot itself: OpenAI says robots.txt rules may not apply to ChatGPT-User because users start those visits, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. The second is your own security setup. Services such as Cloudflare's AI Crawl Control let a site owner set allow or block rules for individual crawlers, so a bot that robots.txt allows can still be blocked by a rule set there.

The only way to know a bot really gets through is to visit as that bot and see what comes back. Two cautions from the companies. Anthropic says blocking its IP addresses may not work as an opt-out, because it prevents its bots from reading robots.txt. Perplexity says that if you use a firewall, you should match its bots' user agent together with their published IP ranges.

What does a sensible robots.txt look like?

A sensible robots.txt names each bot you care about, allows the search bots, and disallows only the training bots you want to keep out. This example is for illustration only, so check each company's page for the current names before you copy it.

# Let AI search bots in
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Keep two training bots out (your choice)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Anthropic says a rule must be added on each subdomain separately. OpenAI says changes to robots.txt can take about 24 hours to take effect for search.

What to do this week

1. Open your robots.txt file and list every bot it names. Check each name against the company's own page.
2. Decide which of the three groups you want to allow, and write the rules down in one sentence each.
3. Look at your CDN and firewall settings for AI bots, and confirm they match your robots.txt.
4. Visit your homepage as each search and fetch bot, or run the free AI bot checker, and note any bot that gets a block page.
5. Put a date in your calendar to repeat the check in three months.