If AI agents cannot crawl your website, nothing else matters. Your robots.txt file is the first thing every AI crawler checks -- and most websites are accidentally blocking the AI bots that power ChatGPT, Perplexity, Claude, and Gemini.
The Complete List of AI Crawlers
| Bot Name | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | General crawling for ChatGPT |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Real-time browsing by ChatGPT |
| PerplexityBot | Perplexity | Perplexity search and answers |
| ClaudeBot | Anthropic | Claude web browsing |
| Claude-Web | Anthropic | Claude web search |
| Google-Extended | Gemini AI training and search | |
| Amazonbot | Amazon | Alexa and Amazon search |
| Bingbot | Microsoft | Copilot and Bing AI |
| cohere-ai | Cohere | Cohere AI models |
| anthropic-ai | Anthropic | Anthropic general crawling |
Recommended robots.txt Configuration
User-agent: *
Allow: /
Disallow: /api/
Disallow: /dashboard/User-agent: GPTBot Allow: /
User-agent: OAI-SearchBot Allow: /
User-agent: PerplexityBot Allow: /
User-agent: ClaudeBot Allow: /
User-agent: Google-Extended Allow: /
Sitemap: https://yoursite.com/sitemap.xml ```
Common Mistakes
- Blanket blocking: Many CMS platforms and hosting providers block all unknown bots by default.
- Forgetting OAI-SearchBot: GPTBot is not the only OpenAI crawler. OAI-SearchBot powers search.
- Blocking /api/ but not /dashboard/: Protect private routes, but allow public content.
- No sitemap reference: Including your sitemap URL helps AI crawlers discover all your pages.
How to Test
Use ConduitScore to scan your URL -- our Crawler Access analyzer checks every major AI bot against your robots.txt and reports which ones are blocked.
Check Your AI Visibility Score
See how your website performs across all 7 categories in 30 seconds.
Scan Your Website Free