Checks
AI assistant crawler access
Checks robots.txt for rules that block AI assistant crawlers like GPTBot, ClaudeBot, and PerplexityBot, which decide whether your site can ever appear in AI-generated answers.
What this check looks for
This check parses robots.txt into its user-agent groups and evaluates access for the assistant-purpose AI crawlers: GPTBot (OpenAI), ClaudeBot and Claude-Web (Anthropic), and PerplexityBot (Perplexity). If robots.txt is missing or empty, every crawler is allowed by default and the check passes. If all of these crawlers are blocked, it fails outright, since the site becomes invisible to ChatGPT, Claude, and Perplexity answers. If some are blocked or only partially restricted, it warns, naming which ones. If all are allowed, it passes.
Why it matters
Unlike traditional search crawlers, these bots don't build a ranked index for a results page — they fetch content live (or near-live) to ground an assistant's answer and decide whether to cite you as a source. A Disallow: / rule aimed at a legacy crawler, or a blanket bot-blocking rule copied from a security template, frequently ends up blocking these assistant crawlers by accident, since they're a newer category most robots.txt files were never written with in mind. If GPTBot, ClaudeBot, or PerplexityBot can't fetch your pages, your site simply cannot be cited in ChatGPT, Claude, or Perplexity answers — no amount of on-page optimization elsewhere in this report compensates for that if the crawler is blocked at the door.
How to fix it
Remove or narrow any Disallow rules covering these user agents:
# Blocks all AI assistant crawlers — remove for AI visibility
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# Fixed: allow them explicitly
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /See how your site scores on this check