Model-training crawler policy

What this check looks for

This check evaluates robots.txt access for training-purpose crawlers: Google-Extended (Google), CCBot (Common Crawl), Bytespider (ByteDance), and Applebot-Extended (Apple) — bots that feed future model training rather than live assistant answers. If robots.txt is missing or empty, access is allowed by default and it passes. If every training crawler is blocked, it warns rather than fails, explicitly noting this can be a legitimate content-protection choice, just one with a tradeoff: future AI models won't learn about your site. Any other configuration passes, noting which crawlers (if any) are blocked.

Why it matters

Training crawlers are fundamentally different from assistant crawlers even though they're often confused with each other: blocking Google-Extended or CCBot doesn't remove your site from ChatGPT, Claude, or Perplexity's live answers — that's controlled by the separate assistant-access check — it only opts your content out of being used to train future model versions. Some publishers deliberately block training crawlers to protect content from being absorbed into a model's weights while still wanting to be cited live by an assistant reading the page in real time; others want maximum future visibility and allow both. This check surfaces the current policy so you can confirm it matches what you actually intend rather than an accidental default.

How to fix it

Decide your training-data stance deliberately and configure it explicitly:

# Block only training crawlers, while assistant crawlers remain allowed elsewhere
User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Or explicitly allow training crawlers if future AI visibility matters more
User-agent: Google-Extended
Allow: /