AI opt-out hints

What this check looks for

Beyond robots.txt, this check scans page-level meta tags (robots and bot-specific names like gptbot, claudebot) and the X-Robots-Tag header for noai/noimageai tokens, or a bot-specific noindex/none/nofollow directive scoped to one of the AI crawlers. Any of these found warns, quoting the specific tag or header. None found passes.

Why it matters

Even when robots.txt allows an AI crawler through the front door, a page can still carry its own opt-out signal that some AI systems are beginning to respect — an emerging convention distinct from, and layered on top of, the older robots.txt and noindex mechanisms search engines have used for decades. Because this signal is newer and less standardized, it's easy to add unintentionally (a security or privacy plugin that ships an "AI protection" toggle enabled by default, for instance) without realizing it silently excludes the page from exactly the AI visibility you might otherwise want. This check exists specifically to catch that mismatch: robots.txt says one thing, but a page-level tag says another.

How to fix it

Remove the opt-out hint if AI visibility is desired for this page:

<!-- Opts this page out of AI training/use -->
<meta name="robots" content="noai, noimageai">

<!-- Fixed: no AI opt-out signal -->
<meta name="robots" content="index, follow">

If a specific page genuinely should be excluded from AI use (proprietary pricing, internal documentation), keep the hint deliberately, but verify it's intentional rather than a default you didn't choose.