robots.txt blocking

What this check looks for

For every crawled page, this check evaluates the site's fetched robots.txt against that page's URL. Any page robots.txt explicitly disallows fails, quoting the relevant robots.txt rule as evidence. If no robots.txt was found at all, every page passes by default (nothing restricts the crawl). If robots.txt couldn't be evaluated for this scan, the result reports "na".

Why it matters

A Disallow rule in robots.txt stops compliant crawlers from fetching the page in the first place — unlike a noindex meta tag, which still requires the crawler to fetch the page to read the instruction, robots.txt blocks the fetch itself, so the crawler never even sees whether the page has good content. This is commonly a leftover from a site migration (an old Disallow: / blanket rule never removed) or an overly broad pattern meant to block one section that accidentally catches pages meant to be public. Blocking your own important pages this way removes them equally from Google, Bing, and any AI assistant crawler that honors robots.txt, since they all read the same file.

How to fix it

Narrow the Disallow rule so it only blocks what you intend, or remove it if it's stale:

# Blocks the entire site — likely a leftover from staging
User-agent: *
Disallow: /

# Fixed: only block admin/internal paths
User-agent: *
Disallow: /admin/
Disallow: /internal/

Test any robots.txt change with a URL inspection tool before deploying it, since a mistake here silently removes pages from every compliant crawler at once.