Does robots.txt keep a page out of Google?

No — it only controls crawling. A disallowed page can still be indexed if other sites link to it, just without Google reading its content. To reliably keep a page out of the index, allow crawling and add a noindex meta robots tag.

Should I block AI crawlers?

It depends on your goal. Blocking assistant crawlers like GPTBot and PerplexityBot removes you from those tools' answers — usually the opposite of what you want. Blocking training crawlers is a reasonable content-protection stance. Make the two decisions separately.

Do I need a sitemap if my site is small?

It still helps. Even a handful of URLs benefit from an explicit, canonical list that you reference from robots.txt — it removes any doubt about which pages you want found.

Learn

Robots.txt and sitemaps: a practical guide

robots.txt controls which crawlers can fetch what; your sitemap lists the pages you want found. Here's how to configure both for search engines and AI bots without accidentally blocking yourself.

Updated 2026-07-19

robots.txt and sitemap.xml are the two files that tell crawlers how to move through your site. robots.txt, at your site root, sets permissions — which user-agents may fetch which paths. Your sitemap is the inverse: an explicit list of the URLs you want crawlers to find and index. One gates access; the other advertises coverage. Together they decide, before any ranking or citation happens, whether search engines and AI answer engines can even see your pages.

They're also two of the easiest places to shoot yourself in the foot — a stray Disallow: / or a sitemap full of dead URLs can quietly suppress a whole site. This guide covers how to get both right.

How does robots.txt work?

robots.txt is a plain-text file of rules grouped by user-agent. Each group names one or more crawlers and lists Allow and Disallow paths that apply to them:

txt

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

The User-agent: * group applies to any crawler without a more specific group. You can also target named bots. The critical thing to understand is that Disallow: / blocks the entire site for that group — a common accident when a staging config ships to production. Sight's robots-blocked check flags any crawled page that robots.txt disallows, because a page you link to but block is a page that can rank for no one.

Two more rules of thumb: robots.txt controls crawling, not indexing — a blocked page can still appear in results if others link to it, so use a noindex meta tag when you truly want a page out of the index. And always reference your sitemap from robots.txt with a Sitemap: line so crawlers discover it.

Which AI bot user-agents should I know about?

AI answer engines fetch with their own named user-agents, and you can allow or block them in robots.txt just like any crawler. Sight's AI assistant access check specifically probes the crawlers that feed live AI answers — GPTBot (OpenAI), ClaudeBot and Claude-Web (Anthropic), and PerplexityBot (Perplexity). If these are blocked, your site is invisible to those assistants' answers.

Separately, there's a class of training crawlers — Google-Extended, CCBot (Common Crawl), Bytespider (ByteDance), and Applebot-Extended (Apple) — that gather content for model training rather than live answers. Blocking those is a legitimate content-protection choice, and Sight treats it as a neutral policy decision, not an error. The distinction matters: you might want to welcome the assistant crawlers that can cite you while opting out of training. Decide each deliberately rather than blanket-blocking everything with AI in the name.

What makes a healthy sitemap?

A sitemap is an XML file listing your indexable URLs. Good sitemap hygiene comes down to two properties Sight checks:

Coverage — every indexable page you crawl should be in the sitemap. Sight's sitemap coverage check flags indexable pages that are missing, because a page absent from the sitemap is one search engines may discover late or not at all.
No dead entries — every URL in the sitemap should return a healthy status. The sitemap 404 check flags entries pointing at broken pages, which waste crawl budget and signal neglect.

Keep the sitemap in sync with reality: list canonical URLs only, drop pages you've noindexed, remove URLs you've retired, and don't pad it with redirect or parameter variants.

Does robots.txt keep a page out of Google?: No — it only controls crawling. A disallowed page can still be indexed if other sites link to it, just without Google reading its content. To reliably keep a page out of the index, allow crawling and add a noindex meta robots tag.
Should I block AI crawlers?: It depends on your goal. Blocking assistant crawlers like GPTBot and PerplexityBot removes you from those tools' answers — usually the opposite of what you want. Blocking training crawlers is a reasonable content-protection stance. Make the two decisions separately.
Do I need a sitemap if my site is small?: It still helps. Even a handful of URLs benefit from an explicit, canonical list that you reference from robots.txt — it removes any doubt about which pages you want found.

The takeaway

robots.txt decides who gets in; your sitemap tells them what's worth finding. Audit robots.txt for accidental Disallow: / rules, decide AI assistant versus training access on purpose, and keep your sitemap to live, canonical, indexable URLs. Getting access right is only half the job — the other half is making sure crawlers can read what they're allowed to fetch.

Check your robots.txt and sitemap for blocking issues — run a free scan

Run a free scan