My site works fine in Google. Doesn't that mean crawlers can read it?

Not necessarily. Google runs a rendering pipeline that executes JavaScript, so it may see content that a lighter AI crawler — which reads raw HTML only — will miss. Passing Google is not proof an AI answer engine sees the same page.

Does this mean I can't use React or Vue?

You can. The fix isn't dropping your framework; it's rendering its output on the server (SSR) or at build time (static generation) so the HTML response already contains your content. The client-side framework then hydrates that HTML instead of building it from scratch.

What's the single fastest way to check?

curl the URL and read the raw HTML. If your headline and body text are in that response, crawlers can read them.

Learn

How AI crawlers read your site

AI answer engines mostly fetch your raw HTML without running JavaScript. If your content only appears after a client-side render, they see an empty shell. Here's what that means.

Updated 2026-07-19

When an AI answer engine like ChatGPT, Claude, or Perplexity looks at your site, it usually does one thing: it fetches the raw HTML at a URL and reads whatever text is already in that response. As of mid-2026 most AI crawlers do not run JavaScript the way a real browser does — they take the HTML the server sends, parse it, and move on. So the question that decides whether your content is even visible to them is simple: is your text present in the initial HTML, or does it only appear after a client-side framework renders it in the browser?

That distinction is invisible to you as a human, because your browser runs the JavaScript and shows you the finished page. But curl https://yoursite.com shows what a non-executing crawler sees. If that raw response is mostly an empty <div id="root"></div> and a bundle of scripts, then to a crawler that doesn't execute JS, your page has almost no content — no matter how rich it looks in Chrome.

Why don't AI crawlers just run the JavaScript?

Some can, but many won't, and you shouldn't count on it. Executing JavaScript is expensive: it means launching a headless browser, waiting for network-idle, and rendering, per page, across billions of pages. Search engines like Google invested years in a two-pass rendering pipeline to do this at scale; most AI crawlers are far newer and lighter, and default to reading raw HTML. Treating server-delivered HTML as the contract — rather than assuming the crawler will render for you — is the safe posture. This is why we say server-side rendering (SSR) or static generation matters for GEO: it guarantees your content is in the response every reader gets, human or machine.

What signals do crawlers read besides the body text?

Beyond the visible text, crawlers lean on a handful of machine-readable signals that live in your HTML <head> and response headers:

The <title> and meta description — the first summary of what a page is about. See meta tags that matter.
Structured data (JSON-LD) — explicit, typed statements about the page's entity that a model can trust without inferring. See an intro to structured data.
Response headers — the Content-Type, caching directives, and any X-Robots-Tag that might opt the page out of AI use.
robots.txt — whether your site permits AI user-agents like GPTBot, ClaudeBot, and PerplexityBot to fetch at all.

Sight probes exactly these AI user-agents when it checks your AI assistant access, because a page can be perfectly rendered and still be unreadable if robots.txt blocks the crawler at the door.

How can I tell what a crawler actually sees?

The fastest check is to view the raw HTML, not the rendered page. In a terminal, curl -s https://yoursite.com | less shows the exact bytes a non-executing crawler receives. If your main heading, body copy, and links are all there in that output, you're in good shape. If you see a skeleton — an empty root element and script tags, with your real content nowhere in the source — your content depends on client-side rendering and is at risk of being invisible.

Sight automates a version of this: it flags pages whose visible text is a tiny fraction of the raw HTML (a common fingerprint of an unrendered single-page app), because that pattern almost always means a crawler that doesn't execute JavaScript will come away with nothing.

My site works fine in Google. Doesn't that mean crawlers can read it?: Not necessarily. Google runs a rendering pipeline that executes JavaScript, so it may see content that a lighter AI crawler — which reads raw HTML only — will miss. Passing Google is not proof an AI answer engine sees the same page.
Does this mean I can't use React or Vue?: You can. The fix isn't dropping your framework; it's rendering its output on the server (SSR) or at build time (static generation) so the HTML response already contains your content. The client-side framework then hydrates that HTML instead of building it from scratch.
What's the single fastest way to check?: curl the URL and read the raw HTML. If your headline and body text are in that response, crawlers can read them.

The takeaway

Design for the reader that doesn't run your code. If your important text, headings, and links are present in the server's HTML response, every crawler — search or AI — can read them. If they only exist after JavaScript runs, you're betting your visibility on each crawler choosing to render, and that's a bet you don't need to make.

See what an AI crawler sees on your site — run a free scan

Run a free scan