AI crawlers and JavaScript: why your page looks empty
A single-page app sends an empty container and a script tag, so a fetcher that does not execute JavaScript receives no text at all; here is how to check and how to fix it.
The answer
A client-rendered page ships an empty container and a script tag. If the fetcher reading it does not execute JavaScript, that is the whole document: no headline, no body copy, no prices, nothing to quote. The fix is to put the text in the HTTP response, and the check takes one curl command.
Here is what that failure looks like, unedited in shape, from a typical single-page app build.
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>Acme Pricing</title>
<link rel="stylesheet" href="/assets/index-8f3a1c.css">
</head>
<body>
<div id="root"></div>
<script type="module" src="/assets/index-4b91de.js"></script>
</body>
</html>Words of readable body text in that response: zero. Strip the markup and the only string a text extractor recovers is the two-word title, and a title is not an answer. The browser version of this page may carry 900 words of pricing detail; none of it exists until a script runs.
The server-rendered version of the same route:
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Acme Pricing</title>
<link rel="stylesheet" href="/assets/index-8f3a1c.css">
</head>
<body>
<div id="root">
<h1>Acme Widgets pricing</h1>
<p>Acme bills per widget shipped, in USD, with no minimum order and
no annual commitment. Volume tiers apply automatically at invoice.</p>
<table>
<caption>Per-unit price by monthly volume</caption>
<thead><tr><th>Units per month</th><th>Price per unit</th></tr></thead>
<tbody>
<tr><td>1 to 99</td><td>$41.00</td></tr>
<tr><td>100 to 999</td><td>$36.50</td></tr>
<tr><td>1,000 and above</td><td>$29.75</td></tr>
</tbody>
</table>
<h2>What is included</h2>
<p>Every order includes load certification, palletised delivery, and
a 30-day return window measured from delivery.</p>
</div>
<script type="module" src="/assets/index-4b91de.js"></script>
</body>
</html>Same application, same bundle, same hydration. The difference is that the answer to "what does a widget cost" is now in the bytes. The interactive parts still work, because the script still loads; it takes over markup that already exists instead of creating it.
What "no JavaScript execution" actually means
Be precise here, because vendor documentation is thin. Of the operators in our registry, none of the AI-specific fetchers publish whether they run JavaScript. Google documents a rendering step for its search crawl (Google JavaScript SEO basics), and Googlebot is also the crawl behind AI Overviews and AI Mode. For OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and the rest, the operators say nothing either way.
Silence is not reassurance. Rendering is expensive: it means running a browser engine per URL instead of an HTTP client. When a vendor invests in that, it says so. The load-bearing assumption is therefore the pessimistic one, and it is also the cheap one to satisfy: if the text is in the response, every fetcher gets it, renderer or not. If the text is only in the DOM after hydration, you are betting your citations on an undocumented capability.
Three consequences follow that people usually discover in the wrong order:
- Content behind a data fetch is invisible even to renderers with a budget. A shell that hydrates and then issues three API calls before painting text is two round trips deep. Anything with a timeout gives up first.
- Client-side routing hides your best pages. If
/pricingonly exists as a route in a bundle, there is no URL to cite. - Inline state blobs make it worse. A 400 KB JSON payload in a script tag inflates the response without adding a word of extractable text, and extractors strip scripts before a model sees anything.
Verify it in one command
Request your own page as a fetcher would, with no browser in the loop. The user agent is the real OAI-SearchBot string from OpenAI's documentation:
curl -sL -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
https://example.com/pricing | wc -cThat gives you bytes. Words of readable text is the number that matters, so strip the markup and count:
curl -sL -A "compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://example.com/pricing \
| tr '\n' ' ' \
| sed -e 's|<script[^>]*>[^<]*</script>| |g' -e 's|<style[^>]*>[^<]*</style>| |g' -e 's|<[^>]*>| |g' \
| tr -s ' ' \
| wc -wThen confirm the one thing that command cannot tell you: whether a specific sentence survived. Grep for a phrase you know is on the page.
curl -sL -A "compatible; ClaudeBot/1.0" https://example.com/pricing | grep -c "no annual commitment"Zero means the sentence exists only after hydration. The browser check is the same test by hand: open DevTools, disable JavaScript, hard-reload. What remains is approximately what a non-rendering fetcher receives. Use view-source rather than the Elements panel, because the Elements panel shows the live DOM and will happily show you content that was never in the response.
Run the same command against a competitor who gets cited for the queries you want. The comparison is usually more persuasive internally than any argument about crawlers.
The fixes, ranked by effort against payoff
- Static generation at build time. Cheapest and most robust where content changes on a deploy cadence: documentation, marketing, blogs, catalog pages. The HTML exists as a file. There is nothing to time out.
- Server-side rendering. The right answer for anything personalised or frequently updated. Every major framework supports it; the work is usually removing browser-only assumptions from components, not adopting a new tool.
- Prerendering for crawlers. A rendering service or edge worker returns pre-built HTML for the routes that matter. Faster to deploy than a framework migration and genuinely useful as a stopgap. Keep the prerendered HTML identical in substance to the browser version; serving crawlers different content than people is a policy problem, not just a technical one.
- Partial hydration and islands. The best long-term shape: ship the article, product description or spec table as HTML, and hydrate only the interactive islands such as filters, carts and comment threads. This is the architecture that makes the whole question go away.
- A
<noscript>block is not a fix. Duplicating your content into<noscript>creates a second copy to drift out of date, and nothing in any vendor documentation says a fetcher prefers it. If you are willing to write the content into the HTML response, write it into the page.
Two Google fetch mechanics worth designing for
From the Google crawler documentation (read 2026-08-21): Google fetches the first 15 MB of a file by default, and content past that byte is not considered. That is generous until an inline state blob, a base64 image or a large embedded dataset pushes your text past it. Compression counts in your favour here, and gzip, deflate and Brotli are all supported.
Google also supports ETag with If-None-Match and Last-Modified with If-Modified-Since, and recommends ETag. No other cache directives are supported. Serving a stable ETag and answering conditional requests with 304 reduces the cost of being crawled frequently, which matters if your objection to crawlers is bandwidth rather than principle. Note that these are documented for Google's crawlers specifically; do not assume other operators implement conditional requests.
How to read the words number in your report
Crawl Census requests your home page once with a desktop browser user agent, strips scripts, styles and markup, and counts the words that remain. That count is reported as "words of text" and it drives the server-rendered body text check, worth 9 of 100 points. The thresholds are fixed and published: 300 words or more passes, 120 to 299 warns, below 120 fails, with partial credit scaled inside the failing band. Full weights are on the methodology page.
Two honest limits on that number. It describes your home page only, not a crawl, so a marketing home page that renders on the server can score well while the documentation that actually answers questions ships an empty shell — check your important routes with the curl commands above. And a high word count is not proof of extractability: 2,000 words of navigation, cookie notices and footer links counts as 2,000 words. The companion check, the ratio of visible text to HTML bytes, exists because of exactly that.
Run a scan to see how many words of text your home page delivers before any JavaScript runs, which framework fingerprints appear in the response, and how much of your payload is markup rather than content.