Can the assistant read you? Crawlers, firewalls and JavaScript

One in twelve could not be read

When we read 274 B2B software homepages the way an AI crawler does, 23 refused us with a 403, a 429 or a challenge page. None of them was down. Twenty still refused when the request claimed to be an ordinary browser. A site that cannot be read is described by what other people say about it, which may be out of date, or a competitor's comparison page.

Check one: robots.txt

The file at /robots.txt tells crawlers what they may fetch, with rules per user agent. A line such as User-agent: GPTBot followed by Disallow: / shuts that crawler out completely. Look for the AI crawlers named in lesson 1 and decide, on purpose, which ones you allow.

Remember the difference between training and retrieval. Blocking GPTBot stops OpenAI's training crawler; it does not by itself stop the crawler behind ChatGPT's search, which has its own name. In our study only 4.4% of sites blocked any AI crawler, and most of those blocks were aimed at bulk scrapers such as Bytespider. GPTBot was blocked by 2 of 251.

Check two: the firewall

This is the block nobody chose. Bot protection on a CDN or firewall can answer an AI crawler with a 403 or a challenge page even though robots.txt says yes. Some providers have a switch that blocks AI crawlers, and in some accounts it is on by default. We found exactly this on our own site, the first time we put it behind a CDN.

The test is simple: request your homepage with an AI crawler's user agent and look at the status code. Anything other than 200 means the door is shut, whatever robots.txt says. If it is shut, find the bot or AI setting in your CDN or firewall and allow the crawlers you chose in check one.

Check three: the words in the HTML

Most AI crawlers do not run JavaScript. They read the HTML the server sends. If your site builds its content in the browser, the crawler sees an empty shell: a <div id="root"> and a script tag.

Use your browser's View Source, not the element inspector, which shows the page after scripts ran. If your headline and the paragraph under it are not in the source, render those pages on the server or pre-render them at build time. Marketing pages almost never need to be built in the browser.