Can AI crawlers read your website? 13 checks

You can write the best page in your market and still be invisible in ChatGPT, Claude or Perplexity. Not because the content is bad. Because their crawlers never got to read it.

I built a free tool that checks this for one page in about 20 seconds. Then I ran it on my own website first. It failed.

My own site: 12 of 13

On 11 November 2026 at 14:35 the check on tomakadlec.com showed 12 of 13. The one miss: no schema.org data. The page was readable, crawlers were allowed, but there was no structured data that tells a machine "this is a person, this is a service, this is an article".

I added JSON-LD to every page in all five languages. At 14:46 the same check showed 13 of 13. Eleven minutes. Most fixes on this list are that small once you know they are missing.

The 13 checks, in three groups

Access: can the crawler get in?

1. robots.txt exists. A working file at the root of the domain. Missing or broken files make crawler behaviour unpredictable.
2. AI search crawlers are allowed. The tool reads robots.txt for 16 AI crawlers. The ones that matter most for visibility are search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot. They fetch pages for AI search answers.
3. The server answers AI crawlers. Some firewalls and bot protections refuse any request with "GPTBot" or "ClaudeBot" in it. robots.txt says yes, the server says 403. The tool sends requests with those names and compares the answer with a normal browser.
4. The page loads. HTTP 200. Sounds obvious, I have seen 301 chains and 404s on pages that "work in the browser".

Readability: can it understand the page?

1. Text readable without JavaScript. Most AI crawlers do not run JavaScript. If your content appears only after scripts load, the crawler sees an empty shell. The tool counts the words that are in the HTML itself.
2. Page title.
3. Meta description.
4. Exactly one H1.
5. Language stated in the html tag.

Signals: is it allowed to use what it read?

1. Not marked noindex. A forgotten noindex from a staging site kills the page for search and AI search alike.
2. Snippets allowed. nosnippet or max-snippet:0 limits how much AI search can quote from you.
3. Structured data. schema.org JSON-LD for what the page is: Organization, Person, Product, Article, FAQPage.
4. Sitemap. A sitemap.xml, ideally named in robots.txt.

Training vs search: block one, keep the other

Many sites blocked "AI bots" in 2023 and 2024 with one line per bot. That often blocks the wrong thing.

The big AI companies run separate crawlers. GPTBot collects data for model training. OAI-SearchBot fetches pages for ChatGPT search answers. Anthropic has ClaudeBot for training and Claude-SearchBot for search. Google-Extended controls Gemini training, while Googlebot stays your normal search crawler.

So you can say no to training and yes to being cited:

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /

The tool shows each crawler with its purpose (training, search or user request), so you see what your robots.txt actually decides.

Two honest limits. OpenAI and Perplexity say their user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt, because a person asked for that page. And the test requests come from my server, not from the crawlers' own IP addresses, so a CDN that verifies crawler IPs can still answer differently.

What to do this week

1. Run the check on your homepage and your most important money page.
2. Fix every red line first: blocked search crawlers, server refusals, noindex, empty pages without JavaScript.
3. Then the yellow ones: structured data, description, one H1, sitemap.
4. Decide on training crawlers on purpose, not by accident.
5. Run it again after every bigger site change. Plugins and CDN settings change robots and headers without telling you.

Summary

AI visibility starts before content. If crawlers can't get in or can't read the page, nothing else matters. 13 checks, most fixes take minutes.

Run the free check: AI crawler check on tomakadlec.com. One URL, no sign-up. The full report with fixes comes by e-mail if you want it.

Which of the 13 do you expect your site to fail?

Sources: own run of the AI crawler check on tomakadlec.com, 11 November 2026 (12 of 13 at 14:35, 13 of 13 at 14:46) · crawler list checked on 11 November 2026 against the crawler documentation of OpenAI, Anthropic and Perplexity · RFC 9309 (Robots Exclusion Protocol).