Skip to content
Tomas KadlecBook a Call

Can AI crawlers read your website?

ChatGPT, Claude, Perplexity and Google's AI answers can only use pages their crawlers are allowed to fetch and able to read.

Enter a page. You see which of 16 AI crawlers get in and what stops the others.

Check a page

Without an e-mail you get the short result on screen and nothing is stored. With an e-mail I send the full report with fixes, keep the e-mail and the address for 12 months and may write to you once about the result. Details in the Privacy.

What it checks

  • robots.txt for 16 AI crawlers

    OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, Common Crawl and ByteDance. For each crawler: allowed or blocked, and whether it serves AI search, user requests or model training.

  • How the server answers

    The page is requested as a normal browser and with the names of three AI crawlers. A firewall that refuses them shows up here.

  • Text without JavaScript

    Most AI crawlers do not run JavaScript. The check counts the words they can actually read.

  • Signals on the page

    Title, description, one main heading, language, noindex and snippet limits.

  • Structured data and sitemap

    schema.org data on the page and a sitemap for the site.

What it cannot tell you

The requests come from my server, not from the crawlers' own addresses, so a firewall can still treat the real crawlers differently. The check reads one page. And a readable page is not yet a page that AI assistants mention.

Should I block the training crawlers?

That is your decision. Blocking a training crawler does not remove you from AI search, as long as the search and user-request crawlers stay allowed. The result shows the two groups apart. Two crawlers that act on a person's request, ChatGPT-User and Perplexity-User, may not follow robots.txt at all, according to their operators.

Does the tool change anything on my site?

No. It reads the page, robots.txt, the sitemap and llms.txt, seven requests in total.

Which page should I check?

Start with the home page, then a product or service page. Templates often differ.

Readable is step one. Mentioned is the goal.

The AI-visibility audit measures whether ChatGPT and Google's AI answers name your brand, and why.

See the AI-visibility auditBook a free 20-minute call