Free website tool
See which AI crawlers your robots.txt lets in and which it blocks, crawler by crawler, and whether the site publishes an llms.txt file.
Instant result, no email needed.
robots.txt is read the way RFC 9309 describes: a crawler follows its own group if one names it, otherwise the "*" group, and for the site root the longest matching rule wins (an allow wins a tie). "Partly blocked" means the root is allowed but some paths are not. robots.txt is a request, not an enforcement mechanism: well-behaved crawlers follow it, others may not. The crawler names are the ones their operators publish.
For the site's HTTP security headers, use the Security Headers Checker.
Blocking GPTBot asks OpenAI not to use your pages for training. ChatGPT search uses OAI-SearchBot, and pages a user asks about are fetched as ChatGPT-User; each has its own rule. The same split exists for Anthropic, Perplexity and others.
A token that controls whether Google may use your content for Gemini. It does not affect Google Search crawling or ranking, which is Googlebot.
A proposed Markdown file at /llms.txt that gives language models a short guide to a site's most useful pages. It is not a standard and is not required.
Some sites block unknown clients or answer with an error. CloudSpex then shows no crawler results rather than guessing what the rules are.
Add your domain to CloudSpex to monitor its public configuration continuously.
Start Monitoring Free