Free tool · no sign-up
robots.txt checker and who can crawl your site
Enter your domain or a page URL. We read robots.txt the way Google does and tell you which crawlers are allowed and which are blocked, from Googlebot to the ChatGPT, Claude, and Perplexity crawlers, plus an llms.txt check.
- Free, no sign-up
- Google and 18 AI crawlers
- Instant result
What we check in robots.txt
Each crawler
Googlebot and AI crawlers: which ones may access your home page and your URL.
File errors
Unknown or malformed lines, with their place in the file.
Content-Signal and protection
Content-use signals, and the CDN that protects your site.
llms.txt
Whether your site has it, and what it contains.
Googlebot
OAI-SearchBot
PerplexityBot
Claude-SearchBot
Google-Extended