Rankawy
Free tool · no sign-up

robots.txt checker and who can crawl your site

Enter your domain or a page URL. We read robots.txt the way Google does and tell you which crawlers are allowed and which are blocked, from Googlebot to the ChatGPT, Claude, and Perplexity crawlers, plus an llms.txt check.

  • Free, no sign-up
  • Google and 18 AI crawlers
  • Instant result

Enter the domain or a page URL to see whether the file allows crawlers to reach it.

What we check in robots.txt

  • Each crawler

    Googlebot and AI crawlers: which ones may access your home page and your URL.

  • File errors

    Unknown or malformed lines, with their place in the file.

  • Content-Signal and protection

    Content-use signals, and the CDN that protects your site.

  • llms.txt

    Whether your site has it, and what it contains.