Rankawy
For developers and site owners

RankawyBot: the Rankawy crawler

RankawyBot reads public pages at Rankawy users' request for tasks such as site audits and page comparisons, or during scheduled monitoring of added sites. It detects SEO issues and shows the information needed to suggest fixes.

Last updated: 2026-10-10

In short

  • It visits pages during audits and comparisons you request, and scheduled monitoring of added sites.
  • It respects available robots.txt rules and applies Crawl-delay up to 10 seconds. It starts at one request per second, may speed up depending on responses, slows down when rate-limited and stops after repeated refusals.
  • It reads public pages only: no logging in, no form submissions, no purchases.
  • If a firewall stops it, the sections below show how to allow it step by step.

How to recognize it

Every request carries this User-Agent:

Mozilla/5.0 (compatible; RankawyBot/1.0; +https://rankawy.com/bot)

It also sends Accept-Language with Arabic first, because Rankawy audits Arabic sites the way their visitors see them.

How it behaves

  • It reads robots.txt before any page and follows the RankawyBot rules there, or the * rules when there are none for it.
  • It honours Crawl-delay up to 10 seconds per request.
  • It starts at one request per second, slows down on the first 429, honours Retry-After, and stops after repeated refusals.
  • On hosted stores (Shopify, Salla, Zid), it never goes above one request per second.
  • It never follows links to other sites and reads at most 1 MB of a page.

How to block it

Add these two lines to robots.txt and it stops auditing your site from its next visit:

User-agent: RankawyBot
Disallow: /

How to allow it

Let Rankawy audit your Shopify store

Shopify's protection may block audit requests. Shopify provides a signature for crawler access through Web Bot Auth; create it in your store settings, then add its values in Rankawy.

  1. 1Open your Shopify admin, then Online Store, then Preferences.
  2. 2Under Crawler access, click Create signature.
  3. 3Name it Rankawy, choose your store's primary domain, and set the expiry to 3 months.
  4. 4Copy the Signature-Input and Signature values, paste them on the audit page in Rankawy, then run the audit again.

Allow Rankawy's crawler in Cloudflare

Cloudflare refused our crawler's requests. To allow the audit, review the rule that refused RankawyBot and configure an exception for it.

  1. 1Open Cloudflare, choose your site, then Security, then Security rules (or WAF, then Custom rules).
  2. 2Create a rule: field User Agent, operator contains, value RankawyBot.
  3. 3Choose the Skip action, select custom rules, rate limiting rules, and Super Bot Fight Mode to skip, then save.
  4. 4If your Free plan uses Bot Fight Mode, no rule can bypass it. Temporarily turn it off under Security, then Settings until the audit finishes, then turn it back on.
  5. 5Run the audit again.

Allow the Rankawy crawler in Vercel

Vercel's bot protection stopped our crawler before it could read your pages.

  1. 1Open your project in Vercel, then Firewall.
  2. 2Add a rule: if the User-Agent contains RankawyBot, the action is Bypass.
  3. 3Save and publish the rule, then run the audit again.

Your store platform stopped the audit

Store platforms such as Salla and Zid protect stores with a firewall the owner cannot control. We kept the pages we already audited, and we audit these stores more slowly.

  1. 1Run the audit again in an hour.
  2. 2If it stops again, contact the platform's support and ask them to allow the RankawyBot crawler.
  3. 3Or contact our support and we will follow up with the platform.

Allow the Rankawy crawler through your firewall

A firewall on your site or host refused our crawler. Add RankawyBot to its allow list, then run the audit again.

  1. 1In Wordfence, Sucuri, Imunify360 or your host's protection: add an allow rule for requests whose User-Agent contains RankawyBot.
  2. 2If you cannot find the setting, send this message to your hosting provider’s support team: “Please allow the RankawyBot crawler to access my site; details at rankawy.com/bot.”
  3. 3Run the audit again.

Contact us

If you see unexpected behavior from RankawyBot on your site, write to us with the page URL and a description of what happened.