For website owners · Last updated 24 October 2026

SabiyaBot

SabiyaBot is how Sabiya reads public websites. If SabiyaBot or SabiyaRankBot shows up in your logs, it was us. Here's what we read, why, and how to control it.

When we visit

We only visit when a Sabiya feature needs a page. We don't crawl the web at large.

1. A business sets up Sabiya with its website

   We read its homepage to suggest its name, industry, location and main offering. If that address doesn't answer, we try it with or without www. The owner reviews every suggestion before Sabiya uses it.

2. Sabiya imports a business's services or products

   For services, we read the page the owner points us to. For products, we read the site's public product listing. Product scans don't keep prices.

3. An owner runs a Rank audit

   We read their site's pages, robots.txt, sitemaps and llms.txt. A Full Audit also reads those files and the homepage and up to two other pages of each competitor the owner names.

4. We check a booking link

   One quick request to a booking link a business gave us, to make sure it still opens. This check doesn't read robots.txt.

5. We prepare a demo

   When we build a Sabiya demo for a business we're talking to, we read a few public pages of its site.

6. A feature needs a screenshot

   Screenshots come from a screenshot service, ScreenshotOne, which visits with its own user agent, not SabiyaBot, so your rules for SabiyaBot don't apply to it. ScreenshotOne's docs explain its user agent.

How we behave on your site

At once

2 requests per site, from each Sabiya server

Gap

At least 250 ms between requests, or your Crawl-delay (up to 5 s)

Page size

Stops at 2 MB (5 MB for sitemaps and product listings)

What we run

Plain page requests only; no JavaScript, forms or logins

• We follow your robots.txt (RFC 9309), including Allow, Disallow, wildcards and Crawl-delay (up to 5 seconds).
• If your robots.txt can't be loaded (your server returns an error or doesn't answer), we don't read your site. We try again the next time a Sabiya feature needs a page. If you have no robots.txt (a 404), we read public pages as normal.
• We reuse your robots.txt for up to 10 minutes, so a change can take that long to apply.
• At most 2 requests to your site at once from each Sabiya server, at least 250 ms apart, or slower if your Crawl-delay asks.
• We read public pages only. We don't log in, fill in forms or run your site's scripts.
• We stop reading a page after 2 MB (5 MB for sitemaps and product listings) and give up on a slow page after 4 to 25 seconds, depending on the feature.

Block or slow us down

Add a group to your robots.txt. A rule for either name covers every page we read; if you list both, each follows its own rules. If you list neither, we follow your User-agent: * group.

User-agent: SabiyaBot
Disallow: /

User-agent: SabiyaBot
Crawl-delay: 5

A block doesn't stop the booking-link check, or Rank reading your robots.txt, sitemaps and llms.txt and checking which address your homepage answers on (with or without www).

Using Sabiya yourself? If you block us, setup and Rank can't read your site. You can still type your details in yourself.