Our crawler.
How VaybelDirectoryResearch identifies itself, what it reads, the limits it keeps, and how to limit or block it.
What it is
VaybelDirectoryResearch is the crawler behind the Vaybel suppliers directory. It reads a few public pages on an apparel business’s own website, such as its home, about and services pages, to describe what the business offers: what it makes, minimum orders, where it is, its certifications, its logo and up to three photos of its premises.
Its user agent starts with VaybelDirectoryResearch and links to this page.
How it identifies itself
Every request is signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). The Signature-Agent header is "https://vaybel.com", and the public keys are published at:
https://vaybel.com/.well-known/http-message-signatures-directory
A site can check any request against those keys to confirm it comes from Vaybel. A request that claims our name without a valid signature isn’t ours.
How it reads a site
It follows robots.txt (RFC 9309), including rules and crawl delays written for VaybelDirectoryResearch or for all crawlers.
It makes one request at a time to any site and reads only a handful of pages per business. It reads public pages only: it doesn’t fill in forms, sign in, add to carts or use search boxes.
A refusal ends the visit. A 403 or 429 answer, a bot check, a CAPTCHA or a login page stops it reading that site, and it doesn’t try to get around it.
What happens to the pages
We keep a private copy of each page it reads, as the source for the details on the profile, and don’t republish it. Profiles describe the business in our own words, and each detail links to the page it came from.
Photos of a business’s premises are loaded by the visitor’s browser straight from the business’s own website. We don’t copy them.
Limiting or blocking it
To stop it reading your site, add this to your robots.txt:
User-agent: VaybelDirectoryResearch Disallow: /
It applies from our next visit. To correct or remove a listing instead, see how the directory works. For anything about the crawler, write to contact@vaybel.com.