GevellBot
GevellBot is Gevell's web crawler. If you found this page in your server logs, this is what it was doing and how to control it.
What it does
GevellBot looks for documents that public agencies post on their websites, mainly PDF reports, and checks a small sample of them for accessibility: whether they are tagged, have a language and title, and give images text alternatives. We use the results to find report series that people using screen readers can't read, so we can offer to fix them.
It does not log in, submit forms, run scripts on your pages, or republish your documents.
How to recognize it
Requests carry this User-Agent:
Mozilla/5.0 (compatible; GevellBot/1.0; +https://gevell.com/bot)
Earlier research runs identified as GevellResearch/1.0 or TheAccessibleBot/1.0 (Gevell was formerly TheAccessible.org). The same rules apply to them.
How it limits itself
- It reads
robots.txtbefore anything else and follows it, includingCrawl-delay. - It sends one request at a time to a site, never in parallel.
- It makes at most 300 requests to a site in one night.
- It visits a site about once a month, or weekly while the site is posting new documents.
- It downloads at most 20 documents from a site per visit, and none larger than 64 MB.
- It prefers sitemaps and, on WordPress sites, the public media listing, so it rarely needs to follow links page by page.
- If a site answers with a bot challenge or blocks it, it stops and does not try to get around the block.
How to slow it down or block it
To block it from your whole site, add this to your robots.txt:
User-agent: GevellBot Disallow: /
To keep it but space out its requests, for example to one every 10 seconds:
User-agent: GevellBot Crawl-delay: 10
Changes take effect the next time it visits your site.
Contact
If GevellBot is causing a problem, or you'd like us to stop visiting, email support@gevell.com with your site's address. A person will reply.