Our crawler
Last updated: September 21, 2026.
If you have seen this user agent in your server logs, this page explains who it is, what it reads, and how to stop it:
- Desvelio/1.0 (+https://desvelio.com/bot)
Who
Desvelio is a tool for local newsrooms. The operator is named in the legal notice. One of its features, Competitors, shows a newsroom which of the day’s stories the nearby publications it names are also covering. To do that, our server reads the public RSS feed and the front page of those publications.
What it reads and what it keeps
- It reads the RSS feed the publication offers its readers, its front page and, where there is one, its most-read page.
- It keeps only the headline, the link, the publication time, the section, and the position on the front page. For 90 days.
- It does not download article bodies, images, comments, or anything behind a login. It does not run JavaScript or accept cookies.
- It republishes nothing. The newsroom using Desvelio sees the headline and a link that opens the original article on the publication’s own site.
How often
The RSS feed and the front page about every two minutes: two or three requests per cycle. Never more than one request per second to the same site, one connection at a time. robots.txt is read once a day. That is less traffic than one reader with the front page open.
From where
Always with this user agent and from our own server in the European Union, on a fixed IP address: the one desvelio.com resolves to. If you need the exact address for an allow list, ask us through the contact form.
How to stop it
We honour robots.txt. To make us stop reading your site, add this block and we apply it on the next read, within a day:
- User-agent: Desvelio
- Disallow: /
If you would rather we did it, or you want to know which publication asked us to read yours, write to us through the contact form. We reply the same business day. How we treat the data of people who write to us is in the privacy policy.