CityVibeBot — who our crawler is

You have seen us in your logs

This is our identifier, in full:

CityVibeBot/1.0 (+https://cityvibe.guide/bot-info)

If it shows up on your server, it is us. We use no other agent, we do not disguise ourselves as a browser and we do not rotate IP addresses to dodge anything.

What CityVibe is

A catalogue of events in Spain. We collect cultural listings published as open data, translate them into seven languages and show them together, always crediting the source and linking to the original. The full list of sources is in Data sources.

What we download

Data feeds: JSON, XML, CSV. Specific addresses from open data catalogues, not a walk through your site.

We do not download:

• Navigation HTML pages, search pages or print versions.
• Images: none, ever. No photos, no posters, no thumbnails. We neither store them nor hotlink them.
• Email addresses or phone numbers, even when they come in the feed. When a field contains a person's contact details, our adapter simply does not read it.
• Anything behind a login or a form.

How often

We check every source every three hours. Between one request and the next to the same server we wait, at minimum:

esmadrid.com

10 seconds — exactly the “Crawl-delay” its robots.txt asks for

do.diba.cat

10 seconds — exactly the “Crawl-delay” its robots.txt asks for

datos.madrid.es

2 seconds

datosabiertos.jcyl.es

2 seconds

A typical run against one source is between one and five requests. If you answer “429” or “503” we back off exponentially, and if you send a “Retry-After” header we honour it to the letter. We use “ETag” and “Last-Modified” when you offer them, so a feed that has not changed costs both of us almost nothing.

robots.txt

We read it and we respect it, and it is worth knowing how: a person reviews it when we write the adapter for your source, not an automatic parser on every run. In practice that makes us stricter, not laxer — the paths you disallow never get written into the code, and the “Crawl-delay” you ask for is fixed in there. A real example: we dropped a dataset we wanted because it redirected to a host whose robots.txt disallows “/api/”, and we kept the original, served from an allowed path.

Nor do we follow redirects blindly: every hop is checked and counted.

How to block us

If you would rather we did not come by, add this to your robots.txt:

User-agent: CityVibeBot
Disallow: /

We will see it at the adapter's next review. If you want us to stop coming by today, write to us and we will switch it off within minutes: in our system a source is a row with an on/off switch.

If what you want is for us to go slower, or only at certain hours, say so and we will adjust it. It is a number in a configuration file.

How to talk to us

Write to us if:

• You want us to stop crawling you, entirely or in part.
• You want us to change the attribution or the licence we credit you with.
• There is a mistake in your events as we show them.
• You have a better feed than the one we are using (it happens often and we are always grateful for the tip).
• You simply want to know what we hold of yours.

We answer. And if you ask us to stop, we stop: it does not have to be a formal complaint.

Our rules, in four lines

1. We identify ourselves with our real name and a link to this page.
2. We ask for less than you would let us ask for.
3. We do not touch personal data, not even when the feed offers it to us.
4. We credit whoever gives us the data, with their licence and their link.