About our crawler

We visit public business websites to work out what kind of creative work a business does. Here's exactly how it behaves.

How to identify it

It sends this user agent string:

KilnBot/0.1 (+https://makersmap.co.uk/about/bot; contact [email protected])

What it does

• Reads and honours robots.txt before fetching anything.
• Fetches your home page, and sometimes your contact or about page. That is the whole visit — two requests, once.
• Waits at least half a second between those two requests, and longer if your robots.txt sets a Crawl-delay — we use whichever is greater.
• Keeps a short text extract, your council area, and whether the site displays a company registration number. Not the whole page.
• Deletes stored text after 90 days.

What it doesn't do

• Submit forms, or collect email addresses for marketing.
• Ignore robots.txt, or crawl behind a login.
• Crawl your whole site, or come back repeatedly. Two pages, and then we leave you alone.

If your site is behind a CDN — Cloudflare, or anything your host puts in front of it — our two requests almost certainly never reach your server at all.

To block it

Add this to your robots.txt and we'll stop:

User-agent: KilnBot
Disallow: /

Or ask us to remove your listing and we won't come back.