SitemappedBot

Sitemapped: the web's sites, indexed for agents. SitemappedBot is the crawler behind it.

What it does

Sitemapped is an index of websites, not pages. For each site it records what the site is, which topics it covers, where its machine-readable entry points are (sitemaps, feeds, search endpoints, llms.txt), and what its robots.txt and content signals allow. AI agents use it to decide which sites to read. It gives them pointers back to your site and never a copy of your content.

Sitemapped is currently in a research phase, visiting a sample of tens of thousands of sites a few times over several weeks. Request volumes are small.

What it fetches

On each visit SitemappedBot makes at most 12 requests to a site, using GET and HEAD only:

• /robots.txt, read first and obeyed
• the homepage
• sitemaps listed in robots.txt, or /sitemap.xml and /sitemap_index.xml
• /llms.txt, plus a HEAD request for /llms-full.txt
• one feed and one OpenSearch description, if your homepage links to them
• up to three ordinary pages taken from your sitemap

It never submits forms, never follows query-string URLs, and does not run JavaScript.

How it behaves

• Obeys robots.txt (RFC 9309) for the SitemappedBot token, or for * if no group names it. If robots.txt returns a server error or cannot be reached, it treats the whole site as disallowed and tries again later.
• Makes one connection per host and waits at least one second between requests, or longer if you set Crawl-delay.
• Honours Retry-After, backs off on 429 and 503, and does not retry after 403.
• Never bypasses a challenge. It does not solve CAPTCHAs or JavaScript challenges and does not use headless browsers, residential proxies or a disguised user agent. If it meets a challenge, it records that and leaves.

Content signals

SitemappedBot reads Content-Signal and Content-Usage lines in robots.txt.

• search=no: your site is not indexed.
• ai-input=no: only structural metadata (URL paths, sitemap and feed structure, schema.org types) is used to classify your site. No page text is sent to a model.
• We never train or fine-tune a model on your text unless you set ai-train=yes.
• We never reproduce your text, whatever the signals say. The only description we publish is our own one- or two-sentence summary of what the site is.

What it stores

• your robots.txt, sitemap URL lists, and hashes used to notice changes
• structural metadata: feeds, sitemaps, languages, schema.org types, HTTP status
• our own short descriptor and topic labels for the site

Page HTML is deleted once the metadata has been extracted. Email addresses and phone numbers are removed before anything is stored or sent to a model. We record organisations, never individuals: a personal site is labelled personal, without a name.

How to opt out

Block it in robots.txt:

User-agent: SitemappedBot
Disallow: /

Or email bot@sitemapped.app with the domain. We will add it to our opt-out list, which means it is never fetched again and is removed from everything we publish.

Verifying it is us

Real SitemappedBot requests send this user agent:

SitemappedBot/<version> (+https://sitemapped.app/bot)

and come only from the host below, whose reverse DNS resolves to a sitemapped.app name that resolves back to the same address:

A request from anywhere else that claims to be SitemappedBot is not us.

Who runs it

Captivated Ltd, a company registered in England and Wales (company number 15936109). Contact: bot@sitemapped.app.