Crawler information
If you found this page in your server logs, the sections below tell you exactly what our crawler does and how to limit or stop it.
Identity
| Operator | Sunshine Archive LLC, a Florida limited liability company |
|---|---|
| Purpose | Archiving — periodic snapshots of public government web pages for historical preservation |
| User-agent | SunshineArchive/0.1 (+https://sunshinearchive.org/bot) appended to a standard Chrome user-agent string |
| Signature | Requests are signed with Web Bot Auth (RFC 9421, Ed25519). Our key directory is at /.well-known/http-message-signatures-directory |
| Contact | social@sunshinearchive.org |
What it does
The crawler requests the page, renders it, extracts the visible text, and stores a hash. Most visits end there. A full snapshot is only taken when the text has actually changed since the previous visit — that is the entire point of the design, and it means we request your pages far more often than we store them.
What it never does
| No credentials | Every request is anonymous and unauthenticated. We never log in, never create accounts, and never use cookies or session tokens carried between sites. A page behind a login is recorded as it appears to the public, and the visit stops there. |
|---|---|
| Read only | We only make requests that read. We never submit anything that files, pays, books, registers, sends, or otherwise changes the state of your systems. Where we retrieve a public record through a search or lookup interface, it is a read of information you already publish — never a write. |
| No evasion | We do not rotate user-agents, mimic other crawlers, or work around bot protection. If you block us, we stay blocked and record that fact. |
| No model training | The archive is not used to train machine learning models, and its contents are not sold as a training corpus. |
| No probing | We do not scan for vulnerabilities, enumerate identifiers, or fish for pages that are not linked, listed in a sitemap, or reachable through a public index your site provides. |
Request rate
We check a page at most once per day, and typically far less often — pages that rarely change
are checked less frequently over time. Requests to a single host are spaced by at least one
second, and by longer if your robots.txt asks for it.
Controlling the crawler
We obey robots.txt, including Crawl-delay. To slow us down:
User-agent: SunshineArchive Crawl-delay: 10
To exclude part of your site:
User-agent: SunshineArchive Disallow: /private/
To stop us entirely:
User-agent: SunshineArchive Disallow: /
Changes take effect within 24 hours. A disallowed page is not fetched; we keep a record that the source exists and that it asked not to be crawled, and nothing else.
Removal requests
If material has already been archived and you believe it should not be public — for example, a page that briefly exposed personal information — write to social@sunshinearchive.org with the URL. We review every request and may remove or restrict material at our discretion. We would much rather hear from you than have you block us.
Why this exists
Plenty of services will save you a copy of a web page. Almost nothing systematically records how a government’s web presence changes — when guidance is quietly rewritten, when a program page disappears, when eligibility language shifts without a formal rulemaking. Those changes are public records in every meaningful sense, and they are routinely lost.