Robots Policy Change Record
A dated, hash-referenced record of what our frozen archive observed in one publisher hostname's robots.txt, including the crawler directives that changed when both selected bodies can be verified.
Counted supply check: the existing archive reader returned 55 stored observations for one in-universe hostname and re-hashed both bodies around the selected change.
A robots.txt file shows the rules served today. It does not carry its own prior versions, so a policy question raised months later often starts with screenshots, ticket notes, and recollections that disagree on the date. The useful missing artifact is narrower: on these dates our collector retrieved this source, these were the stored content hashes, and this is the point where one stored body stopped matching the prior stored body.
This page tests demand for a private Robots Policy Change Record. The fixed scope is one hostname, robots.txt only, and either one observed change or one date window already present in our archive. The proposed deliverable is a matching PDF and CSV plus a SHA-256 manifest. The planned price is $249 one time.
The archive uses a frozen domain universe, and collection coverage is not uniform across every date. We therefore check the hostname, requested dates, retained bodies, re-hashes, and custody rows before accepting a scope. No payment is accepted on this page, no checkout is opened, and nothing is scheduled automatically. A request is a coverage question, not an order.
A robots.txt change record proves what our archive observed, not what a crawler did. RFC 9309 also states that robots.txt rules are not a form of access authorization. The record is an archive observation, not a permission or conduct determination.
Start with archive coverage
$249 one time — proposed
Send one hostname and the date or date window that matters. The request is stored for a human coverage review. It does not create an order, authorize a charge, or put work on a calendar. We first tell you which observations exist and which fields, if any, would be UNKNOWN.
What the proposed record contains
The private record would restate the hostname and requested window exactly as supplied. It would then carry every stored robots.txt observation in scope and preserve the archive state instead of replacing it with a new fetch. The PDF and CSV would contain the same rows:
- Each stored observation date, source URL, UTC retrieval time, and HTTP or transport result state.
- The stored SHA-256 content hash when a body was retained.
- The selected before-and-after hashes across an observed body transition.
- The changed crawler directives when both selected bodies are available and each re-hashes to its recorded value.
- The collection-run batch hash for each selected observation.
- A per-day custody disclosure stating whether all readable store rows for that date were included in the batch commitment.
- An explicit UNKNOWN on every field that the retained source or custody record cannot support.
The separate SHA-256 manifest would cover the delivered PDF and CSV so a recipient can later check whether those delivered files changed. It does not turn our internal observation into an independent timestamp or public certification.
A real archive unit behind the offer
For flickr.com, the existing evidence-pack reader returned 55 stored robots.txt observations and selected an observed body change dated 2026-06-30. It re-hashed both retained bodies to their recorded SHA-256 values and reported no custody gap for either selected date. Those are counted results from that one supply check, not a promise that another hostname or date window will have the same coverage.
The public sample shows the evidence shape and also discloses archive limits. The offer page does not publish stored raw robots.txt bodies. A buyer's private record would quote only the directives needed to explain the selected change, and only when the underlying retained bytes pass the re-hash check.
How coverage is confirmed
- Hostname. We check whether the exact hostname is inside the frozen archive universe. An arbitrary domain is never promised.
- Window. We list the stored dates inside the requested window. A missing calendar date stays a gap; it is not converted into stable policy.
- Selected bodies. If directive-level comparison is requested, both retained bodies must exist and independently re-hash to the stored values.
- Custody. The selected dates must state whether readable store rows and batch-committed rows agree. Any exception stays beside the affected observation.
If the hostname or date is outside coverage, a selected body is unavailable, a re-hash disagrees, or a selected date has a custody gap, the affected field is labeled UNKNOWN. We decline any scope whose important answer would be unsupported before taking payment.
What the record proves—and what it does not
The bounded statement is: retrieval of a named robots.txt URL on the printed dates produced the recorded states and, where content was retained, the printed hashes. Between two selected observations, the stored content hash changed. Where both bodies were retained and re-hashed correctly, the named crawler directives differ as shown.
A body-hash transition is not automatically a crawler-rule change. Whitespace, comments, line order, or unrelated directives can change bytes without changing the rule being investigated. A first observation is a baseline, not proof of a change. A missing date is an observation gap, not evidence that the file stayed the same. HTTP 4xx, 5xx, and transport failures are recorded as result states; none is substituted for body content.
This record is not an affidavit, notarization, certified public record, independent timestamp authority, or legal opinion. It is not a copyright or licensing conclusion and not a determination that access was permitted. It does not prove a crawler fetched, read, honored, or violated a rule; it does not prove publisher intent or that any service behavior changed.
Protocol reference: RFC 9309, Robots Exclusion Protocol. The standard's scope is why this product records observable file states without turning them into permission or enforcement claims.
What to send
- One hostname, exactly as it appears in the robots.txt URL.
- The date, approximate change date, or start and end of the date window.
- Whether you need one selected transition or every stored observation in the window.
- Optionally, the crawler token or directive that matters to your review.
Do not send privileged matter details, account credentials, server logs, or confidential strategy. The scope check needs only the public hostname and date question.
What happens next
- The request is stored. It is a demand signal and coverage question, not an order.
- A human checks the archive. The scope answer names the observations available and any UNKNOWN fields.
- The scope is supportable or declined. The request itself opens no checkout and schedules no work. If the proposed product becomes supportable, any later offer would state its actual boundary before payment.
If you have one hostname and one dated robots.txt question, begin with coverage. The useful answer is either a precise description of the record the archive can support or a direct no before anyone pays.