• Spec version: walsh-research-compliance/v1.3 (2026-05-24). Supersedes v1.2.
• Bot identity governed: Walsh-Research/MAJOR.MINOR (currently 1.2).
• Applies to: every tool that fetches third-party resources under the Walsh-Research identity — crawlers, agents, one-off scripts — in any language.
• Reference implementations: the Clojure crawler jwalsh/tech-crawler and a Python conformance harness (see 15.1). References, not the contract; this document is the contract.
• Authoritative public surface: https://wal.sh/bot/ and the /.well-known/walsh-research/ documents (5).

This is the single, authoritative specification for the Walsh-Research bot contract. It is written to be build-complete: a competent implementer (human or LLM) given only this document — and the URLs it names — can produce a conformant Walsh-Research/x.y tool in any language, then verify it against the 12 and the live 13. It specifies behavior and data contracts, never a particular implementation.

A note on what "conformant" means: the conformant entity is the tool a publisher runs under the Walsh-Research/1.2 identity, bound to this contract. A general-purpose LLM that fetches a URL while reasoning is not a Walsh-Research bot, and generating code that could be such a tool does not make the generator one. Conformance is a property of the bound, deployed tool.

1. Read 3, 4, and the 6 table.
2. Implement the 7 pipeline and requirements R1–R12. Each states its inputs, outputs, ordering, edge cases, and exact constants — everything needed to implement it with no further reference.
3. Verify against the 12 (offline input -> expected pairs) and the live 13 (URLs you can fetch).
4. Self-audit with the 15; optionally emit an 14 document.
5. Follow the 18 for a suggested order of work.

A note on the code. Any code in this document is Clojure, shown for demonstration only (Clojure is the reference implementation's language). It is illustrative, not prescriptive: conformance is defined by the behavior, contracts, constants, and 12 here, never by these snippets. The compliance implementation team chooses its own language and design; the 20 map the handful of primitives across languages.

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are used as in RFC 2119 / RFC 8174. A tool is conformant with a spec version when it satisfies every MUST and MUST NOT of that version. SHOULD-level items are expected; their absence must be deliberate and justifiable.

"A tool" means the software acting under the Walsh-Research identity. "A host" means a registered domain name plus optional port. "A target" means a single URL the tool has been explicitly configured to fetch.

Three independently-versioned contracts:

• Bot product version — Walsh-Research/MAJOR.MINOR, carried in the User-Agent (R1). It identifies the deployed bot line, not the spec version. robots.txt matching uses only the bare product token Walsh-Research (R2), never the version.
• Spec version — walsh-research-compliance/vN[.M]. A breaking change to a MUST bumps the major; additive MUSTs / promoted SHOULDs bump the minor.
• Data contracts — walsh-research-blocklist/vN, walsh-research-test-fixtures/vN, walsh-research-attestation/vN, each versioned on its own.

A conformant tool MUST be able to state which spec version it targets (e.g. in its 14). A tool targeting an older spec version remains conformant against that version. The product version and the spec version move independently: a tool may bump its product version (Walsh-Research/1.2) or its targeted spec revision without the two being locked together, and a future Walsh-Research/1.4 could still target walsh-research-compliance/v1.3.

A tool MAY additionally annotate the spec version it targets out of band in its attestation document. A tool MAY NOT alter the exact R1 User-Agent to carry it (R1 is an exact-match MUST); any extra UA comment token MUST NOT affect the Walsh-Research robots token used for matching.

A tool MUST treat these URLs as the live source of truth (no bundled copy may override a successfully fetched live document):

The blocklist document is self-describing: it carries its own JSON Schema inline under a top-level schema key (R3c), so a tool can fetch one URL and validate it without a second request. The standalone blocklist.schema.json remains the canonical copy (for external tooling and as the schema's $id) and is identical.

From https://wal.sh/bot/ alone a tool can discover this spec; from the spec, the blocklist (schema included) and fixtures; from the fixtures, every URL needed to self-verify. The whole bootstrap is reachable from one starting URL.

The blocklist and fixtures contracts and the schema are published under no restrictions and may be reused by other operators.

These are the only tunables. Values are defaults a tool SHOULD use unless a fetched document dictates otherwise (e.g. refresh, Crawl-delay, Retry-After).

For every target URL, a tool MUST evaluate the gates below in this exact order. The first gate that returns DENY stops the request; no later gate or fetch runs. The order is normative and non-commutative: R3 precedes R2 because an operator opt-out is absolute and may apply to a host whose robots.txt is itself unreachable; R4 follows both because pacing a request we will deny wastes wall clock.

// Walsh-Research pre-request gate: blocklist -> robots.txt -> throttle -> fetch
digraph pre_request_gate {
    rankdir=TB;
    bgcolor=white;
    node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10];
    edge [color="#888888", fontcolor="#555555", fontname="Helvetica", fontsize=9];

    url [label="Target URL", fillcolor="#dbeafe", color="#1d4ed8", fontcolor="#1d4ed8"];

    // Gate 1: Operator blocklist (R3)
    subgraph cluster_r3 {
        label="R3: Operator Blocklist"; style="rounded"; color="#b91c1c"; fontcolor="#b91c1c";
        fontname="Helvetica"; fontsize=11;
        blocklist [label="Fetch blocklist.json\n(cached, PT6H refresh)",
                   fillcolor="#fee2e2", color="#b91c1c", fontcolor="#b91c1c"];
        bl_check [label="Domain in\nblocked[]?",
                  shape=diamond, fillcolor="#fee2e2", color="#b91c1c", fontcolor="#b91c1c"];
    }

    // Gate 2: robots.txt (R2)
    subgraph cluster_r2 {
        label="R2: robots.txt (RFC 9309)"; style="rounded"; color="#6b21a8"; fontcolor="#6b21a8";
        fontname="Helvetica"; fontsize=11;
        robots [label="Fetch /robots.txt\n(cached, 24h TTL)",
                fillcolor="#ede9fe", color="#6b21a8", fontcolor="#6b21a8"];
        group [label="Select group:\nWalsh-Research > *",
               fillcolor="#ede9fe", color="#6b21a8", fontcolor="#6b21a8"];
        disallow [label="Path\ndisallowed?",
                  shape=diamond, fillcolor="#ede9fe", color="#6b21a8", fontcolor="#6b21a8"];
    }

    // Gate 3: Rate limit (R4)
    subgraph cluster_r4 {
        label="R4: Rate Limit"; style="rounded"; color="#b45309"; fontcolor="#b45309";
        fontname="Helvetica"; fontsize=11;
        throttle [label="Throttle\nmax(1s, Crawl-delay)",
                  fillcolor="#fef3c7", color="#b45309", fontcolor="#b45309"];
    }

    // Outcomes
    fetch [label="FETCH\n(with backoff R5)",
           fillcolor="#dcfce7", color="#15803d", fontcolor="#15803d"];
    deny [label="DENY\n(skip URL)",
          fillcolor="#fee2e2", color="#b91c1c", fontcolor="#b91c1c"];

    // Flow
    url -> blocklist;
    blocklist -> bl_check;
    bl_check -> deny [label="yes", color="#b91c1c", fontcolor="#b91c1c"];
    bl_check -> robots [label="no"];
    robots -> group;
    group -> disallow;
    disallow -> deny [label="yes", color="#b91c1c", fontcolor="#b91c1c"];
    disallow -> throttle [label="no"];
    throttle -> fetch;
}

;; Demo (Clojure). The gate order is normative; the language is not.
(defn may-fetch? [url]
  (cond
    (operator-blocklist-blocks? url) :deny   ; R3, checked FIRST
    (not (robots-allows? url))       :deny   ; R2, RFC 9309
    :else (do (throttle! url)                ; R4, blocks until the slot is free
              :allow)))

If may_fetch returns ALLOW the tool issues the request with conditional headers (R7) and applies backoff (R5) to rate-limit responses.

The same lifecycle as a sequence over time — one target URL, the gates in order, and the participants each step talks to:

// Walsh-Research request lifecycle sequence (R3 -> R2 -> R4 -> fetch -> R5/R7)
digraph request_sequence {
    rankdir=LR;
    graph [bgcolor="white", fontname="Helvetica", fontsize=11,
           pad="0.3", nodesep="0.4", ranksep="0.8"];
    node  [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10];
    edge  [fontname="Helvetica", fontsize=9, color="#888888"];

    bot       [label="Walsh-Research\nbot",      fillcolor="#dbeafe", color="#1d4ed8", fontcolor="#1d4ed8"];
    blocklist [label="wal.sh\nblocklist.json",   fillcolor="#fee2e2", color="#b91c1c", fontcolor="#b91c1c"];
    robots    [label="target host\n/robots.txt", fillcolor="#ede9fe", color="#6b21a8", fontcolor="#6b21a8"];
    target    [label="target\nURL",              fillcolor="#dcfce7", color="#15803d", fontcolor="#15803d"];

    // R3: operator blocklist, checked first
    bot -> blocklist [label="1. GET blocklist.json\n(cached PT6H)", color="#b91c1c"];
    blocklist -> bot [label="2. list + inline schema\nvalidate; host blocked?", style=dashed, color="#b91c1c"];

    // R2: robots.txt (RFC 9309)
    bot -> robots [label="3. GET /robots.txt\n(cached 24h)", color="#6b21a8"];
    robots -> bot [label="4. select Walsh-Research\ngroup; longest-match\nallow/deny", style=dashed, color="#6b21a8"];

    // R4 throttle, conditional fetch (R7), backoff (R5)
    bot -> target [label="5. throttle max(1s, Crawl-delay)\nthen GET (If-None-Match /\nIf-Modified-Since)", color="#b45309"];
    target -> bot [label="6. 200 +ETag / 304\n(429/503 -> R5 backoff)", style=dashed, color="#15803d"];

    note1 [label="R3 before R2:\nopt-out is absolute,\nhonored even if\nrobots is unreachable",
           shape=note, fillcolor="#fef3c7", color="#b45309", fontcolor="#b45309", fontsize=9, style="filled"];
    note2 [label="metadata only;\nno recursion,\nno sub-resources (R6)",
           shape=note, fillcolor="#dcfce7", color="#15803d", fontcolor="#15803d", fontsize=9, style="filled"];

    { rank=same; blocklist; note1; }
    { rank=same; target; note2; }
    note1 -> blocklist [style=dotted, color="#b45309", arrowhead=none, constraint=false];
    note2 -> target    [style=dotted, color="#15803d", arrowhead=none, constraint=false];
}

// Walsh-Research backoff state machine (R5): retry on 429/503
digraph backoff_states {
    rankdir=LR;
    bgcolor=white;
    node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10];
    edge [color="#888888", fontcolor="#555555", fontname="Helvetica", fontsize=9];

    fetch [label="Fetch URL",
           fillcolor="#dbeafe", color="#1d4ed8", fontcolor="#1d4ed8"];
    check [label="Status?",
           shape=diamond, fillcolor="#fef3c7", color="#b45309", fontcolor="#b45309"];
    success [label="200/304\nProcess",
             fillcolor="#dcfce7", color="#15803d", fontcolor="#15803d"];
    wait [label="Wait\nRetry-After or\n2^n x base x rand",
          fillcolor="#fef9c3", color="#a16207", fontcolor="#a16207"];
    retry_check [label="n < max?",
                 shape=diamond, fillcolor="#fef3c7", color="#b45309", fontcolor="#b45309"];
    abort [label="Abort\nlog + wait 1h",
           fillcolor="#fee2e2", color="#b91c1c", fontcolor="#b91c1c"];

    fetch -> check;
    check -> success [label="2xx/3xx"];
    check -> wait [label="429/503"];
    check -> abort [label="4xx/5xx\n(other)"];
    wait -> retry_check;
    retry_check -> fetch [label="yes\nn++"];
    retry_check -> abort [label="no\n(max retries)"];
}

These input -> expected pairs are normative and implementation-free; a conformant tool MUST reproduce every expected output. They double as a portable offline suite.

Live URLs a tool can fetch to self-verify the gates end-to-end. They are enumerated machine-readably in test-fixtures.json (contract walsh-research-test-fixtures/v1):

{
  "contract": "walsh-research-test-fixtures/v1",
  "updated":  "2026-05-23T00:00:00Z",
  "fixtures": {
    "robots_allowed_sentinel": "https://wal.sh/research/bots/dogfood-allow",
    "robots_disallowed":       "https://wal.sh/research/bots/dogfood-disallow",
    "walsh_only_allowed":      "https://wal.sh/research/bots/dogfood-walsh-only",
    "external_positive":       "https://pypi.org/project/jsonschema/",
    "operator_blocked_host":   "example.com",
    "flaky_429_endpoint":      null
  },
  "notes": {
    "external_positive": "PyPI project page (robots-allowed). Do NOT use /pypi/<name>/json: pypi.org Disallows /pypi/*/json, so a robots-respecting bot must refuse it.",
    "flaky_429_endpoint": "no live R5 fixture; tools SHOULD use a local mock"
  }
}

Verify these live (same site): the fixtures index test-fixtures.json, the canaries dogfood-allow / dogfood-disallow / dogfood-walsh-only, the self-describing blocklist.json and its blocklist.schema.json, and robots.txt.

The external positive is pypi.org/project/jsonschema/ because PyPI is reachable from inside the sandboxes LLM harnesses run in (the package index is allowlisted almost universally); a self-test target must work there or the spec cannot be self-verified from a single-prompt bootstrap. Note the correction: the obvious pypi.org/pypi/<name>/json API path is robots-disallowed (Disallow: /pypi/*/json), so a conformant bot must refuse it – the project page is the correct allowed positive. No live 429 fixture is provided — R5 is verified against a local mock.

A tool MAY publish a self-attestation so publishers and auditors can compare implementations. It is not consulted by any gate — robots.txt and the blocklist already decide routing — it is ecosystem hygiene only.

Contract walsh-research-attestation/v1:

{
  "contract":     "walsh-research-attestation/v1",
  "tool":         "<implementer-chosen name>",
  "tool_version": "<semver or freeform>",
  "spec":         "walsh-research-compliance/v1.3",
  "user_agent":   "Mozilla/5.0 (compatible; Walsh-Research/1.2; +https://wal.sh/bot/)",
  "requirements": { "R1": true, "R2": true, "R3": true, "...": true }
}

The publishing location is the implementer's choice (a URL on the operator's site, a release artifact). The spec defines the document shape, not where it lives. Signing (e.g. Ed25519 over the JSON canonical form, keys at a published JWKS) is a future option, deferred until a consumer needs it.

A tool is conformant when every MUST holds.

We hold our own tools to this spec against the live site. Three sibling pages under /research/bots/ exercise the R2 gate in all directions at one path depth:

User-agent: *
Disallow: /research/bots/dogfood-walsh-only

User-agent: Walsh-Research
Disallow: /research/bots/dogfood-disallow
Allow:    /research/bots/dogfood-allow
Allow:    /research/bots/dogfood-walsh-only
Crawl-delay: 2

• dogfood-disallow proves we honor Disallow across every content-negotiated representation (.html, .md, extensionless): the decision is made from the path before any fetch.
• dogfood-allow proves we do not over-block an allowed sibling.
• dogfood-walsh-only proves the other direction of R2a: a generic * bot is disallowed, but our named group selects past * and is permitted, so we fetch. It doubles as a honeypot: since * is disallowed, the only legitimate fetchers are humans (browsers ignore robots.txt) and Walsh-Research, so any other agent in the access logs is a robots-violating bot.

These pages are intentionally undocumented elsewhere — they exist only to be crawled or refused by a conformance run. A conformant tool SHOULD assert all three outcomes against the live robots.txt as an attestation.

1. A site self-serves via robots.txt (R2) at any time — honored within the robots cache TTL.
2. Or emails j@wal.sh; the operator appends a {domain, added, reason} entry to blocklist.json and deploys — honored within the refresh TTL.

A tool implementer's only job is to consume both sources correctly.

A suggested order to build a Walsh-Research/x.y tool from this document alone:

1. Primitives. HTTP client, XML+JSON parser, monotonic clock + sleep, SHA-256. (20 maps these per language.)
2. R1. Hard-code USER_AGENT; send it on every request.
3. R2 parser. Groups -> selection -> path matching; pass the robots 12; cache per host (ROBOTS_TTL_SECONDS).
4. R3 + R10. Fetch schema (cache SCHEMA_TTL_SECONDS, stale-on-failure), validate the blocklist, cache it (refresh), domain matching, retain-on-failure; pass the blocklist vectors.
5. Gate. Wire may_fetch in the normative order: blocklist -> robots -> throttle.
6. R4 + R5. Throttle and backoff; pass the Retry-After / duration vectors.
7. R7 + R11. Conditional headers, canonical-URL dedup, persistent caches.
8. R9. Feeds; markdown content negotiation and /llms.txt fallbacks; generic link extraction. Never per-site selectors.
9. Attest. Run against the live 13 and the 16; emit an 14 recording the spec version targeted.

A tool that reproduces every 12 block, passes the live fixtures and all three dogfood canaries, with the gate in the normative order, is conformant with walsh-research-compliance/v1.3.

• Content-Signal (draft-romm-aipref-contentsignals; ai-train, search, ai-input, advertised in wal.sh/robots.txt) is orthogonal to the four primitives and binds only a pipeline that performs the named use. The current bot is a metadata extractor that does not feed an LLM, so ai-input is not binding. A future bot version that adds LLM summarization would make ai-input binding on its intake; defining which pipeline operations count is a v2 item.
• Attestation signing and a keys endpoint — deferred until a consumer needs verifiable attestations.
• Cross-process throttle backend (Redis / SQLite token bucket) — deferred; single-instance scheduling is the v1.2 default (R4a).
• Redirect policy (C6) — 3xx handling is undefined. Following a redirect can leave robots scope (/research/aoc/src redirects to /research/aoc/src/ which may have different robots rules). Proposed for v1.4: re-run the Location header value through the full gate pipeline (R3→R2→R4) before fetching the redirected URL. Current behavior (transparent redirect via the HTTP library) is acceptable for v1.2/v1.3 but should be tightened.
• Transport error retry (C5) — R5 covers 429/503 but is silent on connection/TLS/timeout failures. Proposed for v1.4: apply the same backoff policy to transport errors. Surfaced by 6+ implementations in the compliance harness.
• R13×R2 interaction (C1) — R13 appends query parameters; R2 path matching includes query. Resolution adopted by all 29 implementations: gates evaluate the untagged URL; R13 params are appended only at HTTP transmission. Proposed for v1.4: codify this as normative.

Nothing here depends on a language. Each requirement is behavior over four primitives every language has; only the primitives change.

• RFC 9309 — Robots Exclusion Protocol (robots.txt).
• RFC 8615 — Well-Known URIs (/.well-known/).
• RFC 9110 — HTTP Semantics (User-Agent §10.1.5; Retry-After §10.2.3).
• RFC 2119 / RFC 8174 — requirement keywords.
• draft-romm-aipref-contentsignals / contentsignals.org — Content-Signal.
• llmstxt.org — the llms.txt convention (Howard, 2024).

• v1 (2026-05-23) — initial: R1 UA, R2 robots/RFC 9309 + Crawl-delay, R3 blocklist + schema, R4 rate limit, R5 backoff, R6 scope, R7 conditional fetch + dedup, R8 frequency; rev 2 added R9 (structured formats); rev 3 added the dogfood canary pair.
• v1.1 (2026-05-23) — make the spec a standalone, single-prompt-bootstrappable build target. Folds in v1 rev 1–3. Tightens R2 to full RFC 9309 path matching (Allow precedence + *=/=$, with a safe over-block allowance, R2b/R2c) and longest-match group selection (R2a). Promotes schema validation to MUST (R3c). Adds R10 (schema cache + stale-on-failure), R11 (persistent caches), R12 (cache-miss behavior). Pins exact algorithms for R3 domain matching, R5 Retry-After, R7 canonical URL, and durations, plus a single 6 table. Adds 12, live 13 (test-fixtures.json) with the dogfood-walsh-only bidirectional canary and a sandbox-reachable external positive, an 14 contract, a 18, and the 4 model. Code examples are Clojure (the reference language) shown for demonstration only; conformance is defined by behavior, contracts, and test vectors, and is language-agnostic.
• v1.2 (2026-05-24) — bump bot identity from Walsh-Research/1.1 to Walsh-Research/1.2; update USER_AGENT constant, attestation example, and all normative references accordingly. Add R13 (implementation tagging). No other behavioral requirements changed.
• v1.3 (2026-05-24) — add R13 (implementation tagging, SHOULD). Add bot compliance lab with 19 per-bot canary pages at /research/bots/lab/disallow-{token}/ for empirical robots.txt compliance verification. Seven implementations across six languages (TypeScript, Clojure, Python, Elisp, Guile, JavaScript) updated to v1.2 bot identity and v1.3 spec references.