From bf44a62703c3cec4132dc0c8ab94f555a145ae5c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 16:59:22 +0000 Subject: [PATCH 1/5] =?UTF-8?q?docs(proofs):=20PROOF-NEEDS=20is=20now=20a?= =?UTF-8?q?=20PROOF-POLICY=20=C2=A75=20proof-status=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The file was the pre-policy template of 2026-03-29: a wish list with no statuses, no census and a "Lean4 for algebraic properties" recommendation that is not policy. The mathesis census names it as gitbot-fleet's status document, so it has to say what is and is not established. Now, measured against origin/main 7907099 on 2026-11-17: - Census: 15 rows; 0 proven, 4 argued, 4 tested-only, 7 open. - The four echidnabot theory files (ReceiptBoundary agda/lean/v, FixtureIdentity) are argued: byte-identical to the inputs of echidnabot's kernel receipts, but not re-checked here. echidnabot's PROOF-STATUS governs. - The three echidnabot ABI files are open (template placeholders, three holes in Layout.idr). - Four tested-only rows for the report-back producer (envelopes never clean on failure, slug-only member names, the fixer's exit-0-only-when-wired rule, build failures never fatal), with the bounds each test covers. - The four old needs are open, mapped to PROOF-POLICY §1 (Creusot with Pearlite and Verus for Rust code, Idris2 for the cross-bot contract, Agda rechecked in Rocq and Lean only if stated as theory). - The prover fixtures (admitted_stub.v, sorry_stub.lean, trivial_ok.*) are listed as not obligations. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDjqj5DHRRCrDiGeTmq --- PROOF-NEEDS.adoc | 360 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 333 insertions(+), 27 deletions(-) diff --git a/PROOF-NEEDS.adoc b/PROOF-NEEDS.adoc index 99b67ce8..651a575c 100644 --- a/PROOF-NEEDS.adoc +++ b/PROOF-NEEDS.adoc @@ -1,40 +1,346 @@ -== Proof Requirements +// SPDX-License-Identifier: CC-BY-SA-4.0 +// SPDX-FileCopyrightText: 2025-2026 Jonathan D.A. Jewell += Proof status: gitbot-fleet +:toc: +:toc-placement: preamble -=== Current state +This is the proof-status file of `hyperpolymath/gitbot-fleet` in the sense of +`3-practice/PROOF-POLICY.adoc` §5 (hyperpolymath/standards). It replaces the +pre-policy template of 2026-03-29. -* ABI directory exists (template-level) -* No dangerous patterns found -* 100K lines; bot fleet for repository quality enforcement -* Claims: "`formal mathematical/statistical verification`" +* *Date*: 2026-11-17. +* *Measured against*: gitbot-fleet `origin/main` at + `7907099b4e6912059ad71a4897005463d214f0bf`. +* *Tokens*: the closed vocabulary of PROOF-POLICY §5: `proven`, + `proven-with-postulates`, `tested-only`, `argued`, `open`. +* *Prover for each obligation*: PROOF-POLICY §1. Rust code is verified with + Creusot (Pearlite contracts) and Verus by default, on the code itself (§1.2); + APIs, ABIs and engineering properties with Idris2; Ada with SPARK; theory + with Agda, rechecked in Rocq and Lean. -=== What needs proving +== Census -* *Bot action safety*: Prove that automated bot actions (PRs, issues, -fixes) are reversible and do not destroy repository state -* *Confidence threshold correctness*: Prove that the -robot-repo-automaton confidence thresholds correctly gate automated -merges (no auto-merge below threshold) -* *Rule conflict resolution*: Prove that when multiple bots (rhodibot, -echidnabot, sustainabot, etc.) produce conflicting recommendations, the +15 rows: 0 `proven`, 0 `proven-with-postulates`, 4 `tested-only`, 4 `argued`, +7 `open`. + +No row is `proven` here. The four `argued` rows are echidnabot's theory +files, whose kernel receipts were taken in the standalone echidnabot +repository, not in this one (see the trust-boundary column). No checker +(Agda, Idris2, Lean, Rocq) was available when this file was written, so no +proof file in this repository was checked by the author of this file. + +== Ledger: proof files tracked in this repository + +All eleven tracked proof files sit under `bots/echidnabot/`, a copy of +`hyperpolymath/echidnabot`. Seven of them are obligations (rows below); four +are prover fixtures (next section). + +The authoritative status of every `bots/echidnabot/` row lives in +`docs/PROOF-STATUS.adoc` of the standalone echidnabot repository, read at +echidnabot `origin/main` `7a5b8c48ffef8236b06b7174d855d88124b7f380`. The copy +in this repository does not carry that file. When the two disagree, echidnabot's +file governs and this one is wrong. + +[cols="2,1,2,2,3",options="header"] +|=== +|Obligation |Status |Checker (version, flags) |Where |Trust boundary + +|T-RECEIPT: a general evidence token does not imply its claim +(`no-unconditional-reflection`, `receive-with-soundness`) +|`argued` +|None run here. echidnabot's receipt R-AGDA: Agda 2.8.0.2, `--safe --without-K +--cubical-compatible --exact-split --no-sized-types --no-guardedness +--no-libraries --no-default-libraries --ignore-interfaces`, session receipt of +2026-11-09 at echidnabot `1443adfabb98a9e0daea562bc8dbe764442c14cc` +|`bots/echidnabot/proofs/agda/ReceiptBoundary.agda` +|The file here is byte-identical to the file that receipt hashed (sha256 +`276ee4b5...0c9014`, compared on 2026-11-17). Echidnabot records it as +`proven`, with no postulate in the import cone. This repository re-ran +nothing, so it records `argued`. Grep for escape hatches finds none in the +file. Authoritative status: echidnabot `docs/PROOF-STATUS.adoc`, row T-RECEIPT. + +|T-RECEIPT-LEAN: Lean restatement of T-RECEIPT +|`argued` +|None run here. echidnabot receipt R-LEAN: Lean 4.35.0, `lean -o` plus +`#print axioms` on the three theorems +|`bots/echidnabot/proofs/theory/lean/ReceiptBoundary.lean` +|Byte-identical to the receipt input (sha256 `00ea31f0...28858f`). Echidnabot +records no axioms. The statement match with the Agda source is `BRIDGE-PENDING` +there: agent-drafted, awaiting owner review, so the leg does not count towards a +PWG. Dedukti transport was tried upstream and failed; this is a re-proof. +Authoritative status: echidnabot `docs/PROOF-STATUS.adoc`, row T-RECEIPT-LEAN. + +|T-RECEIPT-ROCQ: Rocq restatement of T-RECEIPT +|`argued` +|None run here. echidnabot receipt R-ROCQ: Coq 8.18.0 (not a Rocq 9 release), +`coqc -q -Q . Recheck` plus `Print Assumptions` +|`bots/echidnabot/proofs/theory/rocq/ReceiptBoundary.v` +|Byte-identical to the receipt input (sha256 `ca65d1dd...f3e66d`). Same +`BRIDGE-PENDING` state as the Lean leg. Authoritative status: echidnabot +`docs/PROOF-STATUS.adoc`, row T-RECEIPT-ROCQ. + +|Fixture identity: `(A : Set) -> A -> A`, a semantic reconstruction of the +constructive identity schema used by the Rocq and Lean prover fixtures +|`argued` +|None run here. echidnabot receipt R-AGDA (same invocation as T-RECEIPT) +|`bots/echidnabot/proofs/agda/FixtureIdentity.agda` +|Byte-identical to the receipt input (sha256 `9c7b80b1...6a91fb`). Not witness +transport: Agda `Set` and Rocq/Lean `Prop` differ, and echidnabot keeps the +correspondence `open`. Authoritative status: echidnabot `docs/PROOF-STATUS.adoc`, +section "Pre-existing proof material". + +|ABI `Types.idr`: echidnabot ABI types +|`open` +|None. Idris2 is the §1 prover for an ABI; none was run +|`bots/echidnabot/src/abi/Types.idr` +|Three `{{...}}` template placeholders remain, so the module does not describe +an instantiated echidnabot ABI. Byte-identical to echidnabot `origin/main`. +Echidnabot's PROOF-STATUS records the ABI scaffold as `open` ("ABI scaffold +debt (not verified)"). Do not read this file as verification. + +|ABI `Layout.idr`: layout contracts +|`open` +|None. Idris2 is the §1 prover; none was run +|`bots/echidnabot/src/abi/Layout.idr` +|Three unresolved holes (`?fieldsAlignedProof` at line 140, +`?exampleFieldsAligned` at line 161, `?offsetInBoundsProof` at line 178) and +two template placeholders. Echidnabot's PROOF-STATUS notes that +`offsetInBounds` quantifies over arbitrary fields and cannot be proved as +written. Holes are not to be replaced by postulates. + +|ABI `Foreign.idr`: foreign-function declarations +|`open` +|None. Idris2 is the §1 prover; none was run +|`bots/echidnabot/src/abi/Foreign.idr` +|Fourteen `{{...}}` template placeholders; the symbols are placeholders, not +exported echidnabot symbols. The only `believe_me` match is in a comment that +bans it. Echidnabot's PROOF-STATUS records the scaffold as `open`. +|=== + +=== Escape-hatch search + +Searched on 2026-11-17 with `grep` over the eleven proof files for +`postulate`, `sorry`, `Admitted`, `admit`, `believe_me`, `assert_total`, +`assert_smaller`, Agda `{! !}` holes, Idris `?name` holes, `trustMe` and +`axiom`. + +* The four obligation files in `proofs/agda/` and `proofs/theory/` contain no + escape hatch. The only `axiom` matches are `#print axioms` and "axioms or + admitted obligations" in comments of the Lean and Rocq files. +* `src/abi/Layout.idr` contains the three holes listed above. `Types.idr` and + `Foreign.idr` contain only template placeholders. +* `proofs/coq/admitted_stub.v` (`Admitted.`) and `proofs/lean/sorry_stub.lean` + (`sorry`) contain escape hatches on purpose (next section). + +== Not obligations + +These four files are prover fixtures for echidnabot's CI, test material for a +multi-prover system (PROOF-POLICY §3, last bullet). They are not rows and are +not counted in the census. Their escape hatches must stay intact. + +* `bots/echidnabot/proofs/coq/admitted_stub.v`: deliberate `Admitted.`, a + negative fixture for echidnabot's axiom scanner. +* `bots/echidnabot/proofs/lean/sorry_stub.lean`: deliberate `sorry`, a negative + fixture checked by echidnabot's `protocol_contract` test. +* `bots/echidnabot/proofs/coq/trivial_ok.v` and + `bots/echidnabot/proofs/lean/trivial_ok.lean`: trivial identity proofs used + as positive fixtures. + +The JSON files in `bots/echidnabot/proofs/test_fixtures/` are webhook inputs, +not proofs. `idris-dependents/tests/fixtures/*.ipkg` are package fixtures for +an Idris-dependents test, not proofs of this repository. + +== Ledger: report-back producer (tested-only) + +The report-back producer is the part of this repository that the Hypatia +verdict grid reads (`docs/REPORT-BACK.adoc`): `scripts/fleet-report-back.sh`, +`scripts/fleet-report-back-build.sh` and the rollout fixer +`scripts/fix-fleet-enrolment.sh`. It is shell. No Creusot, SPARK or Idris2 +artefact exists for it, so the evidence is test evidence only, and each row is +`tested-only` with the bounds the test file actually covers. + +Observed on 2026-11-17 at the commit above (pass counts are the lines each +script itself reports, with 0 failed in every run): + +* `STANDARDS_DIR= bash tests/fleet-report-back-test.sh`: + 229 passed, 0 failed for its own checks, then it runs the build-helper suite + (below) and the run totals 298 `PASS` lines and 0 `FAIL` lines. The renderer + round-trip checks ran (no `SKIP` line); the standards checkout was not + pinned to a commit. +* `bash tests/fix-fleet-enrolment-test.sh`: 139 passed, 0 failed. +* `bash tests/fleet-report-back-build-test.sh`: 69 passed, 0 failed. These 69 + are part of the 298 above, not additional to them. + +[cols="3,1,2,2,3",options="header"] +|=== +|Obligation |Status |Checker (version, flags) |Where |Bounds and trust boundary + +|R-ENVELOPE: `fleet-report-back.sh` writes one envelope per enrolled member. +A missing binary, a crash without JSON, a timeout, or an output of the wrong +shape is an `error` envelope with a reason, never a clean `ran` +(`docs/REPORT-BACK.adoc` §4, last paragraph) +|`tested-only` +|`bash tests/fleet-report-back-test.sh`, 229 checks (298 with the build suite); +bash with `jq` +|`scripts/fleet-report-back.sh` +|Bounds: the adapters are stubs that print the JSON shape of the real adapters +(taken from a live run on 2026-10-27), not the seven Rust crates. Cases: only +`enrolled: true` members run; an unreadable manifest runs nothing; missing +binary, crash without JSON and timeout are `error` with a reason; exit 1 with +valid JSON stays `ran`, any other non-zero status is `error`; empty scans, +malformed output and path normalisation; `echidnabot` and `sustainabot` are +`not_run` with a reason; an unknown member is `error`; the driver never writes +`not_applicable`; the envelope is key-sorted compact JSON with one final LF; +the scanned tree is unchanged. Not covered: real adapter behaviour, other +operating systems, other `jq` or bash versions. The grid-side properties (how +an envelope becomes a row colour) are proved in hyperpolymath/standards +`3-practice/fork-governance/proofs/ReportBack.idr`, which is being added on the +standards branch `claude/gitbot-fleet-feedback-0vdo64` and is *pending*, not +merged. They are not re-proved here. The renderer round-trip in the test +exercises the real standards renderer on these envelopes and is also +`tested-only`. + +|R-SLUG: member names are restricted to the slug `^[a-z][a-z0-9-]{0,63}$`, so a +hostile manifest key cannot steer the build or a path +|`tested-only` +|`bash tests/fleet-report-back-test.sh`; `bash tests/fleet-report-back-build-test.sh` +|`scripts/fleet-report-back.sh` (member filter, line 279); +`scripts/fleet-report-back-build.sh` (crate-name filter) +|Bounds: a 64-character slug is accepted and a 65-character one rejected, in +both planning and execution; a leading digit, uppercase, underscore and a dot +are rejected and warned about; a manifest key containing a tab and `../` +sequences, and a key with a space, are dropped with a warning and write +nothing; an explicit `--member ../escape` writes nothing; a path in the crate +list of the build helper is skipped. Sampled names, not an exhaustive or +property-based check of the pattern. Grid-side trust boundary as in +R-ENVELOPE. + +|R-WIRED: `fix-fleet-enrolment.sh` exits 0 only when the lane that reads the +artifact and the job that produces it are both wired. Exit 3 means written or +checked but not wired end to end; exit 2 is a refusal or invalid input and +changes nothing +|`tested-only` +|`bash tests/fix-fleet-enrolment-test.sh`, 139 checks +|`scripts/fix-fleet-enrolment.sh` +|Bounds: offline fixture repositories and fake standards checkouts, in the +numbered sections 0 to 18: canonical and non-canonical workflows, owner +detection, existing `needs` forms (list, scalar, block list, an item that +merely starts with `fleet`, a job merely named `fleet`), a pin whose Hypatia +reusable does or does not read the artifact, a pin that lacks the producer +(refused), a remote pin with no standards checkout or without the pinned +commit (reported unverified, not `ok`), refusals that change nothing, an +existing manifest left alone, a malformed panic-attack pin rejected before any +file is created, and a second run changing nothing. Case 13 reports a skip on +a shallow clone; no skip occurred in this run. No real GitHub repository, +workflow run or `gh actions-lock` was exercised. Whether a deployed repository +is actually wired is a live fact this test cannot establish. + +|R-BUILD: `fleet-report-back-build.sh`: a crate that fails to build is a +warning and later an `error` row, never a dead run +|`tested-only` +|`bash tests/fleet-report-back-build-test.sh`, 69 checks (also run by the +report-back test) +|`scripts/fleet-report-back-build.sh` +|Bounds: Cargo and `timeout` are stubbed, so no Rust crate is built and no +network is used. Cases: a failed crate's partial binary is not copied and its +warning names the report-back consequence; an all-failed build publishes +nothing and counts both failures; aliases are deduplicated and members without +crates skipped; a path in the crate list is skipped; for `ponsbot`, the +install is copied, a cache hit does not reinstall, a failed install is a +warning, a non-commit value or a commit differing from the pin never reaches +Cargo, an unreadable pin installs nothing, and a spent build budget skips the +install. Real compile failures, timeouts of real builds and cache corruption +are not covered. Grid-side trust boundary as in R-ENVELOPE. +|=== + +== Ledger: open obligations + +These four obligations come from the pre-policy file. Each concerns the +behaviour of first-party code, so under PROOF-POLICY §1 the code mapping takes +precedence: Rust code is verified with Creusot (Pearlite contracts) and Verus +on the code itself (§1.2), and the cross-bot contract that stands apart from any one +implementation is an Idris2 interface. The earlier suggestion of Lean 4 for +"algebraic properties of confidence scoring" is not policy. If the confidence +scoring is stated as a theory theorem, true of any implementation, the source +is Agda, rechecked in Rocq and Lean; a statement about this repository's +functions is engineering and stays with Creusot. + +None has a checker run against it, and none has a receipt. Each stays `open` +until a verifier discharges it. + +[cols="3,1,2,2,3",options="header"] +|=== +|Obligation |Status |Checker (version, flags) |Where |Trust boundary + +|O-ACTION: bot action safety. Automated actions (pull requests, issues, fixes) +are reversible and do not destroy repository state +|`open` +|Rust: Creusot with Pearlite contracts, and Verus, none run. The interface +contract between a bot and the fleet: Idris2, not started +|`robot-repo-automaton/` (Rust), `scripts/dispatch-runner.sh` and `scripts/fix-*.sh` +(shell, no verifier maps to shell) +|Reversibility of a GitHub operation depends on GitHub's behaviour and the +repository's branch protection, so it would be an `ASSUMED` boundary +specification, not something this repository proves. The shell fix scripts are +guarded only by tests and the Repo Integrity Guard +(`.github/workflows/repo-integrity-guard.yml`), which are not proofs. + +|O-THRESHOLD: confidence-threshold gating. Nothing is auto-merged or +auto-executed below its threshold (Safety Triangle: eliminate at 0.95 and +above, substitute at 0.85 and above, control below) +|`open` +|Rust: Creusot with Pearlite contracts, and Verus, none run. If the scoring +order is stated as theory: Agda, rechecked in Rocq and Lean, none started +|`robot-repo-automaton/` (Rust); `scripts/dispatch-runner.sh` (shell, thresholds +read from findings) +|The confidence values are produced by scanners outside this repository +(hypatia), so the property can only be stated relative to the input value +(`ASSUMED` boundary). Where the gate is in shell, no verifier maps to it, and +the obligation would need the gate moved into Rust first. + +|O-CONFLICT: when several bots produce conflicting recommendations, the resolution is deterministic and documented -* *Rate limiting*: Prove bots respect GitHub API rate limits and do not -cause cascading failures across the fleet +|`open` +|Rust arbiter: Creusot with Pearlite contracts, and Verus, none run. The +ordering and tie-break contract: Idris2, not started +|`shared-context/` (Rust crate for inter-bot communication) and +`fleet-coordinator.sh` +|Determinism over the coordinator's inputs only; the order in which concurrent +bots finish is outside the property and would be an `ASSUMED` boundary. -=== Recommended prover +|O-RATE: bots respect GitHub API rate limits and do not cause cascading +failures across the fleet +|`open` +|Rust: Creusot with Pearlite contracts, and Verus, none run +|Rate-limit handling in the Rust crates under `bots/` and `robot-repo-automaton/` +|Only a local bound on the bot's own request counts is provable. GitHub's +limits, clocks and failure behaviour are `ASSUMED` boundaries, and "no +cascading failure" is a fleet-level property that needs a stated model first. +|=== -* *Idris2* — For bot action safety and confidence threshold properties -* *Lean4* — For algebraic properties of confidence scoring if modeled as -an ordered semiring +Using Creusot and Verus together does not make two independent witnesses +(policy §1.2). -=== Priority +== Not verified when this file was written -* *MEDIUM* — The fleet manages automated repository changes. Incorrect -bot behavior at scale could damage many repos. The "`formal -verification`" claim should be substantiated. +* No Agda, Idris2, Lean or Rocq checker was available, so the `argued` rows + rest on echidnabot's receipts plus a byte-identity comparison, and the + receipts themselves were not reproduced. +* The state of `ReportBack.idr` on the standards branch was not checked + beyond its being pending; no claim here depends on it. +* The census counts the 15 rows above and nothing else: the report-back and + open rows cover the properties that this file names, not the whole of the + repository's shell and Rust code, which has no proof coverage. + +== History === Template ABI Cleanup (2026-03-29) -Template ABI removed – was creating false impression of formal -verification. The removed files (Types.idr, Layout.idr, Foreign.idr) +Template ABI removed: it was creating a false impression of formal +verification. The removed files (`Types.idr`, `Layout.idr`, `Foreign.idr`) contained only RSR template scaffolding with unresolved -\{\{PROJECT}}/\{\{AUTHOR}} placeholders and no domain-specific proofs. +`+{{PROJECT}}+`/`+{{AUTHOR}}+` placeholders and no domain-specific proofs. + +The same three file names are tracked again at the date above, under +`bots/echidnabot/src/abi/`, as echidnabot's own scaffold. They still carry +template placeholders and holes, and are recorded `open` above. From 6cf6125b4a4dcf8fde80d9fd8e2c1a501c78d811 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:08:47 +0000 Subject: [PATCH 2/5] fix(enrolment): build the manifest with jq, not Python; write fleet reporters only fix-fleet-enrolment.sh (#629) built the manifest with `python3 -c`, and its test checked it with Python too. Python is banned estate-wide, and the stated reason ("jq 1.6 does NOT preserve object key order") is not true of objects jq constructs. The manifest is now built by reducing over the priority list in jq, which keeps insertion order; the test compares against a jq re-serialisation and the exact key list. The manifest also carried `hypatia` and `robot-repo-automaton` entries. They are in member-priority.json for the Arbiter, but they are not fleet reporters: the `hypatia` entry gave the verdict grid a second row with the scanner's own id. Only the nine reporters get an entry now; the test asserts both roles are absent. bash tests/fix-fleet-enrolment-test.sh: 141 passed, 0 failed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDjqj5DHRRCrDiGeTmq --- scripts/fix-fleet-enrolment.sh | 85 +++++++++---------------------- tests/fix-fleet-enrolment-test.sh | 20 +++----- 2 files changed, 31 insertions(+), 74 deletions(-) diff --git a/scripts/fix-fleet-enrolment.sh b/scripts/fix-fleet-enrolment.sh index c7b49bf1..37d10017 100755 --- a/scripts/fix-fleet-enrolment.sh +++ b/scripts/fix-fleet-enrolment.sh @@ -156,68 +156,29 @@ robot-repo-automaton hypatia' fi - # jq 1.6 does NOT preserve object key order in output regardless of input method. - # Use Python (which preserves dict insertion order since 3.7) to generate the - # manifest with keys in canonical priority order. - # Convert shell true/false to Python True/False - py_rust=$( [ "$has_rust" = "true" ] && echo "True" || echo "False" ) - py_seams=$( [ "$has_seams" = "true" ] && echo "True" || echo "False" ) - py_html=$( [ "$has_html" = "true" ] && echo "True" || echo "False" ) - py_proofs=$( [ "$has_proofs" = "true" ] && echo "True" || echo "False" ) - - python3 -c " -import json -import sys - -has_rust = $py_rust -has_seams = $py_seams -has_html = $py_html -has_proofs = $py_proofs - -priority_order = '''$priority_order'''.strip().split('\n') - -reporters = {} -for member in priority_order: - if member == 'rhodibot': - reporters[member] = {'title': 'rhodibot (RSR compliance)', 'enrolled': True} - elif member == 'echidnabot': - if has_proofs: - reporters[member] = {'title': 'echidnabot (formal verification)', 'enrolled': True} - else: - reporters[member] = {'enrolled': False, 'reason': 'no proof files in the tree'} - elif member == 'sustainabot': - reporters[member] = {'enrolled': False, 'reason': 'reserved fleet slot; no adapter exists'} - elif member == 'panicbot': - if has_rust: - reporters[member] = {'title': 'panicbot (panic recovery, error handling)', 'enrolled': True} - else: - reporters[member] = {'enrolled': False, 'reason': 'no Cargo.toml in the tree; panic-attack scans Rust'} - elif member == 'glambot': - reporters[member] = {'title': 'glambot (presentation quality)', 'enrolled': True} - elif member == 'seambot': - if has_seams: - reporters[member] = {'title': 'seambot (ABI/FFI seam checks)', 'enrolled': True} - else: - reporters[member] = {'enrolled': False, 'reason': 'no seam register is declared (spec/seams/seam-register.json)'} - elif member == 'finishingbot': - reporters[member] = {'title': 'finishingbot (release readiness)', 'enrolled': True} - elif member == 'accessibilitybot': - if has_html: - reporters[member] = {'title': 'accessibilitybot (WCAG conformance)', 'enrolled': True} - else: - reporters[member] = {'enrolled': False, 'reason': 'no HTML files in the tree'} - elif member == 'cipherbot': - reporters[member] = {'title': 'cipherbot (cryptographic hygiene)', 'enrolled': True} - elif member == 'robot-repo-automaton': - reporters[member] = {'enrolled': False, 'reason': 'not a fleet reporter; executor role only'} - elif member == 'hypatia': - reporters[member] = {'enrolled': False, 'reason': 'not a fleet reporter; engine role only'} - -manifest = {'schema': 1, 'reporters': reporters} -# Compact output with single trailing newline (I-JSON/JCS) -json.dump(manifest, sys.stdout, separators=(',', ':')) -print() -" > "$MANIFEST" + # jq builds objects in insertion order, so reducing over the priority list + # writes the manifest in canonical priority order. Only fleet REPORTERS get + # an entry: robot-repo-automaton (executor) and hypatia (engine) are in the + # priority file for the Arbiter, but they are not grid rows, and a manifest + # entry named `hypatia` would give the grid a second row with the scanner's id. + printf '%s\n' "$priority_order" | jq -R . | jq -sc \ + --argjson rust "$has_rust" --argjson seams "$has_seams" \ + --argjson html "$has_html" --argjson proofs "$has_proofs" ' + def on($title): {title: $title, enrolled: true}; + def off($reason): {enrolled: false, reason: $reason}; + def entry: + if . == "rhodibot" then on("rhodibot (RSR compliance)") + elif . == "echidnabot" then (if $proofs then on("echidnabot (formal verification)") else off("no proof files in the tree") end) + elif . == "sustainabot" then off("reserved fleet slot; no adapter exists") + elif . == "panicbot" then (if $rust then on("panicbot (panic recovery, error handling)") else off("no Cargo.toml in the tree; panic-attack scans Rust") end) + elif . == "glambot" then on("glambot (presentation quality)") + elif . == "seambot" then (if $seams then on("seambot (ABI/FFI seam checks)") else off("no seam register is declared (spec/seams/seam-register.json)") end) + elif . == "finishingbot" then on("finishingbot (release readiness)") + elif . == "accessibilitybot" then (if $html then on("accessibilitybot (WCAG conformance)") else off("no HTML files in the tree") end) + elif . == "cipherbot" then on("cipherbot (cryptographic hygiene)") + else null end; + {schema: 1, reporters: (reduce (.[] | select(length > 0)) as $m ({}; ($m | entry) as $e | if $e == null then . else . + {($m): $e} end))} + ' > "$MANIFEST" echo " CREATED .gitbot-fleet/enrolment.json (rust=$has_rust seams=$has_seams html=$has_html proofs=$has_proofs; edit it — enrol what is true of this repository)" fi diff --git a/tests/fix-fleet-enrolment-test.sh b/tests/fix-fleet-enrolment-test.sh index b9ffd72f..8b62b9d7 100755 --- a/tests/fix-fleet-enrolment-test.sh +++ b/tests/fix-fleet-enrolment-test.sh @@ -95,18 +95,14 @@ canonical_workflow "$NEW_SHA" > "$A/.github/workflows/fork-governance.yml"; chmo set +e; bash "$FIXER" "$A" > "$TMP/a.out" 2>&1; rc=$?; set -e check "A: exit 0 — wired end to end" 0 "$rc" check_grep "A: manifest created" 'CREATED \.gitbot-fleet/enrolment\.json' "$TMP/a.out" -check "A: manifest is canonical priority order compact JSON with one final LF (I-JCS as Python produces it)" \ - "$(python3 -c " -import json, sys -with open('$A/.gitbot-fleet/enrolment.json') as f: - data = json.load(f) -# Re-serialize with same priority order as fix-fleet-enrolment.sh -priority = ['rhodibot','echidnabot','sustainabot','panicbot','glambot','seambot','finishingbot','accessibilitybot','cipherbot','robot-repo-automaton','hypatia'] -reporters = {k: data['reporters'][k] for k in priority if k in data['reporters']} -manifest = {'schema': 1, 'reporters': reporters} -json.dump(manifest, sys.stdout, separators=(',', ':')) -print() -")" "$(cat "$A/.gitbot-fleet/enrolment.json")" +check "A: manifest is canonical priority order compact JSON with one final LF (I-JCS)" \ + "$(jq -c '{schema: 1, reporters: (.reporters as $r | reduce ("rhodibot","echidnabot","sustainabot","panicbot","glambot","seambot","finishingbot","accessibilitybot","cipherbot") as $k ({}; if $r | has($k) then . + {($k): $r[$k]} else . end))}' "$A/.gitbot-fleet/enrolment.json")" \ + "$(cat "$A/.gitbot-fleet/enrolment.json")" +check "A: manifest lists exactly the nine fleet reporters, in priority order" \ + "rhodibot echidnabot sustainabot panicbot glambot seambot finishingbot accessibilitybot cipherbot" \ + "$(jq -r '.reporters | keys_unsorted | join(" ")' "$A/.gitbot-fleet/enrolment.json")" +check "A: non-reporter roles (hypatia, robot-repo-automaton) get no manifest entry" "false false" \ + "$(jq -r '[(.reporters | has("hypatia")), (.reporters | has("robot-repo-automaton"))] | map(tostring) | join(" ")' "$A/.gitbot-fleet/enrolment.json")" check "A: manifest ends with exactly one LF" "1" "$(tail -c1 "$A/.gitbot-fleet/enrolment.json" | wc -l | tr -d ' ')" check "A: every-repo members enrolled" "true true true true" \ "$(jq -r '[.reporters.rhodibot, .reporters.finishingbot, .reporters.glambot, .reporters.cipherbot][] | .enrolled' "$A/.gitbot-fleet/enrolment.json" | paste -sd' ' -)" From 5bcf758277f9cd4ab4be357e953bf9cac634ea6b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:08:48 +0000 Subject: [PATCH 3/5] =?UTF-8?q?test(enrolment):=20end-to-end=20canary=20ha?= =?UTF-8?q?rness=20=E2=80=94=20fixer,=20driver,=20renderer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tests/enrolment-canary-test.sh runs the rollout on a synthetic repository (owner decision 2026-11-18: prove the mechanism before touching an estate repo). One chain, each stage the real script: 1. a fork-governance repo with a Hypatia lane and no manifest; 2. fix-fleet-enrolment.sh: exit 0, manifest written, fleet job inserted at the fleet ref, `hypatia: needs: [fleet]`, a second run a no-op; 3. fleet-report-back.sh with stub adapters: one envelope per enrolled member, scanned tree unchanged; 4. standards' verdict-grid.sh, before (every enrolled row "enrolled, but no report-back") and after (none; clean members green, non-enrolled grey). It found the duplicate `hypatia` row fixed in the previous commit. It does not test GitHub's startup rules for the inserted job; that stays a live fact. Wired as a step of fleet-report-back.yml beside the driver suite (no new `uses:`; check-lock-sync.sh in sync, actionlint clean). PROOF-NEEDS R-WIRED now cites it. Local: canary 25 passed; driver suite 243 passed; fixer suite 141 passed. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDjqj5DHRRCrDiGeTmq --- .github/workflows/fleet-report-back.yml | 5 + PROOF-NEEDS.adoc | 10 +- tests/enrolment-canary-test.sh | 171 ++++++++++++++++++++++++ 3 files changed, 183 insertions(+), 3 deletions(-) create mode 100755 tests/enrolment-canary-test.sh diff --git a/.github/workflows/fleet-report-back.yml b/.github/workflows/fleet-report-back.yml index 7dfe15fa..c22aaf5e 100644 --- a/.github/workflows/fleet-report-back.yml +++ b/.github/workflows/fleet-report-back.yml @@ -81,6 +81,11 @@ jobs: - name: Run the driver's test suite against the real renderer run: STANDARDS_DIR="$PWD/.standards-grid" bash tests/fleet-report-back-test.sh + # The enrolment rollout end to end on a synthetic repository: fixer, then + # driver, then the real renderer, before and after (tests/enrolment-canary-test.sh). + - name: Enrolment canary (fixer, driver and renderer, end to end) + run: STANDARDS_DIR="$PWD/.standards-grid" bash tests/enrolment-canary-test.sh + - name: Render the grid from the envelopes and fail on any silent member env: HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }} diff --git a/PROOF-NEEDS.adoc b/PROOF-NEEDS.adoc index 651a575c..2369c9b2 100644 --- a/PROOF-NEEDS.adoc +++ b/PROOF-NEEDS.adoc @@ -220,7 +220,7 @@ artifact and the job that produces it are both wired. Exit 3 means written or checked but not wired end to end; exit 2 is a refusal or invalid input and changes nothing |`tested-only` -|`bash tests/fix-fleet-enrolment-test.sh`, 139 checks +|`bash tests/fix-fleet-enrolment-test.sh`, 141 checks; `STANDARDS_DIR=… bash tests/enrolment-canary-test.sh`, 25 checks (fixer, then driver, then the real renderer, end to end) |`scripts/fix-fleet-enrolment.sh` |Bounds: offline fixture repositories and fake standards checkouts, in the numbered sections 0 to 18: canonical and non-canonical workflows, owner @@ -232,8 +232,12 @@ commit (reported unverified, not `ok`), refusals that change nothing, an existing manifest left alone, a malformed panic-attack pin rejected before any file is created, and a second run changing nothing. Case 13 reports a skip on a shallow clone; no skip occurred in this run. No real GitHub repository, -workflow run or `gh actions-lock` was exercised. Whether a deployed repository -is actually wired is a live fact this test cannot establish. +workflow run or `gh actions-lock` was exercised. The canary harness chains the +fixer, the driver (adapters stubbed) and standards' renderer on one synthetic +repository: before, every enrolled row reads "enrolled, but no report-back"; +after, none does. It does not exercise GitHub's startup rules for the inserted +job. Whether a deployed repository is actually wired is a live fact these tests +cannot establish. |R-BUILD: `fleet-report-back-build.sh`: a crate that fails to build is a warning and later an `error` row, never a dead run diff --git a/tests/enrolment-canary-test.sh b/tests/enrolment-canary-test.sh new file mode 100755 index 00000000..590df8f7 --- /dev/null +++ b/tests/enrolment-canary-test.sh @@ -0,0 +1,171 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# +# enrolment-canary-test.sh — the fleet-enrolment rollout, end to end, on a +# synthetic repository instead of an estate one (owner decision 2026-11-18). +# +# One chain, each stage the real script: +# 1. a repository with a fork-governance Hypatia lane and no manifest; +# 2. scripts/fix-fleet-enrolment.sh writes the manifest, inserts the fleet +# job and wires `hypatia: needs: [fleet]` (exit 0: no actions.lock here); +# 3. scripts/fleet-report-back.sh runs the members the manifest enrols +# (adapters stubbed with their real JSON shapes) and writes envelopes; +# 4. standards' scripts/verdict-grid.sh renders the grid twice: without the +# envelopes (the "before" state: every enrolled row orange, "enrolled, +# but no report-back was received") and with them (the "after" state: no +# enrolled row left orange for want of evidence). +# +# What this does NOT test: GitHub's startup rules for the inserted job (the +# lock records, reusable resolution). Those need a live run on a real repo. +# +# Stage 4 needs STANDARDS_DIR (a standards checkout carrying +# scripts/verdict-grid.sh and scripts/lib/verdict-grid.jq); without it the +# renderer stage is skipped and says so. +# +# Run: STANDARDS_DIR=/path/to/standards bash tests/enrolment-canary-test.sh +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +FIXER="$ROOT/scripts/fix-fleet-enrolment.sh" +DRIVER="$ROOT/scripts/fleet-report-back.sh" +GRID_STANDARDS="${STANDARDS_DIR:-}" +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT +export PIN_FIXER_LOCK_DIR="$TMP/locks"; mkdir -m 700 "$PIN_FIXER_LOCK_DIR" + +pass=0; fail=0 +check() { # name expected actual + if [ "$2" = "$3" ]; then pass=$((pass + 1)); echo "ok $1" + else fail=$((fail + 1)); echo "FAIL $1"; echo " want: $2"; echo " got: $3"; fi +} + +# ── a standards checkout whose Hypatia reusable reads the artifact ─────────── +# The fixer verifies the pinned reusable declares `report-backs-artifact`; a +# synthetic checkout keeps that check real without needing the network. +STD="$TMP/standards"; mkdir -p "$STD/.github/workflows"; git -C "$STD" init -q +printf 'on:\n workflow_call:\n inputs:\n report-backs-artifact:\n type: string\n default: fleet-report-backs\njobs:\n scan:\n steps:\n - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333\n' \ + > "$STD/.github/workflows/hypatia-scan-reusable.yml" +git -C "$STD" add -A +git -C "$STD" -c user.email=t@example.invalid -c user.name=t -c commit.gpgsign=false commit -q -m reads-artifact +STD_SHA=$(git -C "$STD" rev-parse HEAD) + +# ── 1. the canary repository: a Hypatia lane, no manifest ──────────────────── +REPO="$TMP/canary"; mkdir -p "$REPO/.github/workflows" "$REPO/src" +git -C "$REPO" init -q +git -C "$REPO" remote add origin https://github.com/hyperpolymath/canary.git +echo 'x' > "$REPO/src/a.jl" +cat > "$REPO/.github/workflows/fork-governance.yml" < "$TMP/fixer.out" 2>&1 +rc=$? +set -e +check "fixer: exits 0 (manifest written, job inserted and wired, pin reads the artifact)" 0 "$rc" +[ "$rc" = 0 ] || sed 's/^/ | /' "$TMP/fixer.out" +MANIFEST="$REPO/.gitbot-fleet/enrolment.json" +check "fixer: the manifest is a regular file" yes "$([ -f "$MANIFEST" ] && [ ! -L "$MANIFEST" ] && echo yes || echo no)" +check "fixer: the manifest parses as JSON" yes "$(jq -e . "$MANIFEST" >/dev/null 2>&1 && echo yes || echo no)" +WF="$REPO/.github/workflows/fork-governance.yml" +check "fixer: the fleet job calls the fleet reusable at the fleet ref" yes \ + "$(grep -qE "uses: hyperpolymath/gitbot-fleet/\.github/workflows/fleet-report-back-reusable\.yml@$FLEET_REF\$" "$WF" && echo yes || echo no)" +check "fixer: hypatia waits for the fleet job" yes \ + "$(awk '/^ hypatia:/{p=1;next} p&&/^ [a-z]/{exit} p' "$WF" | grep -qE '^ needs: \[fleet\]$' && echo yes || echo no)" +set +e; STANDARDS_DIR="$STD" FLEET_REF="$FLEET_REF" PANIC_ATTACK_SHA=2222222222222222222222222222222222222222 \ + bash "$FIXER" "$REPO" > "$TMP/fixer2.out" 2>&1; rc2=$?; set -e +check "fixer: a second run is a no-op that still exits 0" 0 "$rc2" +check "fixer: a second run inserts no second fleet job" 1 "$(grep -cE '^ fleet:' "$WF")" + +ENROLLED=$(jq -r '.reporters | to_entries[] | select(.value.enrolled == true) | .key' "$MANIFEST" | sort | tr '\n' ' ' | sed 's/ $//') +echo " enrolled: $ENROLLED" + +# ── 3. the producer, with stub adapters ────────────────────────────────────── +BIN="$TMP/bin"; mkdir -p "$BIN" +stub() { printf '#!/usr/bin/env bash\nset -u\n%s\n' "$2" > "$BIN/$1"; chmod +x "$BIN/$1"; } +stub rhodibot 'echo "{\"subject\":\"$3\",\"criteria\":1,\"scored\":1,\"not_applicable\":0,\"counts\":{\"satisfied\":1,\"relocated\":0,\"deprecated\":0,\"missing\":0},\"findings\":[]}"' +stub finishing-bot 'echo "{\"license\":{\"findings\":[]},\"placeholder\":{\"findings\":[]},\"release\":{\"findings\":[]}}"' +stub glambot 'echo "{\"visual\":{\"findings\":[]},\"seo\":{\"findings\":[]}}"' +stub cipherbot 'echo "[]"' +stub panicbot 'echo "[]"' +stub panic-attack 'echo stub' +stub seambot 'echo "{\"status\":\"pass\",\"findings\":[],\"summary\":{\"errors\":0}}"' +stub accessibilitybot 'echo "{\"findings\":[]}"' +repo_before=$(cd "$REPO" && find . -type f -exec sha256sum {} + | sort) +OUT="$TMP/report-backs" +ECHIDNA_URL='' bash "$DRIVER" --repo "$REPO" --out "$OUT" --bin-dir "$BIN" --fleet-ref "$FLEET_REF" > "$TMP/driver.out" 2>&1 \ + || { echo " driver exit $?"; sed 's/^/ | /' "$TMP/driver.out"; } +GOT=$(find "$OUT" -name '*.json' -printf '%f\n' 2>/dev/null | sed 's/\.json$//' | sort | tr '\n' ' ' | sed 's/ $//') +check "producer: one envelope per enrolled member, no more" "$ENROLLED" "$GOT" +check "producer: the scanned repository is unchanged" "$repo_before" "$(cd "$REPO" && find . -type f -exec sha256sum {} + | sort)" +for m in $ENROLLED; do + s=$(jq -r .status "$OUT/$m.json") + case "$m" in + echidnabot | sustainabot) check "producer: $m is not_run with a reason" not_run "$s" ;; + *) check "producer: $m ran" ran "$s" ;; + esac +done + +# ── 4. the renderer: before and after ──────────────────────────────────────── +GRID="$GRID_STANDARDS/scripts/verdict-grid.sh" +if [ -n "$GRID_STANDARDS" ] && [ -f "$GRID" ]; then + EMPTY="$TMP/empty"; mkdir -p "$EMPTY" + render() { # report-backs-dir out-dir + (cd "$REPO" && bash "$GRID" render --out "$2" --hypatia /dev/null --panic /dev/null --pons /dev/null \ + --report-backs "$1" --enrolment .gitbot-fleet/enrolment.json --sha 0000000 > /dev/null) + } + render "$EMPTY" "$TMP/grid-before" + render "$OUT" "$TMP/grid-after" + silent() { jq '[.reporters[] | select(.reason == "enrolled, but no report-back was received")] | length' "$1/report.json"; } + n=$(echo "$ENROLLED" | wc -w) + check "renderer, before: every enrolled member is 'enrolled, but no report-back'" "$n" "$(silent "$TMP/grid-before")" + check "renderer, after: no enrolled member is left without a report-back" 0 "$(silent "$TMP/grid-after")" + for m in $ENROLLED; do + st=$(jq -r --arg id "$m" '.reporters[] | select(.id == $id) | .state' "$TMP/grid-after/report.json") + case "$m" in + echidnabot | sustainabot) check "renderer, after: $m (not_run) stays orange, now with its reason" orange "$st" ;; + *) check "renderer, after: $m ran clean and is green" green "$st" ;; + esac + done + for m in $(jq -r '.reporters | to_entries[] | select(.value.enrolled == false) | .key' "$MANIFEST"); do + check "renderer: $m (not enrolled) is grey" grey \ + "$(jq -r --arg id "$m" '.reporters[] | select(.id == $id) | .state' "$TMP/grid-after/report.json")" + done +else + echo "SKIP: STANDARDS_DIR not set or has no scripts/verdict-grid.sh; renderer stage not run" +fi + +echo +echo "enrolment canary: $pass passed, $fail failed" +[ "$fail" -eq 0 ] From 18ca79c10e8ec6a893274843677dc5d54c25d378 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:26:22 +0000 Subject: [PATCH 4/5] fix(ci): prune the actions.lock entry #697 left behind for learning-loop.yml #697 removed the "Upload rule proposals" step from learning-loop.yml but left actions/upload-artifact@v7.0.2 under that workflow's lock entry. Main has been red since: check-lock-sync.sh fails on the stale entry and audit-lock-closure.sh reports FATAL WF-ORPHAN, which fails Pinned source closure and, through it, E2E tests. Pruned with standards scripts/prune-stale.sh (the tool step of the lock repair chain), not by hand. The dependencies: record stays: two other workflows still use upload-artifact@v7.0.2. Local, after: check-lock-sync.sh rc=0; audit-lock-closure.sh fatal=0 review=0 accepted=3; lock-closure-test 47 passed, 0 failed (45/2 before); audit-pinned-reusable-sources.sh --fetch-sources findings=0. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDjqj5DHRRCrDiGeTmq --- .github/workflows/actions.lock | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/actions.lock b/.github/workflows/actions.lock index 60e83b07..38f208b8 100644 --- a/.github/workflows/actions.lock +++ b/.github/workflows/actions.lock @@ -43,7 +43,6 @@ workflows: '.github/workflows/labels.yml': [] '.github/workflows/learning-loop.yml': - 'actions/checkout@v7.0.1' - - 'actions/upload-artifact@v7.0.2' '.github/workflows/lock-sync-gate.yml': [] '.github/workflows/mirror.yml': [] '.github/workflows/pages.yml': From 59ac207e9200bb8e7f1a9d642991d1279aacda68 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 18 Nov 2026 00:26:22 +0000 Subject: [PATCH 5/5] test(enrolment-canary): a featured fixture; correct three ledger claims The canary now runs its chain on two synthetic repositories. The second carries a Cargo.toml, a seam register, an HTML page and an Agda file, so panicbot, seambot, accessibilitybot and echidnabot are enrolled, and the echidnabot not_run branch (orange, with its reason, in the after grid) executes instead of being dead code. 56 checks, 0 failed; a negative control (the proof file renamed to .txt) fails the conditional-enrolment check as intended. PROOF-NEEDS.adoc, from review: - R-ENVELOPE: a panicbot or ponsbot status object keeps not_run/error on a non-zero exit; "any other non-zero is error" holds only on the generic findings path. - R-WIRED: exit 0 does not establish that actions.lock is in sync; the fixer reports a lock to regenerate only on the run that inserts the job. - R-WIRED evidence names its date and revision and the renderer pin. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDjqj5DHRRCrDiGeTmq --- PROOF-NEEDS.adoc | 16 ++- tests/enrolment-canary-test.sh | 201 +++++++++++++++++++-------------- 2 files changed, 128 insertions(+), 89 deletions(-) diff --git a/PROOF-NEEDS.adoc b/PROOF-NEEDS.adoc index 2369c9b2..3cedb9fd 100644 --- a/PROOF-NEEDS.adoc +++ b/PROOF-NEEDS.adoc @@ -187,7 +187,10 @@ bash with `jq` (taken from a live run on 2026-10-27), not the seven Rust crates. Cases: only `enrolled: true` members run; an unreadable manifest runs nothing; missing binary, crash without JSON and timeout are `error` with a reason; exit 1 with -valid JSON stays `ran`, any other non-zero status is `error`; empty scans, +valid JSON stays `ran`, and on the generic findings path any other non-zero +status is `error`, while a panicbot or ponsbot status object (`error` or +`not_run` with a reason, as panicbot prints with exit 3 or 4) keeps its own +status unless the adapter timed out; empty scans, malformed output and path normalisation; `echidnabot` and `sustainabot` are `not_run` with a reason; an unknown member is `error`; the driver never writes `not_applicable`; the envelope is key-sorted compact JSON with one final LF; @@ -216,11 +219,14 @@ property-based check of the pattern. Grid-side trust boundary as in R-ENVELOPE. |R-WIRED: `fix-fleet-enrolment.sh` exits 0 only when the lane that reads the -artifact and the job that produces it are both wired. Exit 3 means written or -checked but not wired end to end; exit 2 is a refusal or invalid input and -changes nothing +artifact and the job that produces it are both wired. Exit 0 does not +establish that `actions.lock` is in sync: the fixer reports a lock that needs +regenerating (exit 3) only on the run that inserts the job, so a later run can +exit 0 over a stale lock, and the repository's own lock-sync gate is the check +for that. Exit 3 means written or checked but not wired end to end; exit 2 is +a refusal or invalid input and changes nothing |`tested-only` -|`bash tests/fix-fleet-enrolment-test.sh`, 141 checks; `STANDARDS_DIR=… bash tests/enrolment-canary-test.sh`, 25 checks (fixer, then driver, then the real renderer, end to end) +|`bash tests/fix-fleet-enrolment-test.sh`, 141 checks; `STANDARDS_DIR=… bash tests/enrolment-canary-test.sh`, 56 checks over two synthetic repositories, one bare and one that enrols the conditional members (fixer, then driver, then the real renderer at standards `9ae23875`, end to end). Measured 2026-11-18 on the commit that records these counts (the canary's second fixture lands in the same commit), not at the ledger's `7907099` |`scripts/fix-fleet-enrolment.sh` |Bounds: offline fixture repositories and fake standards checkouts, in the numbered sections 0 to 18: canonical and non-canonical workflows, owner diff --git a/tests/enrolment-canary-test.sh b/tests/enrolment-canary-test.sh index 590df8f7..065305b8 100755 --- a/tests/enrolment-canary-test.sh +++ b/tests/enrolment-canary-test.sh @@ -2,8 +2,8 @@ # SPDX-License-Identifier: MPL-2.0 # SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) # -# enrolment-canary-test.sh — the fleet-enrolment rollout, end to end, on a -# synthetic repository instead of an estate one (owner decision 2026-11-18). +# enrolment-canary-test.sh — the fleet-enrolment rollout, end to end, on +# synthetic repositories instead of estate ones (owner decision 2026-11-18). # # One chain, each stage the real script: # 1. a repository with a fork-governance Hypatia lane and no manifest; @@ -16,6 +16,13 @@ # but no report-back was received") and with them (the "after" state: no # enrolled row left orange for want of evidence). # +# The chain runs on two repositories: +# * bare: one Julia file, so only the always-enrolled members run; +# * featured: a Cargo.toml, a seam register, an HTML page and a proof file, +# so panicbot, seambot, accessibilitybot and echidnabot are enrolled too, +# and echidnabot (no ECHIDNA_URL) takes the not_run path: orange, with its +# reason, in the "after" grid. +# # What this does NOT test: GitHub's startup rules for the inserted job (the # lock records, reusable resolution). Those need a live run on a real repo. # @@ -49,13 +56,39 @@ printf 'on:\n workflow_call:\n inputs:\n report-backs-artifact:\n git -C "$STD" add -A git -C "$STD" -c user.email=t@example.invalid -c user.name=t -c commit.gpgsign=false commit -q -m reads-artifact STD_SHA=$(git -C "$STD" rev-parse HEAD) +FLEET_REF=$(git -C "$ROOT" rev-parse HEAD) + +# ── stub adapters, printing the JSON shapes of the real ones ───────────────── +BIN="$TMP/bin"; mkdir -p "$BIN" +stub() { printf '#!/usr/bin/env bash\nset -u\n%s\n' "$2" > "$BIN/$1"; chmod +x "$BIN/$1"; } +stub rhodibot 'echo "{\"subject\":\"$3\",\"criteria\":1,\"scored\":1,\"not_applicable\":0,\"counts\":{\"satisfied\":1,\"relocated\":0,\"deprecated\":0,\"missing\":0},\"findings\":[]}"' +stub finishing-bot 'echo "{\"license\":{\"findings\":[]},\"placeholder\":{\"findings\":[]},\"release\":{\"findings\":[]}}"' +stub glambot 'echo "{\"visual\":{\"findings\":[]},\"seo\":{\"findings\":[]}}"' +stub cipherbot 'echo "[]"' +stub panicbot 'echo "[]"' +stub panic-attack 'echo stub' +stub seambot 'echo "{\"status\":\"pass\",\"findings\":[],\"summary\":{\"errors\":0}}"' +stub accessibilitybot 'echo "{\"findings\":[]}"' + +# canary LABEL FEATURES(bare|featured) EXPECTED-CONDITIONAL-MEMBERS +canary() { + local label="$1" features="$2" want_conditional="$3" + local dir="$TMP/$label" REPO="$TMP/$label/repo" + echo "── $label ──" -# ── 1. the canary repository: a Hypatia lane, no manifest ──────────────────── -REPO="$TMP/canary"; mkdir -p "$REPO/.github/workflows" "$REPO/src" -git -C "$REPO" init -q -git -C "$REPO" remote add origin https://github.com/hyperpolymath/canary.git -echo 'x' > "$REPO/src/a.jl" -cat > "$REPO/.github/workflows/fork-governance.yml" < "$REPO/src/a.jl" + if [ "$features" = featured ]; then + printf '[package]\nname = "canary"\nversion = "0.1.0"\nedition = "2021"\n' > "$REPO/Cargo.toml" + mkdir -p "$REPO/spec/seams" "$REPO/docs" "$REPO/verification" + printf '{"seams":[]}\n' > "$REPO/spec/seams/seam-register.json" + printf 'c\n' > "$REPO/docs/index.html" + printf 'module Canary where\n' > "$REPO/verification/Canary.agda" + fi + cat > "$REPO/.github/workflows/fork-governance.yml" < "$TMP/fixer.out" 2>&1 -rc=$? -set -e -check "fixer: exits 0 (manifest written, job inserted and wired, pin reads the artifact)" 0 "$rc" -[ "$rc" = 0 ] || sed 's/^/ | /' "$TMP/fixer.out" -MANIFEST="$REPO/.gitbot-fleet/enrolment.json" -check "fixer: the manifest is a regular file" yes "$([ -f "$MANIFEST" ] && [ ! -L "$MANIFEST" ] && echo yes || echo no)" -check "fixer: the manifest parses as JSON" yes "$(jq -e . "$MANIFEST" >/dev/null 2>&1 && echo yes || echo no)" -WF="$REPO/.github/workflows/fork-governance.yml" -check "fixer: the fleet job calls the fleet reusable at the fleet ref" yes \ - "$(grep -qE "uses: hyperpolymath/gitbot-fleet/\.github/workflows/fleet-report-back-reusable\.yml@$FLEET_REF\$" "$WF" && echo yes || echo no)" -check "fixer: hypatia waits for the fleet job" yes \ - "$(awk '/^ hypatia:/{p=1;next} p&&/^ [a-z]/{exit} p' "$WF" | grep -qE '^ needs: \[fleet\]$' && echo yes || echo no)" -set +e; STANDARDS_DIR="$STD" FLEET_REF="$FLEET_REF" PANIC_ATTACK_SHA=2222222222222222222222222222222222222222 \ - bash "$FIXER" "$REPO" > "$TMP/fixer2.out" 2>&1; rc2=$?; set -e -check "fixer: a second run is a no-op that still exits 0" 0 "$rc2" -check "fixer: a second run inserts no second fleet job" 1 "$(grep -cE '^ fleet:' "$WF")" - -ENROLLED=$(jq -r '.reporters | to_entries[] | select(.value.enrolled == true) | .key' "$MANIFEST" | sort | tr '\n' ' ' | sed 's/ $//') -echo " enrolled: $ENROLLED" - -# ── 3. the producer, with stub adapters ────────────────────────────────────── -BIN="$TMP/bin"; mkdir -p "$BIN" -stub() { printf '#!/usr/bin/env bash\nset -u\n%s\n' "$2" > "$BIN/$1"; chmod +x "$BIN/$1"; } -stub rhodibot 'echo "{\"subject\":\"$3\",\"criteria\":1,\"scored\":1,\"not_applicable\":0,\"counts\":{\"satisfied\":1,\"relocated\":0,\"deprecated\":0,\"missing\":0},\"findings\":[]}"' -stub finishing-bot 'echo "{\"license\":{\"findings\":[]},\"placeholder\":{\"findings\":[]},\"release\":{\"findings\":[]}}"' -stub glambot 'echo "{\"visual\":{\"findings\":[]},\"seo\":{\"findings\":[]}}"' -stub cipherbot 'echo "[]"' -stub panicbot 'echo "[]"' -stub panic-attack 'echo stub' -stub seambot 'echo "{\"status\":\"pass\",\"findings\":[],\"summary\":{\"errors\":0}}"' -stub accessibilitybot 'echo "{\"findings\":[]}"' -repo_before=$(cd "$REPO" && find . -type f -exec sha256sum {} + | sort) -OUT="$TMP/report-backs" -ECHIDNA_URL='' bash "$DRIVER" --repo "$REPO" --out "$OUT" --bin-dir "$BIN" --fleet-ref "$FLEET_REF" > "$TMP/driver.out" 2>&1 \ - || { echo " driver exit $?"; sed 's/^/ | /' "$TMP/driver.out"; } -GOT=$(find "$OUT" -name '*.json' -printf '%f\n' 2>/dev/null | sed 's/\.json$//' | sort | tr '\n' ' ' | sed 's/ $//') -check "producer: one envelope per enrolled member, no more" "$ENROLLED" "$GOT" -check "producer: the scanned repository is unchanged" "$repo_before" "$(cd "$REPO" && find . -type f -exec sha256sum {} + | sort)" -for m in $ENROLLED; do - s=$(jq -r .status "$OUT/$m.json") - case "$m" in - echidnabot | sustainabot) check "producer: $m is not_run with a reason" not_run "$s" ;; - *) check "producer: $m ran" ran "$s" ;; - esac -done - -# ── 4. the renderer: before and after ──────────────────────────────────────── -GRID="$GRID_STANDARDS/scripts/verdict-grid.sh" -if [ -n "$GRID_STANDARDS" ] && [ -f "$GRID" ]; then - EMPTY="$TMP/empty"; mkdir -p "$EMPTY" - render() { # report-backs-dir out-dir - (cd "$REPO" && bash "$GRID" render --out "$2" --hypatia /dev/null --panic /dev/null --pons /dev/null \ - --report-backs "$1" --enrolment .gitbot-fleet/enrolment.json --sha 0000000 > /dev/null) - } - render "$EMPTY" "$TMP/grid-before" - render "$OUT" "$TMP/grid-after" - silent() { jq '[.reporters[] | select(.reason == "enrolled, but no report-back was received")] | length' "$1/report.json"; } - n=$(echo "$ENROLLED" | wc -w) - check "renderer, before: every enrolled member is 'enrolled, but no report-back'" "$n" "$(silent "$TMP/grid-before")" - check "renderer, after: no enrolled member is left without a report-back" 0 "$(silent "$TMP/grid-after")" + # 2. the fixer + local rc rc2 + set +e + STANDARDS_DIR="$STD" FLEET_REF="$FLEET_REF" PANIC_ATTACK_SHA=2222222222222222222222222222222222222222 \ + bash "$FIXER" "$REPO" > "$dir/fixer.out" 2>&1 + rc=$? + set -e + check "$label, fixer: exits 0 (manifest written, job inserted and wired, pin reads the artifact)" 0 "$rc" + [ "$rc" = 0 ] || sed 's/^/ | /' "$dir/fixer.out" + local MANIFEST="$REPO/.gitbot-fleet/enrolment.json" + check "$label, fixer: the manifest is a regular file" yes "$([ -f "$MANIFEST" ] && [ ! -L "$MANIFEST" ] && echo yes || echo no)" + check "$label, fixer: the manifest parses as JSON" yes "$(jq -e . "$MANIFEST" >/dev/null 2>&1 && echo yes || echo no)" + local WF="$REPO/.github/workflows/fork-governance.yml" + check "$label, fixer: the fleet job calls the fleet reusable at the fleet ref" yes \ + "$(grep -qE "uses: hyperpolymath/gitbot-fleet/\.github/workflows/fleet-report-back-reusable\.yml@$FLEET_REF\$" "$WF" && echo yes || echo no)" + check "$label, fixer: hypatia waits for the fleet job" yes \ + "$(awk '/^ hypatia:/{p=1;next} p&&/^ [a-z]/{exit} p' "$WF" | grep -qE '^ needs: \[fleet\]$' && echo yes || echo no)" + set +e; STANDARDS_DIR="$STD" FLEET_REF="$FLEET_REF" PANIC_ATTACK_SHA=2222222222222222222222222222222222222222 \ + bash "$FIXER" "$REPO" > "$dir/fixer2.out" 2>&1; rc2=$?; set -e + check "$label, fixer: a second run is a no-op that still exits 0" 0 "$rc2" + check "$label, fixer: a second run inserts no second fleet job" 1 "$(grep -cE '^ fleet:' "$WF")" + + local ENROLLED conditional + ENROLLED=$(jq -r '.reporters | to_entries[] | select(.value.enrolled == true) | .key' "$MANIFEST" | sort | tr '\n' ' ' | sed 's/ $//') + echo " enrolled: $ENROLLED" + conditional=$(jq -r '[.reporters | to_entries[] | select(.value.enrolled == true) | .key + | select(IN("echidnabot", "panicbot", "seambot", "accessibilitybot"))] | sort | join(" ")' "$MANIFEST") + check "$label, fixer: the conditionally enrolled members follow the tree" "$want_conditional" "$conditional" + + # 3. the producer, with stub adapters + local repo_before OUT GOT m s + repo_before=$(cd "$REPO" && find . -type f -exec sha256sum {} + | sort) + OUT="$dir/report-backs" + ECHIDNA_URL='' bash "$DRIVER" --repo "$REPO" --out "$OUT" --bin-dir "$BIN" --fleet-ref "$FLEET_REF" > "$dir/driver.out" 2>&1 \ + || { echo " driver exit $?"; sed 's/^/ | /' "$dir/driver.out"; } + GOT=$(find "$OUT" -name '*.json' -printf '%f\n' 2>/dev/null | sed 's/\.json$//' | sort | tr '\n' ' ' | sed 's/ $//') + check "$label, producer: one envelope per enrolled member, no more" "$ENROLLED" "$GOT" + check "$label, producer: the scanned repository is unchanged" "$repo_before" "$(cd "$REPO" && find . -type f -exec sha256sum {} + | sort)" for m in $ENROLLED; do - st=$(jq -r --arg id "$m" '.reporters[] | select(.id == $id) | .state' "$TMP/grid-after/report.json") + s=$(jq -r .status "$OUT/$m.json") case "$m" in - echidnabot | sustainabot) check "renderer, after: $m (not_run) stays orange, now with its reason" orange "$st" ;; - *) check "renderer, after: $m ran clean and is green" green "$st" ;; + echidnabot | sustainabot) check "$label, producer: $m is not_run with a reason" not_run "$s" ;; + *) check "$label, producer: $m ran" ran "$s" ;; esac done - for m in $(jq -r '.reporters | to_entries[] | select(.value.enrolled == false) | .key' "$MANIFEST"); do - check "renderer: $m (not enrolled) is grey" grey \ - "$(jq -r --arg id "$m" '.reporters[] | select(.id == $id) | .state' "$TMP/grid-after/report.json")" - done -else - echo "SKIP: STANDARDS_DIR not set or has no scripts/verdict-grid.sh; renderer stage not run" -fi + + # 4. the renderer: before and after + local GRID="$GRID_STANDARDS/scripts/verdict-grid.sh" + if [ -n "$GRID_STANDARDS" ] && [ -f "$GRID" ]; then + local EMPTY="$dir/empty" n st + mkdir -p "$EMPTY" + render() { # report-backs-dir out-dir + (cd "$REPO" && bash "$GRID" render --out "$2" --hypatia /dev/null --panic /dev/null --pons /dev/null \ + --report-backs "$1" --enrolment .gitbot-fleet/enrolment.json --sha 0000000 > /dev/null) + } + render "$EMPTY" "$dir/grid-before" + render "$OUT" "$dir/grid-after" + silent() { jq '[.reporters[] | select(.reason == "enrolled, but no report-back was received")] | length' "$1/report.json"; } + n=$(echo "$ENROLLED" | wc -w) + check "$label, renderer, before: every enrolled member is 'enrolled, but no report-back'" "$n" "$(silent "$dir/grid-before")" + check "$label, renderer, after: no enrolled member is left without a report-back" 0 "$(silent "$dir/grid-after")" + for m in $ENROLLED; do + st=$(jq -r --arg id "$m" '.reporters[] | select(.id == $id) | .state' "$dir/grid-after/report.json") + case "$m" in + echidnabot | sustainabot) check "$label, renderer, after: $m (not_run) stays orange, now with its reason" orange "$st" ;; + *) check "$label, renderer, after: $m ran clean and is green" green "$st" ;; + esac + done + for m in $(jq -r '.reporters | to_entries[] | select(.value.enrolled == false) | .key' "$MANIFEST"); do + check "$label, renderer: $m (not enrolled) is grey" grey \ + "$(jq -r --arg id "$m" '.reporters[] | select(.id == $id) | .state' "$dir/grid-after/report.json")" + done + else + echo "SKIP: STANDARDS_DIR not set or has no scripts/verdict-grid.sh; renderer stage not run" + fi +} + +canary bare bare "" +canary featured featured "accessibilitybot echidnabot panicbot seambot" echo echo "enrolment canary: $pass passed, $fail failed"