diff --git a/.github/workflows/actions.lock b/.github/workflows/actions.lock index 60e83b07..38f208b8 100644 --- a/.github/workflows/actions.lock +++ b/.github/workflows/actions.lock @@ -43,7 +43,6 @@ workflows: '.github/workflows/labels.yml': [] '.github/workflows/learning-loop.yml': - 'actions/checkout@v7.0.1' - - 'actions/upload-artifact@v7.0.2' '.github/workflows/lock-sync-gate.yml': [] '.github/workflows/mirror.yml': [] '.github/workflows/pages.yml': diff --git a/.github/workflows/fleet-report-back.yml b/.github/workflows/fleet-report-back.yml index 7dfe15fa..c22aaf5e 100644 --- a/.github/workflows/fleet-report-back.yml +++ b/.github/workflows/fleet-report-back.yml @@ -81,6 +81,11 @@ jobs: - name: Run the driver's test suite against the real renderer run: STANDARDS_DIR="$PWD/.standards-grid" bash tests/fleet-report-back-test.sh + # The enrolment rollout end to end on a synthetic repository: fixer, then + # driver, then the real renderer, before and after (tests/enrolment-canary-test.sh). + - name: Enrolment canary (fixer, driver and renderer, end to end) + run: STANDARDS_DIR="$PWD/.standards-grid" bash tests/enrolment-canary-test.sh + - name: Render the grid from the envelopes and fail on any silent member env: HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }} diff --git a/PROOF-NEEDS.adoc b/PROOF-NEEDS.adoc index 99b67ce8..3cedb9fd 100644 --- a/PROOF-NEEDS.adoc +++ b/PROOF-NEEDS.adoc @@ -1,40 +1,356 @@ -== Proof Requirements +// SPDX-License-Identifier: CC-BY-SA-4.0 +// SPDX-FileCopyrightText: 2025-2026 Jonathan D.A. Jewell += Proof status: gitbot-fleet +:toc: +:toc-placement: preamble -=== Current state +This is the proof-status file of `hyperpolymath/gitbot-fleet` in the sense of +`3-practice/PROOF-POLICY.adoc` §5 (hyperpolymath/standards). It replaces the +pre-policy template of 2026-03-29. -* ABI directory exists (template-level) -* No dangerous patterns found -* 100K lines; bot fleet for repository quality enforcement -* Claims: "`formal mathematical/statistical verification`" +* *Date*: 2026-11-17. +* *Measured against*: gitbot-fleet `origin/main` at + `7907099b4e6912059ad71a4897005463d214f0bf`. +* *Tokens*: the closed vocabulary of PROOF-POLICY §5: `proven`, + `proven-with-postulates`, `tested-only`, `argued`, `open`. +* *Prover for each obligation*: PROOF-POLICY §1. Rust code is verified with + Creusot (Pearlite contracts) and Verus by default, on the code itself (§1.2); + APIs, ABIs and engineering properties with Idris2; Ada with SPARK; theory + with Agda, rechecked in Rocq and Lean. -=== What needs proving +== Census -* *Bot action safety*: Prove that automated bot actions (PRs, issues, -fixes) are reversible and do not destroy repository state -* *Confidence threshold correctness*: Prove that the -robot-repo-automaton confidence thresholds correctly gate automated -merges (no auto-merge below threshold) -* *Rule conflict resolution*: Prove that when multiple bots (rhodibot, -echidnabot, sustainabot, etc.) produce conflicting recommendations, the +15 rows: 0 `proven`, 0 `proven-with-postulates`, 4 `tested-only`, 4 `argued`, +7 `open`. + +No row is `proven` here. The four `argued` rows are echidnabot's theory +files, whose kernel receipts were taken in the standalone echidnabot +repository, not in this one (see the trust-boundary column). No checker +(Agda, Idris2, Lean, Rocq) was available when this file was written, so no +proof file in this repository was checked by the author of this file. + +== Ledger: proof files tracked in this repository + +All eleven tracked proof files sit under `bots/echidnabot/`, a copy of +`hyperpolymath/echidnabot`. Seven of them are obligations (rows below); four +are prover fixtures (next section). + +The authoritative status of every `bots/echidnabot/` row lives in +`docs/PROOF-STATUS.adoc` of the standalone echidnabot repository, read at +echidnabot `origin/main` `7a5b8c48ffef8236b06b7174d855d88124b7f380`. The copy +in this repository does not carry that file. When the two disagree, echidnabot's +file governs and this one is wrong. + +[cols="2,1,2,2,3",options="header"] +|=== +|Obligation |Status |Checker (version, flags) |Where |Trust boundary + +|T-RECEIPT: a general evidence token does not imply its claim +(`no-unconditional-reflection`, `receive-with-soundness`) +|`argued` +|None run here. echidnabot's receipt R-AGDA: Agda 2.8.0.2, `--safe --without-K +--cubical-compatible --exact-split --no-sized-types --no-guardedness +--no-libraries --no-default-libraries --ignore-interfaces`, session receipt of +2026-11-09 at echidnabot `1443adfabb98a9e0daea562bc8dbe764442c14cc` +|`bots/echidnabot/proofs/agda/ReceiptBoundary.agda` +|The file here is byte-identical to the file that receipt hashed (sha256 +`276ee4b5...0c9014`, compared on 2026-11-17). Echidnabot records it as +`proven`, with no postulate in the import cone. This repository re-ran +nothing, so it records `argued`. Grep for escape hatches finds none in the +file. Authoritative status: echidnabot `docs/PROOF-STATUS.adoc`, row T-RECEIPT. + +|T-RECEIPT-LEAN: Lean restatement of T-RECEIPT +|`argued` +|None run here. echidnabot receipt R-LEAN: Lean 4.35.0, `lean -o` plus +`#print axioms` on the three theorems +|`bots/echidnabot/proofs/theory/lean/ReceiptBoundary.lean` +|Byte-identical to the receipt input (sha256 `00ea31f0...28858f`). Echidnabot +records no axioms. The statement match with the Agda source is `BRIDGE-PENDING` +there: agent-drafted, awaiting owner review, so the leg does not count towards a +PWG. Dedukti transport was tried upstream and failed; this is a re-proof. +Authoritative status: echidnabot `docs/PROOF-STATUS.adoc`, row T-RECEIPT-LEAN. + +|T-RECEIPT-ROCQ: Rocq restatement of T-RECEIPT +|`argued` +|None run here. echidnabot receipt R-ROCQ: Coq 8.18.0 (not a Rocq 9 release), +`coqc -q -Q . Recheck` plus `Print Assumptions` +|`bots/echidnabot/proofs/theory/rocq/ReceiptBoundary.v` +|Byte-identical to the receipt input (sha256 `ca65d1dd...f3e66d`). Same +`BRIDGE-PENDING` state as the Lean leg. Authoritative status: echidnabot +`docs/PROOF-STATUS.adoc`, row T-RECEIPT-ROCQ. + +|Fixture identity: `(A : Set) -> A -> A`, a semantic reconstruction of the +constructive identity schema used by the Rocq and Lean prover fixtures +|`argued` +|None run here. echidnabot receipt R-AGDA (same invocation as T-RECEIPT) +|`bots/echidnabot/proofs/agda/FixtureIdentity.agda` +|Byte-identical to the receipt input (sha256 `9c7b80b1...6a91fb`). Not witness +transport: Agda `Set` and Rocq/Lean `Prop` differ, and echidnabot keeps the +correspondence `open`. Authoritative status: echidnabot `docs/PROOF-STATUS.adoc`, +section "Pre-existing proof material". + +|ABI `Types.idr`: echidnabot ABI types +|`open` +|None. Idris2 is the §1 prover for an ABI; none was run +|`bots/echidnabot/src/abi/Types.idr` +|Three `{{...}}` template placeholders remain, so the module does not describe +an instantiated echidnabot ABI. Byte-identical to echidnabot `origin/main`. +Echidnabot's PROOF-STATUS records the ABI scaffold as `open` ("ABI scaffold +debt (not verified)"). Do not read this file as verification. + +|ABI `Layout.idr`: layout contracts +|`open` +|None. Idris2 is the §1 prover; none was run +|`bots/echidnabot/src/abi/Layout.idr` +|Three unresolved holes (`?fieldsAlignedProof` at line 140, +`?exampleFieldsAligned` at line 161, `?offsetInBoundsProof` at line 178) and +two template placeholders. Echidnabot's PROOF-STATUS notes that +`offsetInBounds` quantifies over arbitrary fields and cannot be proved as +written. Holes are not to be replaced by postulates. + +|ABI `Foreign.idr`: foreign-function declarations +|`open` +|None. Idris2 is the §1 prover; none was run +|`bots/echidnabot/src/abi/Foreign.idr` +|Fourteen `{{...}}` template placeholders; the symbols are placeholders, not +exported echidnabot symbols. The only `believe_me` match is in a comment that +bans it. Echidnabot's PROOF-STATUS records the scaffold as `open`. +|=== + +=== Escape-hatch search + +Searched on 2026-11-17 with `grep` over the eleven proof files for +`postulate`, `sorry`, `Admitted`, `admit`, `believe_me`, `assert_total`, +`assert_smaller`, Agda `{! !}` holes, Idris `?name` holes, `trustMe` and +`axiom`. + +* The four obligation files in `proofs/agda/` and `proofs/theory/` contain no + escape hatch. The only `axiom` matches are `#print axioms` and "axioms or + admitted obligations" in comments of the Lean and Rocq files. +* `src/abi/Layout.idr` contains the three holes listed above. `Types.idr` and + `Foreign.idr` contain only template placeholders. +* `proofs/coq/admitted_stub.v` (`Admitted.`) and `proofs/lean/sorry_stub.lean` + (`sorry`) contain escape hatches on purpose (next section). + +== Not obligations + +These four files are prover fixtures for echidnabot's CI, test material for a +multi-prover system (PROOF-POLICY §3, last bullet). They are not rows and are +not counted in the census. Their escape hatches must stay intact. + +* `bots/echidnabot/proofs/coq/admitted_stub.v`: deliberate `Admitted.`, a + negative fixture for echidnabot's axiom scanner. +* `bots/echidnabot/proofs/lean/sorry_stub.lean`: deliberate `sorry`, a negative + fixture checked by echidnabot's `protocol_contract` test. +* `bots/echidnabot/proofs/coq/trivial_ok.v` and + `bots/echidnabot/proofs/lean/trivial_ok.lean`: trivial identity proofs used + as positive fixtures. + +The JSON files in `bots/echidnabot/proofs/test_fixtures/` are webhook inputs, +not proofs. `idris-dependents/tests/fixtures/*.ipkg` are package fixtures for +an Idris-dependents test, not proofs of this repository. + +== Ledger: report-back producer (tested-only) + +The report-back producer is the part of this repository that the Hypatia +verdict grid reads (`docs/REPORT-BACK.adoc`): `scripts/fleet-report-back.sh`, +`scripts/fleet-report-back-build.sh` and the rollout fixer +`scripts/fix-fleet-enrolment.sh`. It is shell. No Creusot, SPARK or Idris2 +artefact exists for it, so the evidence is test evidence only, and each row is +`tested-only` with the bounds the test file actually covers. + +Observed on 2026-11-17 at the commit above (pass counts are the lines each +script itself reports, with 0 failed in every run): + +* `STANDARDS_DIR= bash tests/fleet-report-back-test.sh`: + 229 passed, 0 failed for its own checks, then it runs the build-helper suite + (below) and the run totals 298 `PASS` lines and 0 `FAIL` lines. The renderer + round-trip checks ran (no `SKIP` line); the standards checkout was not + pinned to a commit. +* `bash tests/fix-fleet-enrolment-test.sh`: 139 passed, 0 failed. +* `bash tests/fleet-report-back-build-test.sh`: 69 passed, 0 failed. These 69 + are part of the 298 above, not additional to them. + +[cols="3,1,2,2,3",options="header"] +|=== +|Obligation |Status |Checker (version, flags) |Where |Bounds and trust boundary + +|R-ENVELOPE: `fleet-report-back.sh` writes one envelope per enrolled member. +A missing binary, a crash without JSON, a timeout, or an output of the wrong +shape is an `error` envelope with a reason, never a clean `ran` +(`docs/REPORT-BACK.adoc` §4, last paragraph) +|`tested-only` +|`bash tests/fleet-report-back-test.sh`, 229 checks (298 with the build suite); +bash with `jq` +|`scripts/fleet-report-back.sh` +|Bounds: the adapters are stubs that print the JSON shape of the real adapters +(taken from a live run on 2026-10-27), not the seven Rust crates. Cases: only +`enrolled: true` members run; an unreadable manifest runs nothing; missing +binary, crash without JSON and timeout are `error` with a reason; exit 1 with +valid JSON stays `ran`, and on the generic findings path any other non-zero +status is `error`, while a panicbot or ponsbot status object (`error` or +`not_run` with a reason, as panicbot prints with exit 3 or 4) keeps its own +status unless the adapter timed out; empty scans, +malformed output and path normalisation; `echidnabot` and `sustainabot` are +`not_run` with a reason; an unknown member is `error`; the driver never writes +`not_applicable`; the envelope is key-sorted compact JSON with one final LF; +the scanned tree is unchanged. Not covered: real adapter behaviour, other +operating systems, other `jq` or bash versions. The grid-side properties (how +an envelope becomes a row colour) are proved in hyperpolymath/standards +`3-practice/fork-governance/proofs/ReportBack.idr`, which is being added on the +standards branch `claude/gitbot-fleet-feedback-0vdo64` and is *pending*, not +merged. They are not re-proved here. The renderer round-trip in the test +exercises the real standards renderer on these envelopes and is also +`tested-only`. + +|R-SLUG: member names are restricted to the slug `^[a-z][a-z0-9-]{0,63}$`, so a +hostile manifest key cannot steer the build or a path +|`tested-only` +|`bash tests/fleet-report-back-test.sh`; `bash tests/fleet-report-back-build-test.sh` +|`scripts/fleet-report-back.sh` (member filter, line 279); +`scripts/fleet-report-back-build.sh` (crate-name filter) +|Bounds: a 64-character slug is accepted and a 65-character one rejected, in +both planning and execution; a leading digit, uppercase, underscore and a dot +are rejected and warned about; a manifest key containing a tab and `../` +sequences, and a key with a space, are dropped with a warning and write +nothing; an explicit `--member ../escape` writes nothing; a path in the crate +list of the build helper is skipped. Sampled names, not an exhaustive or +property-based check of the pattern. Grid-side trust boundary as in +R-ENVELOPE. + +|R-WIRED: `fix-fleet-enrolment.sh` exits 0 only when the lane that reads the +artifact and the job that produces it are both wired. Exit 0 does not +establish that `actions.lock` is in sync: the fixer reports a lock that needs +regenerating (exit 3) only on the run that inserts the job, so a later run can +exit 0 over a stale lock, and the repository's own lock-sync gate is the check +for that. Exit 3 means written or checked but not wired end to end; exit 2 is +a refusal or invalid input and changes nothing +|`tested-only` +|`bash tests/fix-fleet-enrolment-test.sh`, 141 checks; `STANDARDS_DIR=… bash tests/enrolment-canary-test.sh`, 56 checks over two synthetic repositories, one bare and one that enrols the conditional members (fixer, then driver, then the real renderer at standards `9ae23875`, end to end). Measured 2026-11-18 on the commit that records these counts (the canary's second fixture lands in the same commit), not at the ledger's `7907099` +|`scripts/fix-fleet-enrolment.sh` +|Bounds: offline fixture repositories and fake standards checkouts, in the +numbered sections 0 to 18: canonical and non-canonical workflows, owner +detection, existing `needs` forms (list, scalar, block list, an item that +merely starts with `fleet`, a job merely named `fleet`), a pin whose Hypatia +reusable does or does not read the artifact, a pin that lacks the producer +(refused), a remote pin with no standards checkout or without the pinned +commit (reported unverified, not `ok`), refusals that change nothing, an +existing manifest left alone, a malformed panic-attack pin rejected before any +file is created, and a second run changing nothing. Case 13 reports a skip on +a shallow clone; no skip occurred in this run. No real GitHub repository, +workflow run or `gh actions-lock` was exercised. The canary harness chains the +fixer, the driver (adapters stubbed) and standards' renderer on one synthetic +repository: before, every enrolled row reads "enrolled, but no report-back"; +after, none does. It does not exercise GitHub's startup rules for the inserted +job. Whether a deployed repository is actually wired is a live fact these tests +cannot establish. + +|R-BUILD: `fleet-report-back-build.sh`: a crate that fails to build is a +warning and later an `error` row, never a dead run +|`tested-only` +|`bash tests/fleet-report-back-build-test.sh`, 69 checks (also run by the +report-back test) +|`scripts/fleet-report-back-build.sh` +|Bounds: Cargo and `timeout` are stubbed, so no Rust crate is built and no +network is used. Cases: a failed crate's partial binary is not copied and its +warning names the report-back consequence; an all-failed build publishes +nothing and counts both failures; aliases are deduplicated and members without +crates skipped; a path in the crate list is skipped; for `ponsbot`, the +install is copied, a cache hit does not reinstall, a failed install is a +warning, a non-commit value or a commit differing from the pin never reaches +Cargo, an unreadable pin installs nothing, and a spent build budget skips the +install. Real compile failures, timeouts of real builds and cache corruption +are not covered. Grid-side trust boundary as in R-ENVELOPE. +|=== + +== Ledger: open obligations + +These four obligations come from the pre-policy file. Each concerns the +behaviour of first-party code, so under PROOF-POLICY §1 the code mapping takes +precedence: Rust code is verified with Creusot (Pearlite contracts) and Verus +on the code itself (§1.2), and the cross-bot contract that stands apart from any one +implementation is an Idris2 interface. The earlier suggestion of Lean 4 for +"algebraic properties of confidence scoring" is not policy. If the confidence +scoring is stated as a theory theorem, true of any implementation, the source +is Agda, rechecked in Rocq and Lean; a statement about this repository's +functions is engineering and stays with Creusot. + +None has a checker run against it, and none has a receipt. Each stays `open` +until a verifier discharges it. + +[cols="3,1,2,2,3",options="header"] +|=== +|Obligation |Status |Checker (version, flags) |Where |Trust boundary + +|O-ACTION: bot action safety. Automated actions (pull requests, issues, fixes) +are reversible and do not destroy repository state +|`open` +|Rust: Creusot with Pearlite contracts, and Verus, none run. The interface +contract between a bot and the fleet: Idris2, not started +|`robot-repo-automaton/` (Rust), `scripts/dispatch-runner.sh` and `scripts/fix-*.sh` +(shell, no verifier maps to shell) +|Reversibility of a GitHub operation depends on GitHub's behaviour and the +repository's branch protection, so it would be an `ASSUMED` boundary +specification, not something this repository proves. The shell fix scripts are +guarded only by tests and the Repo Integrity Guard +(`.github/workflows/repo-integrity-guard.yml`), which are not proofs. + +|O-THRESHOLD: confidence-threshold gating. Nothing is auto-merged or +auto-executed below its threshold (Safety Triangle: eliminate at 0.95 and +above, substitute at 0.85 and above, control below) +|`open` +|Rust: Creusot with Pearlite contracts, and Verus, none run. If the scoring +order is stated as theory: Agda, rechecked in Rocq and Lean, none started +|`robot-repo-automaton/` (Rust); `scripts/dispatch-runner.sh` (shell, thresholds +read from findings) +|The confidence values are produced by scanners outside this repository +(hypatia), so the property can only be stated relative to the input value +(`ASSUMED` boundary). Where the gate is in shell, no verifier maps to it, and +the obligation would need the gate moved into Rust first. + +|O-CONFLICT: when several bots produce conflicting recommendations, the resolution is deterministic and documented -* *Rate limiting*: Prove bots respect GitHub API rate limits and do not -cause cascading failures across the fleet +|`open` +|Rust arbiter: Creusot with Pearlite contracts, and Verus, none run. The +ordering and tie-break contract: Idris2, not started +|`shared-context/` (Rust crate for inter-bot communication) and +`fleet-coordinator.sh` +|Determinism over the coordinator's inputs only; the order in which concurrent +bots finish is outside the property and would be an `ASSUMED` boundary. -=== Recommended prover +|O-RATE: bots respect GitHub API rate limits and do not cause cascading +failures across the fleet +|`open` +|Rust: Creusot with Pearlite contracts, and Verus, none run +|Rate-limit handling in the Rust crates under `bots/` and `robot-repo-automaton/` +|Only a local bound on the bot's own request counts is provable. GitHub's +limits, clocks and failure behaviour are `ASSUMED` boundaries, and "no +cascading failure" is a fleet-level property that needs a stated model first. +|=== -* *Idris2* — For bot action safety and confidence threshold properties -* *Lean4* — For algebraic properties of confidence scoring if modeled as -an ordered semiring +Using Creusot and Verus together does not make two independent witnesses +(policy §1.2). -=== Priority +== Not verified when this file was written -* *MEDIUM* — The fleet manages automated repository changes. Incorrect -bot behavior at scale could damage many repos. The "`formal -verification`" claim should be substantiated. +* No Agda, Idris2, Lean or Rocq checker was available, so the `argued` rows + rest on echidnabot's receipts plus a byte-identity comparison, and the + receipts themselves were not reproduced. +* The state of `ReportBack.idr` on the standards branch was not checked + beyond its being pending; no claim here depends on it. +* The census counts the 15 rows above and nothing else: the report-back and + open rows cover the properties that this file names, not the whole of the + repository's shell and Rust code, which has no proof coverage. + +== History === Template ABI Cleanup (2026-03-29) -Template ABI removed – was creating false impression of formal -verification. The removed files (Types.idr, Layout.idr, Foreign.idr) +Template ABI removed: it was creating a false impression of formal +verification. The removed files (`Types.idr`, `Layout.idr`, `Foreign.idr`) contained only RSR template scaffolding with unresolved -\{\{PROJECT}}/\{\{AUTHOR}} placeholders and no domain-specific proofs. +`+{{PROJECT}}+`/`+{{AUTHOR}}+` placeholders and no domain-specific proofs. + +The same three file names are tracked again at the date above, under +`bots/echidnabot/src/abi/`, as echidnabot's own scaffold. They still carry +template placeholders and holes, and are recorded `open` above. diff --git a/scripts/fix-fleet-enrolment.sh b/scripts/fix-fleet-enrolment.sh index c7b49bf1..37d10017 100755 --- a/scripts/fix-fleet-enrolment.sh +++ b/scripts/fix-fleet-enrolment.sh @@ -156,68 +156,29 @@ robot-repo-automaton hypatia' fi - # jq 1.6 does NOT preserve object key order in output regardless of input method. - # Use Python (which preserves dict insertion order since 3.7) to generate the - # manifest with keys in canonical priority order. - # Convert shell true/false to Python True/False - py_rust=$( [ "$has_rust" = "true" ] && echo "True" || echo "False" ) - py_seams=$( [ "$has_seams" = "true" ] && echo "True" || echo "False" ) - py_html=$( [ "$has_html" = "true" ] && echo "True" || echo "False" ) - py_proofs=$( [ "$has_proofs" = "true" ] && echo "True" || echo "False" ) - - python3 -c " -import json -import sys - -has_rust = $py_rust -has_seams = $py_seams -has_html = $py_html -has_proofs = $py_proofs - -priority_order = '''$priority_order'''.strip().split('\n') - -reporters = {} -for member in priority_order: - if member == 'rhodibot': - reporters[member] = {'title': 'rhodibot (RSR compliance)', 'enrolled': True} - elif member == 'echidnabot': - if has_proofs: - reporters[member] = {'title': 'echidnabot (formal verification)', 'enrolled': True} - else: - reporters[member] = {'enrolled': False, 'reason': 'no proof files in the tree'} - elif member == 'sustainabot': - reporters[member] = {'enrolled': False, 'reason': 'reserved fleet slot; no adapter exists'} - elif member == 'panicbot': - if has_rust: - reporters[member] = {'title': 'panicbot (panic recovery, error handling)', 'enrolled': True} - else: - reporters[member] = {'enrolled': False, 'reason': 'no Cargo.toml in the tree; panic-attack scans Rust'} - elif member == 'glambot': - reporters[member] = {'title': 'glambot (presentation quality)', 'enrolled': True} - elif member == 'seambot': - if has_seams: - reporters[member] = {'title': 'seambot (ABI/FFI seam checks)', 'enrolled': True} - else: - reporters[member] = {'enrolled': False, 'reason': 'no seam register is declared (spec/seams/seam-register.json)'} - elif member == 'finishingbot': - reporters[member] = {'title': 'finishingbot (release readiness)', 'enrolled': True} - elif member == 'accessibilitybot': - if has_html: - reporters[member] = {'title': 'accessibilitybot (WCAG conformance)', 'enrolled': True} - else: - reporters[member] = {'enrolled': False, 'reason': 'no HTML files in the tree'} - elif member == 'cipherbot': - reporters[member] = {'title': 'cipherbot (cryptographic hygiene)', 'enrolled': True} - elif member == 'robot-repo-automaton': - reporters[member] = {'enrolled': False, 'reason': 'not a fleet reporter; executor role only'} - elif member == 'hypatia': - reporters[member] = {'enrolled': False, 'reason': 'not a fleet reporter; engine role only'} - -manifest = {'schema': 1, 'reporters': reporters} -# Compact output with single trailing newline (I-JSON/JCS) -json.dump(manifest, sys.stdout, separators=(',', ':')) -print() -" > "$MANIFEST" + # jq builds objects in insertion order, so reducing over the priority list + # writes the manifest in canonical priority order. Only fleet REPORTERS get + # an entry: robot-repo-automaton (executor) and hypatia (engine) are in the + # priority file for the Arbiter, but they are not grid rows, and a manifest + # entry named `hypatia` would give the grid a second row with the scanner's id. + printf '%s\n' "$priority_order" | jq -R . | jq -sc \ + --argjson rust "$has_rust" --argjson seams "$has_seams" \ + --argjson html "$has_html" --argjson proofs "$has_proofs" ' + def on($title): {title: $title, enrolled: true}; + def off($reason): {enrolled: false, reason: $reason}; + def entry: + if . == "rhodibot" then on("rhodibot (RSR compliance)") + elif . == "echidnabot" then (if $proofs then on("echidnabot (formal verification)") else off("no proof files in the tree") end) + elif . == "sustainabot" then off("reserved fleet slot; no adapter exists") + elif . == "panicbot" then (if $rust then on("panicbot (panic recovery, error handling)") else off("no Cargo.toml in the tree; panic-attack scans Rust") end) + elif . == "glambot" then on("glambot (presentation quality)") + elif . == "seambot" then (if $seams then on("seambot (ABI/FFI seam checks)") else off("no seam register is declared (spec/seams/seam-register.json)") end) + elif . == "finishingbot" then on("finishingbot (release readiness)") + elif . == "accessibilitybot" then (if $html then on("accessibilitybot (WCAG conformance)") else off("no HTML files in the tree") end) + elif . == "cipherbot" then on("cipherbot (cryptographic hygiene)") + else null end; + {schema: 1, reporters: (reduce (.[] | select(length > 0)) as $m ({}; ($m | entry) as $e | if $e == null then . else . + {($m): $e} end))} + ' > "$MANIFEST" echo " CREATED .gitbot-fleet/enrolment.json (rust=$has_rust seams=$has_seams html=$has_html proofs=$has_proofs; edit it — enrol what is true of this repository)" fi diff --git a/tests/enrolment-canary-test.sh b/tests/enrolment-canary-test.sh new file mode 100755 index 00000000..065305b8 --- /dev/null +++ b/tests/enrolment-canary-test.sh @@ -0,0 +1,204 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell (hyperpolymath) +# +# enrolment-canary-test.sh — the fleet-enrolment rollout, end to end, on +# synthetic repositories instead of estate ones (owner decision 2026-11-18). +# +# One chain, each stage the real script: +# 1. a repository with a fork-governance Hypatia lane and no manifest; +# 2. scripts/fix-fleet-enrolment.sh writes the manifest, inserts the fleet +# job and wires `hypatia: needs: [fleet]` (exit 0: no actions.lock here); +# 3. scripts/fleet-report-back.sh runs the members the manifest enrols +# (adapters stubbed with their real JSON shapes) and writes envelopes; +# 4. standards' scripts/verdict-grid.sh renders the grid twice: without the +# envelopes (the "before" state: every enrolled row orange, "enrolled, +# but no report-back was received") and with them (the "after" state: no +# enrolled row left orange for want of evidence). +# +# The chain runs on two repositories: +# * bare: one Julia file, so only the always-enrolled members run; +# * featured: a Cargo.toml, a seam register, an HTML page and a proof file, +# so panicbot, seambot, accessibilitybot and echidnabot are enrolled too, +# and echidnabot (no ECHIDNA_URL) takes the not_run path: orange, with its +# reason, in the "after" grid. +# +# What this does NOT test: GitHub's startup rules for the inserted job (the +# lock records, reusable resolution). Those need a live run on a real repo. +# +# Stage 4 needs STANDARDS_DIR (a standards checkout carrying +# scripts/verdict-grid.sh and scripts/lib/verdict-grid.jq); without it the +# renderer stage is skipped and says so. +# +# Run: STANDARDS_DIR=/path/to/standards bash tests/enrolment-canary-test.sh +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +FIXER="$ROOT/scripts/fix-fleet-enrolment.sh" +DRIVER="$ROOT/scripts/fleet-report-back.sh" +GRID_STANDARDS="${STANDARDS_DIR:-}" +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT +export PIN_FIXER_LOCK_DIR="$TMP/locks"; mkdir -m 700 "$PIN_FIXER_LOCK_DIR" + +pass=0; fail=0 +check() { # name expected actual + if [ "$2" = "$3" ]; then pass=$((pass + 1)); echo "ok $1" + else fail=$((fail + 1)); echo "FAIL $1"; echo " want: $2"; echo " got: $3"; fi +} + +# ── a standards checkout whose Hypatia reusable reads the artifact ─────────── +# The fixer verifies the pinned reusable declares `report-backs-artifact`; a +# synthetic checkout keeps that check real without needing the network. +STD="$TMP/standards"; mkdir -p "$STD/.github/workflows"; git -C "$STD" init -q +printf 'on:\n workflow_call:\n inputs:\n report-backs-artifact:\n type: string\n default: fleet-report-backs\njobs:\n scan:\n steps:\n - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333\n' \ + > "$STD/.github/workflows/hypatia-scan-reusable.yml" +git -C "$STD" add -A +git -C "$STD" -c user.email=t@example.invalid -c user.name=t -c commit.gpgsign=false commit -q -m reads-artifact +STD_SHA=$(git -C "$STD" rev-parse HEAD) +FLEET_REF=$(git -C "$ROOT" rev-parse HEAD) + +# ── stub adapters, printing the JSON shapes of the real ones ───────────────── +BIN="$TMP/bin"; mkdir -p "$BIN" +stub() { printf '#!/usr/bin/env bash\nset -u\n%s\n' "$2" > "$BIN/$1"; chmod +x "$BIN/$1"; } +stub rhodibot 'echo "{\"subject\":\"$3\",\"criteria\":1,\"scored\":1,\"not_applicable\":0,\"counts\":{\"satisfied\":1,\"relocated\":0,\"deprecated\":0,\"missing\":0},\"findings\":[]}"' +stub finishing-bot 'echo "{\"license\":{\"findings\":[]},\"placeholder\":{\"findings\":[]},\"release\":{\"findings\":[]}}"' +stub glambot 'echo "{\"visual\":{\"findings\":[]},\"seo\":{\"findings\":[]}}"' +stub cipherbot 'echo "[]"' +stub panicbot 'echo "[]"' +stub panic-attack 'echo stub' +stub seambot 'echo "{\"status\":\"pass\",\"findings\":[],\"summary\":{\"errors\":0}}"' +stub accessibilitybot 'echo "{\"findings\":[]}"' + +# canary LABEL FEATURES(bare|featured) EXPECTED-CONDITIONAL-MEMBERS +canary() { + local label="$1" features="$2" want_conditional="$3" + local dir="$TMP/$label" REPO="$TMP/$label/repo" + echo "── $label ──" + + # 1. the canary repository: a Hypatia lane, no manifest + mkdir -p "$REPO/.github/workflows" "$REPO/src" + git -C "$REPO" init -q + git -C "$REPO" remote add origin "https://github.com/hyperpolymath/canary-$label.git" + echo 'x' > "$REPO/src/a.jl" + if [ "$features" = featured ]; then + printf '[package]\nname = "canary"\nversion = "0.1.0"\nedition = "2021"\n' > "$REPO/Cargo.toml" + mkdir -p "$REPO/spec/seams" "$REPO/docs" "$REPO/verification" + printf '{"seams":[]}\n' > "$REPO/spec/seams/seam-register.json" + printf 'c\n' > "$REPO/docs/index.html" + printf 'module Canary where\n' > "$REPO/verification/Canary.agda" + fi + cat > "$REPO/.github/workflows/fork-governance.yml" < "$dir/fixer.out" 2>&1 + rc=$? + set -e + check "$label, fixer: exits 0 (manifest written, job inserted and wired, pin reads the artifact)" 0 "$rc" + [ "$rc" = 0 ] || sed 's/^/ | /' "$dir/fixer.out" + local MANIFEST="$REPO/.gitbot-fleet/enrolment.json" + check "$label, fixer: the manifest is a regular file" yes "$([ -f "$MANIFEST" ] && [ ! -L "$MANIFEST" ] && echo yes || echo no)" + check "$label, fixer: the manifest parses as JSON" yes "$(jq -e . "$MANIFEST" >/dev/null 2>&1 && echo yes || echo no)" + local WF="$REPO/.github/workflows/fork-governance.yml" + check "$label, fixer: the fleet job calls the fleet reusable at the fleet ref" yes \ + "$(grep -qE "uses: hyperpolymath/gitbot-fleet/\.github/workflows/fleet-report-back-reusable\.yml@$FLEET_REF\$" "$WF" && echo yes || echo no)" + check "$label, fixer: hypatia waits for the fleet job" yes \ + "$(awk '/^ hypatia:/{p=1;next} p&&/^ [a-z]/{exit} p' "$WF" | grep -qE '^ needs: \[fleet\]$' && echo yes || echo no)" + set +e; STANDARDS_DIR="$STD" FLEET_REF="$FLEET_REF" PANIC_ATTACK_SHA=2222222222222222222222222222222222222222 \ + bash "$FIXER" "$REPO" > "$dir/fixer2.out" 2>&1; rc2=$?; set -e + check "$label, fixer: a second run is a no-op that still exits 0" 0 "$rc2" + check "$label, fixer: a second run inserts no second fleet job" 1 "$(grep -cE '^ fleet:' "$WF")" + + local ENROLLED conditional + ENROLLED=$(jq -r '.reporters | to_entries[] | select(.value.enrolled == true) | .key' "$MANIFEST" | sort | tr '\n' ' ' | sed 's/ $//') + echo " enrolled: $ENROLLED" + conditional=$(jq -r '[.reporters | to_entries[] | select(.value.enrolled == true) | .key + | select(IN("echidnabot", "panicbot", "seambot", "accessibilitybot"))] | sort | join(" ")' "$MANIFEST") + check "$label, fixer: the conditionally enrolled members follow the tree" "$want_conditional" "$conditional" + + # 3. the producer, with stub adapters + local repo_before OUT GOT m s + repo_before=$(cd "$REPO" && find . -type f -exec sha256sum {} + | sort) + OUT="$dir/report-backs" + ECHIDNA_URL='' bash "$DRIVER" --repo "$REPO" --out "$OUT" --bin-dir "$BIN" --fleet-ref "$FLEET_REF" > "$dir/driver.out" 2>&1 \ + || { echo " driver exit $?"; sed 's/^/ | /' "$dir/driver.out"; } + GOT=$(find "$OUT" -name '*.json' -printf '%f\n' 2>/dev/null | sed 's/\.json$//' | sort | tr '\n' ' ' | sed 's/ $//') + check "$label, producer: one envelope per enrolled member, no more" "$ENROLLED" "$GOT" + check "$label, producer: the scanned repository is unchanged" "$repo_before" "$(cd "$REPO" && find . -type f -exec sha256sum {} + | sort)" + for m in $ENROLLED; do + s=$(jq -r .status "$OUT/$m.json") + case "$m" in + echidnabot | sustainabot) check "$label, producer: $m is not_run with a reason" not_run "$s" ;; + *) check "$label, producer: $m ran" ran "$s" ;; + esac + done + + # 4. the renderer: before and after + local GRID="$GRID_STANDARDS/scripts/verdict-grid.sh" + if [ -n "$GRID_STANDARDS" ] && [ -f "$GRID" ]; then + local EMPTY="$dir/empty" n st + mkdir -p "$EMPTY" + render() { # report-backs-dir out-dir + (cd "$REPO" && bash "$GRID" render --out "$2" --hypatia /dev/null --panic /dev/null --pons /dev/null \ + --report-backs "$1" --enrolment .gitbot-fleet/enrolment.json --sha 0000000 > /dev/null) + } + render "$EMPTY" "$dir/grid-before" + render "$OUT" "$dir/grid-after" + silent() { jq '[.reporters[] | select(.reason == "enrolled, but no report-back was received")] | length' "$1/report.json"; } + n=$(echo "$ENROLLED" | wc -w) + check "$label, renderer, before: every enrolled member is 'enrolled, but no report-back'" "$n" "$(silent "$dir/grid-before")" + check "$label, renderer, after: no enrolled member is left without a report-back" 0 "$(silent "$dir/grid-after")" + for m in $ENROLLED; do + st=$(jq -r --arg id "$m" '.reporters[] | select(.id == $id) | .state' "$dir/grid-after/report.json") + case "$m" in + echidnabot | sustainabot) check "$label, renderer, after: $m (not_run) stays orange, now with its reason" orange "$st" ;; + *) check "$label, renderer, after: $m ran clean and is green" green "$st" ;; + esac + done + for m in $(jq -r '.reporters | to_entries[] | select(.value.enrolled == false) | .key' "$MANIFEST"); do + check "$label, renderer: $m (not enrolled) is grey" grey \ + "$(jq -r --arg id "$m" '.reporters[] | select(.id == $id) | .state' "$dir/grid-after/report.json")" + done + else + echo "SKIP: STANDARDS_DIR not set or has no scripts/verdict-grid.sh; renderer stage not run" + fi +} + +canary bare bare "" +canary featured featured "accessibilitybot echidnabot panicbot seambot" + +echo +echo "enrolment canary: $pass passed, $fail failed" +[ "$fail" -eq 0 ] diff --git a/tests/fix-fleet-enrolment-test.sh b/tests/fix-fleet-enrolment-test.sh index b9ffd72f..8b62b9d7 100755 --- a/tests/fix-fleet-enrolment-test.sh +++ b/tests/fix-fleet-enrolment-test.sh @@ -95,18 +95,14 @@ canonical_workflow "$NEW_SHA" > "$A/.github/workflows/fork-governance.yml"; chmo set +e; bash "$FIXER" "$A" > "$TMP/a.out" 2>&1; rc=$?; set -e check "A: exit 0 — wired end to end" 0 "$rc" check_grep "A: manifest created" 'CREATED \.gitbot-fleet/enrolment\.json' "$TMP/a.out" -check "A: manifest is canonical priority order compact JSON with one final LF (I-JCS as Python produces it)" \ - "$(python3 -c " -import json, sys -with open('$A/.gitbot-fleet/enrolment.json') as f: - data = json.load(f) -# Re-serialize with same priority order as fix-fleet-enrolment.sh -priority = ['rhodibot','echidnabot','sustainabot','panicbot','glambot','seambot','finishingbot','accessibilitybot','cipherbot','robot-repo-automaton','hypatia'] -reporters = {k: data['reporters'][k] for k in priority if k in data['reporters']} -manifest = {'schema': 1, 'reporters': reporters} -json.dump(manifest, sys.stdout, separators=(',', ':')) -print() -")" "$(cat "$A/.gitbot-fleet/enrolment.json")" +check "A: manifest is canonical priority order compact JSON with one final LF (I-JCS)" \ + "$(jq -c '{schema: 1, reporters: (.reporters as $r | reduce ("rhodibot","echidnabot","sustainabot","panicbot","glambot","seambot","finishingbot","accessibilitybot","cipherbot") as $k ({}; if $r | has($k) then . + {($k): $r[$k]} else . end))}' "$A/.gitbot-fleet/enrolment.json")" \ + "$(cat "$A/.gitbot-fleet/enrolment.json")" +check "A: manifest lists exactly the nine fleet reporters, in priority order" \ + "rhodibot echidnabot sustainabot panicbot glambot seambot finishingbot accessibilitybot cipherbot" \ + "$(jq -r '.reporters | keys_unsorted | join(" ")' "$A/.gitbot-fleet/enrolment.json")" +check "A: non-reporter roles (hypatia, robot-repo-automaton) get no manifest entry" "false false" \ + "$(jq -r '[(.reporters | has("hypatia")), (.reporters | has("robot-repo-automaton"))] | map(tostring) | join(" ")' "$A/.gitbot-fleet/enrolment.json")" check "A: manifest ends with exactly one LF" "1" "$(tail -c1 "$A/.gitbot-fleet/enrolment.json" | wc -l | tr -d ' ')" check "A: every-repo members enrolled" "true true true true" \ "$(jq -r '[.reporters.rhodibot, .reporters.finishingbot, .reporters.glambot, .reporters.cipherbot][] | .enrolled' "$A/.gitbot-fleet/enrolment.json" | paste -sd' ' -)"