From b893d4ddbfbdc8d043f765b6869145f28a18dccc Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 03:23:57 +0000 Subject: [PATCH 01/10] feat(scripts): fix-a2ml-via-kyx.sh converts .a2ml with kyX or fails loudly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fixing half of the owner's 2026-11-17 directive to rid the estate of A2ML (the finding half is rhodibot's P1 rule, #690). Conversion is kyX's front end only (D313): the script never hand-converts, never mints an id and never edits a deed. For each tracked .a2ml it either converts it (kyX output staged as the deed, the .a2ml staged for deletion, a PR body written) or refuses it with every reason (a fingerprinted P1 issue body written, the tree untouched). A converted file passes all four gates: 1. kyX exits 0 (fail-closed field accounting under the mapping spec); 2. kyX drops nothing (any spec §4 drop is "not lossless" unless the owner opts in with A2ML_ACCEPT_SPEC_DROPS=1); 3. standards deed_lint --ecosystem --root passes (canon grammar; a k9 output, when a front end emits one, must pass k9-validate --strict); 4. scripts/lib/a2ml-census.js: every non-empty source value is in the deed's --ast-json, checked independently of kyX. Only ECOSYSTEM.a2ml at a canonical location, in a repo with exactly one chora deed, reaches kyX; every other family is refused with the mapping spec that will decide it. Never commits, pushes or publishes; reports go outside the tree. Idempotent. `--check` is the outcome predicate. Registered as recipe-a2ml-kyx / RetiredA2ML. Not dispatched: the fleet dispatch is quarantined, and campaigns/A2ML-KYX-CAMPAIGN.adoc sets out kyX's real status, the prerequisites and the rollout options for the owner. Tests: 42 (scripts/tests/fix-a2ml-via-kyx-test.sh), stub kyX, the REAL standards deed_lint; every gate refuses and leaves the tree untouched. Dogfood on a clone of this repo: 0 converted, 42 refused (no chora deed). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhFeexmokLqLRfZNce --- campaigns/A2ML-KYX-CAMPAIGN.adoc | 137 ++++++++++ scripts/fix-a2ml-via-kyx.sh | 331 +++++++++++++++++++++++++ scripts/fix-script-registry.json | 6 +- scripts/lib/a2ml-census.js | 203 +++++++++++++++ scripts/tests/fix-a2ml-via-kyx-test.sh | 181 ++++++++++++++ 5 files changed, 856 insertions(+), 2 deletions(-) create mode 100644 campaigns/A2ML-KYX-CAMPAIGN.adoc create mode 100755 scripts/fix-a2ml-via-kyx.sh create mode 100644 scripts/lib/a2ml-census.js create mode 100644 scripts/tests/fix-a2ml-via-kyx-test.sh diff --git a/campaigns/A2ML-KYX-CAMPAIGN.adoc b/campaigns/A2ML-KYX-CAMPAIGN.adoc new file mode 100644 index 00000000..4877e2ca --- /dev/null +++ b/campaigns/A2ML-KYX-CAMPAIGN.adoc @@ -0,0 +1,137 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 += A2ML Retirement Campaign (kyX): 2026-11-17 +:toc: +:toclevels: 2 + +== Purpose + +The owner's directive of 2026-11-17: "make sure that a2ml is sorted and +converted opportunistically and automatically when a bot sees it as a +priority one task, I need to rid the estate of this". + +A2ML is retired (owner rulings D99, D269c, D312, D313). Two pieces carry the +directive: + +* *Seeing it.* rhodibot reports every tracked `.a2ml` as a priority-1 finding: + `rhodibot.no-a2ml` (severity `retired`, `critical` on the verdict grid) in + `canon`, and `no-*.a2ml` (RSR-A2ML-001, Required in every policy pack) in the + GitHub App check. (Draft PR #690.) +* *Sorting it.* `scripts/fix-a2ml-via-kyx.sh` (recipe `recipe-a2ml-kyx`, + category `RetiredA2ML`) converts what kyX can convert losslessly and fails + loudly, with a P1 issue body, on everything else. It never hand-converts. + +== Status + +*Nothing is dispatched.* The fleet dispatch is under quarantine +(`docs/AUTOMATION-QUARANTINE.adoc`), and this change does not touch the +interlock. There is no manifest, because a manifest needs an estate scan, and +the quarantine document forbids a scan as a workaround. Running the campaign is +the owner's decision; the options are below. + +== What kyX can do today (checked 2026-11-17) + +* `metadatastician/kyx` main `731e63a`: CI green, 39 acceptance checks, every + output checked with standards `deed_lint` at a pinned commit. +* *One front end: `ECOSYSTEM.a2ml`.* Every other family is "not started" + (kyX README, Status). kyX refuses any field its spec does not list (the real + hypatia ECOSYSTEM is refused with 19 fields named), and refuses a repository + with no chora deed: it reads the id and never mints one (U1). +* *No released artifact.* A consumer builds the AffineScript compiler at kyX's + pinned commit (OCaml) and then kyX. `metadatastician/kyx` is *private*, so a + fleet runner needs read access to it. + +Dogfood on this repository (`fix-a2ml-via-kyx.sh` on a scratch clone): +*0 converted, 42 refused* across 13 families, because gitbot-fleet has no +`gitbot-fleet_chora.deed` and only its 3 ECOSYSTEM files are kyX-routable at +all. Expect the estate to look the same: until kyX grows front ends, almost +every repository gets a P1 issue and very few get a PR. + +== The fixer's gates + +Every converted file passes all four, or nothing in the tree changes for it: + +. kyX exits 0 (fail-closed field accounting under the mapping spec). +. kyX drops nothing. Spec §4 lets kyX drop empty values, comment lines and + `@directives`; by default any drop is a refusal ("not lossless"). + `A2ML_ACCEPT_SPEC_DROPS=1` accepts spec-sanctioned drops and lists each in + the PR (an owner decision, see Q-LOSSLESS below). +. The deed passes standards `deed_lint.js --ecosystem --root ` (deed.abnf + plus ecosystem.abnf). A `.k9`/`.k9.ncl` output, when a future front end emits + one, must pass `k9-validate.sh --strict`. +. The value census (`scripts/lib/a2ml-census.js`): every non-empty source value + appears in the deed (`deed_lint --ast-json`), independently of kyX. It errs + towards refusing. + +Bots open PRs, never push: the fixer stages the change and writes the PR and +P1 issue bodies; it never commits, pushes or publishes. A refusal's issue body +carries a fingerprint, so a caller can find an open issue for the same refusal +instead of opening a second one. + +== Prerequisites (each blocks a real run) + +[cols="1,3,2"] +|=== +| # | Prerequisite | Whose + +| P-1 | A kyX build a runner can fetch: a release asset `kyx.bun.js` with its SHA-256 and source commit (preferred), or a read token for the private repo. Building AffineScript on every run is minutes per repository. | owner / kyx +| P-2 | The quarantine lifted for this recipe, by a reviewed change, with PR-only token scope (branches, PRs, issues; no default-branch push, which branch protection enforces independently). | owner +| P-3 | A chora deed in each target repository (kyX never mints one). This repository has none. | per repo +| P-4 | Issue labels `P1` and `a2ml` exist where issues are opened. | per repo +|=== + +== Rollout options (owner question Q-ROLLOUT) + +Rated, most elegant and correct first; the second number is dependable and +secure/private. + +[cols="1,4,1,1"] +|=== +| Option | What | Elegance | Safety + +| A. Repo-local opportunistic job (recommended) +| `fix-fleet-enrolment.sh` adds an `a2ml` step to each repo's own fleet job: + when rhodibot's report-back carries `rhodibot.no-a2ml`, the job runs the + fixer in that repo's own CI and opens a draft PR or the P1 issue with the + repo's own `GITHUB_TOKEN`. No cross-repo credential, truly "when a bot sees + it", and enrolment is already per repo. +| 5 | 4 + +| B. Waved campaign manifest through `dispatch-runner` +| A JSONL manifest built from the rhodibot findings the report-backs already + hold (no new scan), dispatched in waves: 3 pilot repos with an ECOSYSTEM.a2ml + and a chora deed, then 25, then all. Needs P-2 for the runner. +| 4 | 4 + +| C. Report only, owner converts +| Keep rhodibot's P1 finding; the owner runs the fixer locally per repo. No + automation to qualify. +| 2 | 5 + +| D. kyX-side batch (alternative path) +| kyX grows a `--repo` batch front end and its own CI opens the PRs; the fleet + only reports. Puts conversion and its proof in one place, but moves work to + kyX. +| 4 | 3 +|=== + +Whichever is chosen, issue volume is the cost to weigh: about 500 repositories +and one front end today means hundreds of P1 issues. A variant of any option +opens *one tracking issue per family in metadatastician/kyx* and a short P1 +issue per repository that links to it. + +== Owner questions + +* *Q-ROLLOUT:* A, B, C or D above, and whether to open per-repo P1 issues now + or wait for more kyX front ends. +* *Q-LOSSLESS:* is a spec §4 drop (empty value, comment line, `@directive`) a + loss? Default: yes, refuse. Alternative: accept and list it in the PR. + +== Beyond this repository (to track) + +* kyX: a released `kyx.bun.js` (P-1); front ends for STATE, META, AGENTIC, + NEUROSYM, PLAYBOOK, CLADE and the manifest; mapping specs for ANCHOR, + rsr-profile, bot directives and the contractiles. +* standards: the vendored `rsr-criteria-v2.a2ml` here is a pinned copy of the + live standards canon; it goes when standards converts the canon (A7, P2.5). +* gitbot-fleet itself: a `gitbot-fleet_chora.deed` (P-3), then its own 42 + `.a2ml` files are the pilot. diff --git a/scripts/fix-a2ml-via-kyx.sh b/scripts/fix-a2ml-via-kyx.sh new file mode 100755 index 00000000..1ca25c69 --- /dev/null +++ b/scripts/fix-a2ml-via-kyx.sh @@ -0,0 +1,331 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell +# +# fix-a2ml-via-kyx.sh — convert a repository's retired .a2ml files with kyX, +# or fail loudly. Owner directive 2026-11-17: rid the estate of A2ML, +# opportunistically, when a bot sees it as a priority-1 task (rhodibot +# `rhodibot.no-a2ml` / RSR-A2ML-001). +# +# Conversion is kyX's front end and nothing else (D313): this script NEVER +# hand-converts, never guesses a field, and never edits a deed. For each +# tracked .a2ml it either +# +# * converts it: kyX translates it under the standards mapping spec, and the +# output must pass every gate below; the deed is staged, the .a2ml is +# staged for deletion, and a PR body is written; or +# * refuses it, with every reason, and writes a P1 issue body. Nothing in the +# tree changes for a refused file. +# +# The gates, all of them, for every converted file: +# 1. kyX exits 0. kyX is fail-closed: a table, field or value shape its +# mapping spec does not list is refused with its line (exit 1). +# 2. kyX drops nothing. kyX reports what spec §4 lets it drop (empty values, +# comment lines, @directives) on stderr; any such line is information +# that does not reach the deed, so it is a refusal. A2ML_ACCEPT_SPEC_DROPS=1 +# accepts drops the spec sanctions and lists each in the PR body instead +# (an owner decision, off by default). +# 3. The deed parses under the canon grammar: standards +# `deed_lint.js --ecosystem --root ` (deed.abnf + ecosystem.abnf). +# A .k9/.k9.ncl output, when a kyX front end emits one, must pass +# standards `k9-validate.sh --strict` (K9_VALIDATE); none does today. +# 4. No value is lost: scripts/lib/a2ml-census.js checks that every non-empty +# scalar value in the source appears in the deed (deed_lint --ast-json). +# Independent of kyX, and conservative: a value a spec rewrites reads as +# lost, so the error is always towards refusing. +# +# kyX translates ECOSYSTEM.a2ml only (metadatastician/kyx README, "Status"); +# every other family is refused here as "no kyX front end", with the mapping +# spec that will decide it when there is one. An ECOSYSTEM.a2ml is converted +# only at a canonical location (.machine_readable/descriptiles/, 6a2/, or the +# retired direct .machine_readable/ layout), into +# .machine_readable/descriptiles/ECOSYSTEM.deed, and only when the repository +# has exactly one chora deed: kyX reads the id from it and never mints one. +# +# Writes in the repository: only the staged deed and the staged deletion of +# the .a2ml it replaces. Never commits, never pushes, never opens anything: +# the caller opens a PR on a branch (never the default branch) and the P1 +# issue. Report files go to A2ML_REPORT_DIR (default: a fresh temp directory, +# printed), never into the tree: +# a2ml-report.json {converted: [...], refused: [...], ...} +# a2ml-pr.md when something was converted +# a2ml-p1-issue.md when something was refused (fingerprinted, so a caller +# can find an open issue for the same refusal) +# +# Environment (needed only when a file is routed to kyX): +# KYX path to kyX's built Bun module (_build/kyx/kyx.bun.js) +# KYX_COMMIT the 40-hex kyX commit it was built from (recorded) +# DEED_LINT path to standards 1-formats/deed/tools/deed_lint.js +# STANDARDS_COMMIT the 40-hex standards commit DEED_LINT is from (recorded) +# AFFIRMED YYYY-MM-DD for :affirmed (default: today, UTC) +# K9_VALIDATE standards 1-formats/k9/tools/k9-validate.sh (k9 outputs) +# +# Idempotent: a converted file is gone on the next run; a refused one is +# refused again with the same fingerprint. +# +# Exit: 0 no .a2ml left (converted, or none to begin with) · 2 refused and +# nothing converted (P1 issue) · 3 partial: some converted (PR), some refused +# (P1 issue) — a dispatcher must not record 2 or 3 as a clean success · 64 +# usage · 69 a file needs kyX and a tool or its provenance is missing. +# +# outcome-predicate: --check +# +# Usage: +# fix-a2ml-via-kyx.sh [finding.json] +# fix-a2ml-via-kyx.sh --check print the number of tracked .a2ml +set -euo pipefail + +SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CENSUS="$SELF_DIR/lib/a2ml-census.js" + +usage() { sed -n '/^# Usage:/,/^set -euo/p' "$0" | sed '$d; s/^# \{0,1\}//' >&2; exit 64; } +die() { printf 'fix-a2ml-via-kyx: %s\n' "$1" >&2; exit "${2:-2}"; } + +# The tracked .a2ml files, NUL-safe, sorted, case-insensitive suffix. +a2ml_files() { + git -C "$1" ls-files -z | tr '\0' '\n' | grep -iE '\.a2ml$' | LC_ALL=C sort || true +} + +check_mode=0 +if [[ "${1:-}" == "--check" ]]; then check_mode=1; shift; fi +[[ $# -ge 1 && $# -le 2 ]] || usage +REPO="$1" +[[ -d "$REPO" ]] || die "not a directory: $REPO" 64 +git -C "$REPO" rev-parse --is-inside-work-tree >/dev/null 2>&1 || die "not a git work tree: $REPO" 64 +REPO="$(cd "$REPO" && git rev-parse --show-toplevel)" + +if [[ "$check_mode" -eq 1 ]]; then + a2ml_files "$REPO" | grep -c . || true + exit 0 +fi + +mapfile -t FILES < <(a2ml_files "$REPO") +if [[ "${#FILES[@]}" -eq 0 ]]; then + echo "fix-a2ml-via-kyx: no .a2ml in $REPO; nothing to do" + exit 0 +fi + +for f in "${FILES[@]}"; do + case "$f" in + *$'\n'*|*$'\t'*|-*) die "refusing a tracked path with a control character or leading dash: ${f@Q}" 2 ;; + esac +done + +REPORT_DIR="${A2ML_REPORT_DIR:-}" +if [[ -z "$REPORT_DIR" ]]; then REPORT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/a2ml-kyx.XXXXXX")"; fi +mkdir -p "$REPORT_DIR" +case "$(cd "$REPORT_DIR" && pwd)/" in "$REPO"/*) die "A2ML_REPORT_DIR must be outside the repository" 64 ;; esac +WORK="$(mktemp -d "${TMPDIR:-/tmp}/a2ml-kyx-work.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT + +: > "$WORK/converted.jsonl" +: > "$WORK/refused.jsonl" + +refuse() { # path family spec reason... + local path="$1" family="$2" spec="$3"; shift 3 + jq -nc --arg path "$path" --arg family "$family" --arg spec "$spec" \ + --args '{path:$path, family:$family, spec:(if $spec=="" then null else $spec end), reasons:$ARGS.positional}' \ + "$@" >> "$WORK/refused.jsonl" +} + +# family -> "familyspeckyx(0|1)"; the table rhodibot's +# canon/a2ml.rs reports from, so the finding and the fixer agree. +family_of() { + local lower name + lower="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" + name="${lower##*/}" + case "$name" in + ecosystem.a2ml) printf 'ECOSYSTEM\tmeta-ecosystem-to-repo-deed.adoc\t1\n' ;; + meta.a2ml) printf 'META\tmeta-ecosystem-to-repo-deed.adoc\t0\n' ;; + state.a2ml) printf 'STATE\tstate-v1-decision.adoc\t0\n' ;; + agentic.a2ml) printf 'AGENTIC\tagentic-neurosym-playbook-to-repo-deed.adoc\t0\n' ;; + neurosym.a2ml) printf 'NEUROSYM\tagentic-neurosym-playbook-to-repo-deed.adoc\t0\n' ;; + playbook.a2ml) printf 'PLAYBOOK\tagentic-neurosym-playbook-to-repo-deed.adoc\t0\n' ;; + clade.a2ml) printf 'CLADE\tclade-to-repo-deed.adoc\t0\n' ;; + 0-ai-manifest.a2ml) printf 'AI-MANIFEST\tai-manifest-decision.adoc\t0\n' ;; + anchor.a2ml) printf 'ANCHOR\t\t0\n' ;; + rsr-profile.a2ml) printf 'RSR-PROFILE\t\t0\n' ;; + rsr-criteria-v2.a2ml) printf 'RSR-CRITERIA\trsr-criteria-to-praxis-deed.adoc\t0\n' ;; + *) + if [[ "$lower" == *bot_directives/* ]]; then printf 'BOT-DIRECTIVES\t\t0\n' + elif [[ "$name" == *file.a2ml ]]; then printf 'CONTRACTILE\t\t0\n' + else printf 'UNMAPPED\t\t0\n'; fi ;; + esac +} + +# Canonical ECOSYSTEM.a2ml sources (anything else is refused, not guessed). +is_canonical_ecosystem() { + case "$1" in + .machine_readable/descriptiles/ECOSYSTEM.a2ml|.machine_readable/6a2/ECOSYSTEM.a2ml|.machine_readable/ECOSYSTEM.a2ml) return 0 ;; + *) return 1 ;; + esac +} +TARGET=".machine_readable/descriptiles/ECOSYSTEM.deed" + +need_tools() { + local missing=() + [[ -n "${KYX:-}" && -f "${KYX:-}" ]] || missing+=("KYX (kyX's built kyx.bun.js)") + [[ "${KYX_COMMIT:-}" =~ ^[0-9a-f]{40}$ ]] || missing+=("KYX_COMMIT (40-hex)") + [[ -n "${DEED_LINT:-}" && -f "${DEED_LINT:-}" ]] || missing+=("DEED_LINT (standards deed_lint.js)") + [[ "${STANDARDS_COMMIT:-}" =~ ^[0-9a-f]{40}$ ]] || missing+=("STANDARDS_COMMIT (40-hex)") + command -v bun >/dev/null 2>&1 || missing+=("bun") + command -v jq >/dev/null 2>&1 || missing+=("jq") + if [[ "${#missing[@]}" -gt 0 ]]; then + printf 'fix-a2ml-via-kyx: a file needs kyX, and these are missing: %s\n' "${missing[*]}" >&2 + printf 'fix-a2ml-via-kyx: nothing was changed; this is an operator error, not the repository'"'"'s\n' >&2 + exit 69 + fi +} + +AFFIRMED="${AFFIRMED:-$(date -u +%Y-%m-%d)}" +[[ "$AFFIRMED" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || die "AFFIRMED must be YYYY-MM-DD" 64 +ACCEPT_DROPS="${A2ML_ACCEPT_SPEC_DROPS:-0}" +[[ "$ACCEPT_DROPS" == 0 || "$ACCEPT_DROPS" == 1 ]] || die "A2ML_ACCEPT_SPEC_DROPS must be 0 or 1" 64 + +# Exactly one chora deed at the root (D312), or none to convert against. +shopt -s nullglob +choras=("$REPO"/*_chora.deed) +shopt -u nullglob + +eco_sources=() +for f in "${FILES[@]}"; do + IFS=$'\t' read -r fam spec kyx <<< "$(family_of "$f")" + if [[ "$kyx" != 1 ]]; then + if [[ -n "$spec" ]]; then + refuse "$f" "$fam" "$spec" "kyX has no front end for $fam yet; $spec decides where it goes. Do not hand-convert (D313)." + else + refuse "$f" "$fam" "" "no mapping spec decides $fam yet, and kyX has no front end for it. Do not hand-convert (D313)." + fi + continue + fi + if ! is_canonical_ecosystem "$f"; then + refuse "$f" "$fam" "$spec" "not at a canonical descriptile location, so it may describe another tree (a template, a vendored copy); kyX needs the chora deed of the tree it describes." + continue + fi + eco_sources+=("$f") +done + +if [[ "${#eco_sources[@]}" -gt 1 ]]; then + for f in "${eco_sources[@]}"; do + refuse "$f" ECOSYSTEM meta-ecosystem-to-repo-deed.adoc "more than one ECOSYSTEM.a2ml (${eco_sources[*]}) for one $TARGET; which is authoritative is a human decision." + done + eco_sources=() +fi + +for src in "${eco_sources[@]}"; do + spec=meta-ecosystem-to-repo-deed.adoc + if [[ "${#choras[@]}" -ne 1 ]]; then + refuse "$src" ECOSYSTEM "$spec" "kyX reads :canonical-name and the chora id from the repository's one *_chora.deed and never mints one (U1); found ${#choras[@]}." + continue + fi + if [[ -e "$REPO/$TARGET" ]]; then + refuse "$src" ECOSYSTEM "$spec" "$TARGET already exists; it is never overwritten. Reconcile the two by hand-review, not by hand-conversion." + continue + fi + need_tools + out="$WORK/ECOSYSTEM.deed"; rm -f "$out" "$WORK/kyx.err" "$WORK/ast.json" + rc=0 + bun "$KYX" a2ml ecosystem "$REPO/$src" --chora "${choras[0]}" --affirmed "$AFFIRMED" --out "$out" \ + > "$WORK/kyx.out" 2> "$WORK/kyx.err" || rc=$? + mapfile -t kyx_err < "$WORK/kyx.err" + if [[ "$rc" -eq 1 ]]; then + mapfile -t why < <(grep '^kyx: refused: ' "$WORK/kyx.err" | sed 's/^kyx: refused: /kyX refused: /') + [[ "${#why[@]}" -gt 0 ]] || why=("kyX refused it (exit 1) without a reason line: ${kyx_err[*]:-}") + refuse "$src" ECOSYSTEM "$spec" "${why[@]}" + continue + elif [[ "$rc" -ne 0 || ! -s "$out" ]]; then + refuse "$src" ECOSYSTEM "$spec" "kyX failed (exit $rc), so nothing is known about the conversion: ${kyx_err[*]:-no output}" + continue + fi + mapfile -t drops < <(grep '^kyx: dropped' "$WORK/kyx.err" || true) + if [[ "${#drops[@]}" -gt 0 && "$ACCEPT_DROPS" != 1 ]]; then + refuse "$src" ECOSYSTEM "$spec" "not lossless: kyX dropped content the deed does not carry (spec §4); A2ML_ACCEPT_SPEC_DROPS is off:" "${drops[@]}" + continue + fi + if ! bun "$DEED_LINT" --ecosystem --root "$REPO" "$out" > "$WORK/lint.out" 2>&1; then + mapfile -t lint < "$WORK/lint.out" + refuse "$src" ECOSYSTEM "$spec" "kyX's output does not pass standards deed_lint --ecosystem --root (canon grammar):" "${lint[@]}" + continue + fi + for k9 in "$WORK"/*.k9 "$WORK"/*.k9.ncl; do + [[ -e "$k9" ]] || continue + if [[ -z "${K9_VALIDATE:-}" ]] || ! bash "$K9_VALIDATE" --strict "$k9" > "$WORK/k9.out" 2>&1; then + refuse "$src" ECOSYSTEM "$spec" "a k9 output does not pass standards k9-validate.sh --strict (or K9_VALIDATE is unset)" + continue 2 + fi + done + if ! bun "$DEED_LINT" --ast-json "$out" > "$WORK/ast.json" 2> "$WORK/ast.err"; then + refuse "$src" ECOSYSTEM "$spec" "deed_lint --ast-json could not read kyX's output: $(head -c 400 "$WORK/ast.err")" + continue + fi + crc=0 + census_json="$(bun "$CENSUS" check "$REPO/$src" "$WORK/ast.json" 2> "$WORK/census.err")" || crc=$? + if [[ "$crc" -ne 0 ]]; then + if [[ "$crc" -eq 1 ]]; then + mapfile -t lost < <(jq -r '.missing[] | "value not found in the deed: " + (tojson)' <<< "$census_json") + refuse "$src" ECOSYSTEM "$spec" "not lossless: the value census found source values the deed does not carry:" "${lost[@]}" + else + refuse "$src" ECOSYSTEM "$spec" "the value census could not read the source, so losslessness is unproven: $(head -c 400 "$WORK/census.err")" + fi + continue + fi + # Every gate passed: stage the deed and the deletion. + mkdir -p "$REPO/$(dirname "$TARGET")" + cp "$out" "$REPO/$TARGET" + git -C "$REPO" add -- "$TARGET" + git -C "$REPO" rm -q -- "$src" + jq -nc --arg source "$src" --arg target "$TARGET" --arg kyx "$KYX_COMMIT" --arg std "$STANDARDS_COMMIT" \ + --arg affirmed "$AFFIRMED" --argjson census "$census_json" --args \ + '{source:$source, target:$target, kyx_commit:$kyx, standards_commit:$std, affirmed:$affirmed, + values_checked:$census.source_values, spec_drops:$ARGS.positional}' \ + "${drops[@]}" >> "$WORK/converted.jsonl" +done + +repo_name="$(basename "$REPO")" +jq -n --arg repo "$repo_name" --arg accept "$ACCEPT_DROPS" \ + --slurpfile converted "$WORK/converted.jsonl" --slurpfile refused "$WORK/refused.jsonl" \ + '{repo:$repo, accept_spec_drops:($accept=="1"), converted:$converted, refused:$refused}' \ + > "$REPORT_DIR/a2ml-report.json" +rm -f "$REPORT_DIR/a2ml-pr.md" "$REPORT_DIR/a2ml-p1-issue.md" + +n_conv="$(jq '.converted | length' "$REPORT_DIR/a2ml-report.json")" +n_ref="$(jq '.refused | length' "$REPORT_DIR/a2ml-report.json")" + +if [[ "$n_conv" -gt 0 ]]; then + jq -r ' + "## Retire A2ML: kyX conversion\n", + "A2ML is retired (D99, D269c, D312, D313). Each file below was converted by **kyX** under its standards mapping spec. Nothing was converted by hand.\n", + "| Source (deleted) | Deed (added) | kyX | standards | values checked |", + "|---|---|---|---|---|", + (.converted[] | "| `\(.source)` | `\(.target)` | `\(.kyx_commit[0:12])` | `\(.standards_commit[0:12])` | \(.values_checked) |"), + "", + "Gates passed for every file: kyX exit 0 (fail-closed), " + (if .accept_spec_drops then "spec §4 drops accepted by A2ML_ACCEPT_SPEC_DROPS=1 and listed below" else "no content dropped" end) + ", `deed_lint --ecosystem --root` (canon grammar), and the value census (every source value is in the deed).", + (if any(.converted[]; (.spec_drops|length) > 0) then ("\nDropped under spec §4:\n" + ([.converted[] | .spec_drops[] | "- " + .] | join("\n"))) else empty end), + "", + "Review it like any PR: the deed is owner-facing, and :affirmed records the date of this conversion.\n", + "_Opened by gitbot-fleet `scripts/fix-a2ml-via-kyx.sh` (opportunistic A2ML retirement)._" + ' "$REPORT_DIR/a2ml-report.json" > "$REPORT_DIR/a2ml-pr.md" +fi + +if [[ "$n_ref" -gt 0 ]]; then + fp="$(jq -cS '[.refused[] | {path, reasons}]' "$REPORT_DIR/a2ml-report.json" | sha256sum | cut -c1-16)" + jq -r --arg fp "$fp" ' + "", + "## P1: retired A2ML that kyX cannot convert (\(.refused | length) file(s))\n", + "A2ML is retired (D99, D269c, D312, D313), and conversion is kyX'"'"'s front end only (D313). These files could not be converted **losslessly by kyX**, so nothing was changed. **Do not hand-convert them.**\n", + (.refused | group_by(.family)[] | + "### \(.[0].family)" + (if .[0].spec then " — `1-formats/deed/mappings/\(.[0].spec)`" else " — no mapping spec yet" end) + "\n", + (.[] | "- `\(.path)`\n" + ([.reasons[] | " - " + .] | join("\n"))), + ""), + "What unblocks each: a kyX front end for the family (metadatastician/kyx), a mapping spec where there is none, or a fix to the source that kyX names.\n", + "_Opened by gitbot-fleet `scripts/fix-a2ml-via-kyx.sh`; it will not open a second issue for the same refusal._" + ' "$REPORT_DIR/a2ml-report.json" > "$REPORT_DIR/a2ml-p1-issue.md" +fi + +echo "fix-a2ml-via-kyx: $n_conv converted, $n_ref refused; reports in $REPORT_DIR" +if [[ "$n_ref" -gt 0 ]]; then + jq -r '.refused[] | "REFUSED\t\(.path)\t\(.reasons[0])"' "$REPORT_DIR/a2ml-report.json" >&2 +fi +if [[ "$n_ref" -eq 0 ]]; then exit 0; fi +if [[ "$n_conv" -gt 0 ]]; then exit 3; fi +exit 2 diff --git a/scripts/fix-script-registry.json b/scripts/fix-script-registry.json index 6892bae9..7735498b 100644 --- a/scripts/fix-script-registry.json +++ b/scripts/fix-script-registry.json @@ -56,7 +56,8 @@ "RetiredDescriptilePolicy": "fix-retired-descriptile-policy.sh", "ToolchainInputMissing": "fix-rust-toolchain-input.sh", "StaleReusablePin": "fix-stale-reusable-pin.sh", - "LockSelfShaRef": "fix-lock-self-sha-refs.sh" + "LockSelfShaRef": "fix-lock-self-sha-refs.sh", + "RetiredA2ML": "fix-a2ml-via-kyx.sh" }, "by_recipe": { "recipe-actions-allow-all": "fix-actions-policy.sh", @@ -130,7 +131,8 @@ "pin_integrity/self_sha_lock_record": "fix-lock-self-sha-refs.sh", "recipe-workflow-linter-spdx-rule": "fix-workflow-linter-spdx-rule.sh", "recipe-sonarqube-token-guard": "fix-sonarqube-token-guard.sh", - "recipe-workflow-split-comment-keys": "fix-split-comment-keys.sh" + "recipe-workflow-split-comment-keys": "fix-split-comment-keys.sh", + "recipe-a2ml-kyx": "fix-a2ml-via-kyx.sh" } } } diff --git a/scripts/lib/a2ml-census.js b/scripts/lib/a2ml-census.js new file mode 100644 index 00000000..dd249dd5 --- /dev/null +++ b/scripts/lib/a2ml-census.js @@ -0,0 +1,203 @@ +// SPDX-License-Identifier: MPL-2.0 +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell +// +// a2ml-census.js — the value census behind fix-a2ml-via-kyx.sh's "no +// information is lost" gate. Runs under Bun, zero dependencies. +// +// bun a2ml-census.js check SOURCE.a2ml AST.json +// +// AST.json is `bun deed_lint.js --ast-json OUT.deed` (standards). Every +// non-empty scalar VALUE in the A2ML source (strings, numbers, booleans, dates) +// must appear as a value somewhere in the deed. Keys and table names are not +// values (a mapping spec renames them), and comments are not values (kyX's own +// drop report covers those). Prints one JSON object +// {"source_values": N, "missing": [ ... ]} +// and exits 0 when nothing is missing, 1 when something is, 2 on a usage or +// parse error. It is a NECESSARY condition for losslessness, checked +// independently of kyX: it errs towards reporting loss (a value a spec +// legitimately rewrites shows as missing), never towards hiding it. +// +// This is a verifier, not a converter: it never writes a deed. + +import { readFileSync } from "node:fs"; + +function fail(msg) { + process.stderr.write(`a2ml-census: ${msg}\n`); + process.exit(2); +} + +// --- A2ML (the retired TOML-like record dialect) value lexer -------------- + +export function a2mlValues(text) { + const s = text.replace(/\r\n/g, "\n"); + const out = []; + let i = 0; + let line = 1; + const peek = (k = 0) => s[i + k]; + const err = (m) => fail(`A2ML line ${line}: ${m}`); + const skipWs = (newlines) => { + for (;;) { + const c = peek(); + if (c === " " || c === "\t") i++; + else if (newlines && c === "\n") { i++; line++; } + else if (c === "#") { while (i < s.length && peek() !== "\n") i++; } + else return; + } + }; + const basic = () => { + // after the opening quote + let v = ""; + for (;;) { + const c = s[i++]; + if (c === undefined || c === "\n") err("unterminated string"); + if (c === '"') return v; + if (c === "\\") { + const e = s[i++]; + const map = { n: "\n", t: "\t", r: "\r", '"': '"', "\\": "\\", b: "\b", f: "\f" }; + if (e in map) v += map[e]; + else if (e === "u" || e === "U") { + const n = e === "u" ? 4 : 8; + const hex = s.slice(i, i + n); + if (!/^[0-9A-Fa-f]+$/.test(hex) || hex.length !== n) err("bad \\u escape"); + v += String.fromCodePoint(parseInt(hex, 16)); + i += n; + } else err(`bad escape \\${e}`); + } else v += c; + } + }; + const multi = (q) => { + // after the opening triple quote; a newline right after it is trimmed + if (peek() === "\n") { i++; line++; } + let v = ""; + for (;;) { + if (i >= s.length) err("unterminated multi-line string"); + if (s.startsWith(q + q + q, i)) { i += 3; return v; } + const c = s[i++]; + if (c === "\n") line++; + if (q === '"' && c === "\\") { + if (peek() === "\n") { // line-ending backslash: trim the break and indent + while (peek() === "\n" || peek() === " " || peek() === "\t") { if (peek() === "\n") line++; i++; } + continue; + } + const e = s[i++]; + const map = { n: "\n", t: "\t", r: "\r", '"': '"', "\\": "\\", b: "\b", f: "\f" }; + if (e in map) v += map[e]; else err(`bad escape \\${e}`); + } else v += c; + } + }; + const value = () => { + skipWs(false); + const c = peek(); + if (s.startsWith('"""', i)) { i += 3; out.push(multi('"')); return; } + if (s.startsWith("'''", i)) { i += 3; out.push(multi("'")); return; } + if (c === '"') { i++; out.push(basic()); return; } + if (c === "'") { + i++; + const j = s.indexOf("'", i); + const nl = s.indexOf("\n", i); + if (j < 0 || (nl >= 0 && nl < j)) err("unterminated literal string"); + out.push(s.slice(i, j)); + i = j + 1; + return; + } + if (c === "[") { + i++; + for (;;) { + skipWs(true); + if (peek() === "]") { i++; return; } + value(); + skipWs(true); + if (peek() === ",") { i++; continue; } + if (peek() === "]") { i++; return; } + err("expected , or ] in an array"); + } + } + if (c === "{") { + i++; + for (;;) { + skipWs(false); + if (peek() === "}") { i++; return; } + key(); + value(); + skipWs(false); + if (peek() === ",") { i++; continue; } + if (peek() === "}") { i++; return; } + err("expected , or } in an inline table"); + } + } + // bare scalar: bool, number, date + let j = i; + while (j < s.length && !/[\s,\]}#]/.test(s[j])) j++; + const tok = s.slice(i, j); + if (tok === "") err("expected a value"); + out.push(tok); + i = j; + }; + const key = () => { + // a bare, quoted or dotted key, then '=' + skipWs(false); + while (i < s.length && peek() !== "=") { + if (peek() === "\n") err("a key without '='"); + if (peek() === '"') { i++; basic(); continue; } + if (peek() === "'") { const j = s.indexOf("'", i + 1); if (j < 0) err("bad key"); i = j + 1; continue; } + i++; + } + if (peek() !== "=") err("expected '='"); + i++; + }; + while (i < s.length) { + skipWs(true); + if (i >= s.length) break; + if (peek() === "[" || peek() === "@") { + // A table header is a name, not a value. An `@directive(...)` line is + // kyX's drop doctrine's to report (spec §4), so the drop gate covers it. + while (i < s.length && peek() !== "\n") i++; + continue; + } + key(); + value(); + skipWs(false); + if (i < s.length && peek() !== "\n") err("trailing content after a value"); + } + return out; +} + +// --- deed AST (deed_lint.js --ast-json) value collector ------------------- + +export function deedValues(ast) { + const out = []; + const node = (n) => { + if (!Array.isArray(n) || n.length < 2) return; + const [kind, v] = n; + if (kind === "list") { for (const x of v) node(x); return; } + if (kind === "boolean") { out.push(v === "#t" ? "true" : "false"); return; } + if (typeof v === "string") out.push(v); + }; + const items = (xs) => { + for (const it of xs) { + if (it[0] === "field") node(it[2]); + else if (it[0] === "clause") items(it[2]); + } + }; + if (!ast || !Array.isArray(ast.items)) fail("AST has no items (is it deed_lint --ast-json output?)"); + items(ast.items); + return out; +} + +export function census(sourceText, ast) { + const want = a2mlValues(sourceText).filter((v) => v.trim() !== ""); + const have = new Set(deedValues(ast)); + const missing = [...new Set(want.filter((v) => !have.has(v)))]; + return { source_values: want.length, missing }; +} + +if (import.meta.main) { + const [cmd, src, astPath] = process.argv.slice(2); + if (cmd !== "check" || !src || !astPath) fail("usage: a2ml-census.js check SOURCE.a2ml AST.json"); + let text, ast; + try { text = readFileSync(src, "utf8"); } catch (e) { fail(`cannot read ${src}: ${e.message}`); } + try { ast = JSON.parse(readFileSync(astPath, "utf8")); } catch (e) { fail(`cannot read ${astPath}: ${e.message}`); } + const r = census(text, ast); + process.stdout.write(JSON.stringify(r) + "\n"); + process.exit(r.missing.length === 0 ? 0 : 1); +} diff --git a/scripts/tests/fix-a2ml-via-kyx-test.sh b/scripts/tests/fix-a2ml-via-kyx-test.sh new file mode 100644 index 00000000..bd36e1e0 --- /dev/null +++ b/scripts/tests/fix-a2ml-via-kyx-test.sh @@ -0,0 +1,181 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# +# Tests for scripts/fix-a2ml-via-kyx.sh and scripts/lib/a2ml-census.js. +# +# kyX is a STUB here (a Bun script driven by STUB_KYX): these tests pin the +# fixer's gates and outputs, not kyX's translation, which kyX's own acceptance +# suite covers (metadatastician/kyx tests/kyx/run.sh). The canon grammar gate +# uses the REAL standards deed_lint.js: set DEED_LINT, or STANDARDS to a +# metadatastician/standards checkout. Without it the suite does not run and +# exits 2 (NOT RUN), never 0. +# +# Usage: STANDARDS=/path/to/standards bash scripts/tests/fix-a2ml-via-kyx-test.sh +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +FIX="$ROOT/scripts/fix-a2ml-via-kyx.sh" +CENSUS="$ROOT/scripts/lib/a2ml-census.js" +DEED_LINT="${DEED_LINT:-${STANDARDS:+$STANDARDS/1-formats/deed/tools/deed_lint.js}}" +if [[ -z "$DEED_LINT" || ! -f "$DEED_LINT" ]] || ! command -v bun >/dev/null || ! command -v jq >/dev/null; then + echo "NOT RUN: needs bun, jq and the real standards deed_lint.js (set STANDARDS or DEED_LINT)" >&2 + exit 2 +fi +export DEED_LINT +export STANDARDS_COMMIT=b49a33e991f37a2249d873cbdc02c27a08f05d6d +export KYX_COMMIT=731e63ab4c3b02a6e4751de97208ea65eb8bdf6e +export AFFIRMED=2026-11-17 + +pass=0 fail=0 +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +ok() { echo "PASS: $1"; pass=$((pass+1)); } +no() { echo "FAIL: $1"; fail=$((fail+1)); } +check() { if [[ "$2" == "$3" ]]; then ok "$1"; else no "$1 (want ${2@Q}, got ${3@Q})"; fi; } + +CHORA=';; SPDX-License-Identifier: MPL-2.0 +(repo-deed + :schema-version "1.0.0" + :canonical-name "demo" + :repo-uuid #uuid"00000000-0000-8000-8000-0000000000a1" + (identity :owner "hyperpolymath" :primary-forge github :prefixed-name "demo"))' + +SOURCE='# SPDX-License-Identifier: MPL-2.0 +[project] +name = "demo" +purpose = "A demo used by the fixer tests." +[position-in-ecosystem] +ecosystem = "hyperpolymath" +type = "tool" +[[related-projects.projects]] +name = "standards" +relationship = "standard-source" +notes = "Reads its grammar."' + +# The deed text is data with literal quotes, exported to the stub as-is. +# shellcheck disable=SC2089,SC2090 +GOOD_DEED=';; SPDX-License-Identifier: MPL-2.0 +(descriptile-deed + :schema-version "1.0.0" + :canonical-name "demo" + :of-chora #uuid"00000000-0000-8000-8000-0000000000a1" + :affirmed "2026-11-17" :affirmed-by agent + :project "demo" + :ecosystem "hyperpolymath" + :position-type "tool" + :purpose "A demo used by the fixer tests." + (related :name "standards" :relationship standard-source :notes "Reads its grammar."))' + +# The stub: STUB_KYX = ok | refuse | drop | lossy | badgrammar | crash +cat > "$TMP/kyx.js" <<'JS' +import { writeFileSync } from "node:fs"; +const a = process.argv.slice(2); +const out = a[a.indexOf("--out") + 1]; +const mode = process.env.STUB_KYX; +const good = process.env.GOOD_DEED; +if (a[0] !== "a2ml" || a[1] !== "ecosystem" || !a.includes("--chora") || !a.includes("--affirmed")) { + process.stderr.write("kyx: usage\n"); process.exit(2); +} +if (mode === "refuse") { process.stderr.write("kyx: refused: line 3: [extra] is not in §3\nkyx: 1 reason(s); nothing written\n"); process.exit(1); } +if (mode === "crash") { process.stderr.write("kyx: cannot read source\n"); process.exit(2); } +let text = good; +if (mode === "lossy") text = good.replace('"A demo used by the fixer tests."', '"A demo."'); +if (mode === "badgrammar") text = good.replace("standard-source", "depends-on"); +writeFileSync(out, text + "\n"); +if (mode === "drop") process.stderr.write("kyx: dropped by the drop doctrine (spec §4): 1 comment line(s), 0 directive(s)\n"); +process.exit(0); +JS +# shellcheck disable=SC2090 +export KYX="$TMP/kyx.js" GOOD_DEED + +mkrepo() { # name -> path of a fresh repo with chora + sources + local r="$TMP/$1"; rm -rf "$r"; mkdir -p "$r/.machine_readable/descriptiles" "$r/contractiles/must" + git -C "$r" init -q; git -C "$r" config user.email t@t; git -C "$r" config user.name t + printf '%s\n' "$CHORA" > "$r/demo_chora.deed" + printf '%s\n' "$SOURCE" > "$r/.machine_readable/descriptiles/ECOSYSTEM.a2ml" + printf '[state]\nphase = "beta"\n' > "$r/.machine_readable/descriptiles/STATE.a2ml" + printf '# x\n' > "$r/contractiles/must/Mustfile.a2ml" + git -C "$r" add -A; git -C "$r" commit -qm init + printf '%s\n' "$r" +} +run() { # repo -> sets RC, REPORT + REPORT="$TMP/report-$RANDOM"; mkdir -p "$REPORT"; RC=0 + A2ML_REPORT_DIR="$REPORT" bash "$FIX" "$1" > "$TMP/out" 2> "$TMP/err" || RC=$? +} +staged() { git -C "$1" diff --cached --name-status | tr '\t' ' ' | LC_ALL=C sort | tr '\n' ';'; } + +# --- the census on its own --------------------------------------------------- +printf '%s\n' "$GOOD_DEED" > "$TMP/ECOSYSTEM.deed" +bun "$DEED_LINT" --ast-json "$TMP/ECOSYSTEM.deed" > "$TMP/ast.json" +printf '%s\n' "$SOURCE" > "$TMP/src.a2ml" +check "census: every source value is in the good deed" 0 "$(bun "$CENSUS" check "$TMP/src.a2ml" "$TMP/ast.json" >/dev/null; echo $?)" +printf '%s\nextra = "never mapped"\n' "$SOURCE" > "$TMP/src2.a2ml" +check "census: a value the deed lacks is reported" '["never mapped"]' "$(bun "$CENSUS" check "$TMP/src2.a2ml" "$TMP/ast.json" | jq -c .missing || true)" +printf 'a = """\nmulti\nline"""\nb = [ "x", # c\n "y" ]\nc = { d = 1, e = true }\n@directive(skip)\nf = \x27lit\x27\n' > "$TMP/src3.a2ml" +check "census: multi-line, arrays with comments, inline tables, literals; directives skipped" \ + '["multi\nline","x","y","1","true","lit"]' \ + "$(bun -e 'const {a2mlValues} = await import(process.argv[1]); console.log(JSON.stringify(a2mlValues(require("fs").readFileSync(process.argv[2],"utf8"))))' "$CENSUS" "$TMP/src3.a2ml")" +printf 'oops\n' > "$TMP/bad.a2ml" +check "census: an unreadable source is exit 2, not a pass" 2 "$(bun "$CENSUS" check "$TMP/bad.a2ml" "$TMP/ast.json" >/dev/null 2>&1; echo $?)" + +# --- --check (the outcome predicate) ----------------------------------------- +r=$(mkrepo check) +check "--check counts tracked .a2ml" 3 "$(bash "$FIX" --check "$r")" +check "the outcome predicate is declared" 1 "$(grep -cx '# outcome-predicate: --check' "$FIX")" + +# --- a clean conversion, plus refusals for what kyX cannot do ---------------- +r=$(mkrepo ok); STUB_KYX=ok run "$r" +check "partial: ECOSYSTEM converted, STATE and Mustfile refused -> exit 3" 3 "$RC" +check "the deed is staged and the source deletion staged" \ + "A .machine_readable/descriptiles/ECOSYSTEM.deed;D .machine_readable/descriptiles/ECOSYSTEM.a2ml;" "$(staged "$r")" +check "the staged deed is kyX's output, byte for byte" "$(printf '%s\n' "$GOOD_DEED")" "$(cat "$r/.machine_readable/descriptiles/ECOSYSTEM.deed")" +check "nothing is committed" 1 "$(git -C "$r" rev-list --count HEAD)" +check "report: one converted, with provenance" "$KYX_COMMIT" "$(jq -r '.converted[0].kyx_commit' "$REPORT/a2ml-report.json")" +check "report: STATE refused as no front end" STATE "$(jq -r '.refused[] | select(.path|endswith("STATE.a2ml")) | .family' "$REPORT/a2ml-report.json")" +check "report: Mustfile refused as a contractile with no spec" CONTRACTILE "$(jq -r '.refused[] | select(.path|endswith("Mustfile.a2ml")) | .family' "$REPORT/a2ml-report.json")" +check "a PR body is written" 1 "$(grep -c 'kyX conversion' "$REPORT/a2ml-pr.md")" +check "a P1 issue body is written, fingerprinted" 1 "$(grep -c '^ --- scripts/tests/a2ml-census-test.sh | 108 +++++++++++++++++++++++++ scripts/tests/fix-a2ml-via-kyx-test.sh | 29 +++++++ 2 files changed, 137 insertions(+) create mode 100644 scripts/tests/a2ml-census-test.sh diff --git a/scripts/tests/a2ml-census-test.sh b/scripts/tests/a2ml-census-test.sh new file mode 100644 index 00000000..e19fb6dd --- /dev/null +++ b/scripts/tests/a2ml-census-test.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# Unit tests for the value census, independent of kyX and standards. +# Usage: bash scripts/tests/a2ml-census-test.sh (Bun only; normally < 5s). +set -euo pipefail +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +command -v bun >/dev/null || { echo 'NOT RUN: needs bun' >&2; exit 2; } +CENSUS="$ROOT/scripts/lib/a2ml-census.js" bun run - <<'JS' +import assert from 'node:assert/strict'; +import { mkdtempSync, writeFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { spawnSync } from 'node:child_process'; +const { a2mlValues, deedValues, census } = await import(process.env.CENSUS); +let passed = 0; +function test(name, fn) { fn(); console.log(`PASS: ${name}`); passed++; } +const field = (key, kind, value) => ['field', key, [kind, value]]; +const ast = (...items) => ({ items }); +const strings = (...values) => ast(...values.map(v => field('renamed', 'string', v))); + +const lexCases = [ + ['empty input and comments', '# ignored = "value"\r\n\t\r\n', []], + ['headers, directives and keys are not values', '[project]\n[[related.projects]]\n@drop("ignored")\n"key=with equals".name = "kept" # ignored\n', ['kept']], + ['literal strings preserve escapes and comment markers', "a = 'C:\\tmp#file'\nb = '''\nraw\\n\nline'''\n", ['C:\\tmp#file', 'raw\\n\nline']], + ['CRLF normalizes before multiline parsing', 'a = """\r\nfirst\r\nsecond"""\r\n', ['first\nsecond']], + ['nested containers collect only values in order', 'a = [{name = "x", nested = [false, 0]}, ["y",], {}]\n', ['x', 'false', '0', 'y']], + ['bare scalars retain their spelling', 'a = [-1, 1.20, 2026-11-17, true, false]\n', ['-1', '1.20', '2026-11-17', 'true', 'false']], + ['basic escapes decode', String.raw`a = "\b\t\n\f\r\"\\"`, ['\b\t\n\f\r"\\']], + ['Unicode scalar boundaries decode', String.raw`a = "\u0000\uD7FF\uE000\U0010FFFF"`, ['\0\uD7FF\uE000\u{10FFFF}']], + ['continuations trim blank lines and indentation', 'a = """\none \\\n \t\n two"""', ['one two']], +]; +for (const [name, source, expected] of lexCases) test(name, () => assert.deepEqual(a2mlValues(source), expected)); + +test('AST recursively collects clauses and lists, normalizing booleans', () => { + assert.deepEqual(deedValues(ast(['clause', 'ignored-name', [ + field('ignored-key', 'list', [['string', 'x'], ['list', [['boolean', '#t'], ['boolean', '#f']]], ['integer', '0']]), + ['clause', 'nested', [field('id', 'uuid', 'id-value')]], + ]])), ['x', 'true', 'false', '0', 'id-value']); +}); +test('AST names and unknown items cannot satisfy missing values', () => { + assert.deepEqual(census('a = "needle"', ast(field('needle', 'string', 'other'), ['unknown', 'needle'])), { + source_values: 1, missing: ['needle'], short: [{ value: 'needle', source: 1, deed: 0 }], + }); +}); +test('empty and whitespace-only source values are excluded', () => { + assert.deepEqual(census('a = ["", " ", "\\t", "x"]', strings('x')), {source_values: 1, missing: [], short: []}); +}); +test('nonempty values retain significant whitespace', () => { + assert.deepEqual(census('a = " x "', strings('x')).missing, [' x ']); +}); +test('duplicate deficits report exact counts and stable first-seen order', () => { + assert.deepEqual(census('a = ["b", "a", "b", "a", "b"]', strings('b', 'a')), { + source_values: 5, missing: ['b', 'a'], short: [{value: 'b', source: 3, deed: 1}, {value: 'a', source: 2, deed: 1}], + }); +}); +test('surplus deed values and reordered fields are permitted', () => { + assert.deepEqual(census('a = ["x", "x", true, 0]', ast(...strings('extra', 'x', 'x', 'x').items, field('n', 'integer', '0'), field('b', 'boolean', '#t'))), {source_values: 4, missing: [], short: []}); +}); +test('census does not mutate caller AST', () => { + const input = strings('x'); const before = structuredClone(input); + census('a = "x"', input); assert.deepEqual(input, before); +}); + +// Parse failures exit the process: exercise the public CLI in isolated children. +const tmp = mkdtempSync(join(tmpdir(), 'a2ml-census-test-')); +try { + const source = join(tmp, 'source.a2ml'), deed = join(tmp, 'ast.json'); + const cli = (...args) => { + const r = spawnSync(process.execPath, [process.env.CENSUS, ...args], {encoding: 'utf8', timeout: 3000}); + assert.equal(r.error, undefined); assert.equal(r.signal, null); return r; + }; + writeFileSync(deed, JSON.stringify(strings('x'))); + const invalid = [ + ['missing assignment', 'oops\n', /key without '='/], + ['missing value', 'a =', /expected a value/], + ['unterminated basic string', 'a = "oops', /unterminated string/], + ['newline in literal', "a = 'one\ntwo'", /unterminated literal/], + ['unterminated multiline', 'a = """oops', /unterminated multi-line/], + ['unknown escape', String.raw`a = "\q"`, /bad escape/], + ['short Unicode escape', String.raw`a = "\u123"`, /bad \\u escape/], + ['surrogate escape', String.raw`a = "\uD800"`, /not a Unicode scalar/], + ['out of range escape', String.raw`a = "\U00110000"`, /not a Unicode scalar/], + ['missing array comma', 'a = ["x" "y"]', /expected , or \]/], + ['missing table comma', 'a = {b = "x" c = "y"}', /expected , or }/], + ['trailing content', 'a = "x" junk', /trailing content/], + ['line number after CRLF', '# comment\r\na = "oops\r\n', /A2ML line 2/], + ]; + for (const [name, text, error] of invalid) test(`CLI rejects ${name}`, () => { + writeFileSync(source, text); const r = cli('check', source, deed); + assert.equal(r.status, 2); assert.equal(r.stdout, ''); assert.match(r.stderr, error); + }); + for (const [name, args] of [['no arguments', []], ['wrong command', ['convert', source, deed]], ['missing AST argument', ['check', source]]]) { + test(`CLI usage: ${name}`, () => { const r = cli(...args); assert.equal(r.status, 2); assert.match(r.stderr, /usage:/); }); + } + test('CLI unreadable source', () => { const r = cli('check', join(tmp, 'absent'), deed); assert.equal(r.status, 2); assert.match(r.stderr, /cannot read/); }); + writeFileSync(source, 'a = "x"'); + test('CLI invalid JSON', () => { writeFileSync(deed, '{'); const r = cli('check', source, deed); assert.equal(r.status, 2); assert.match(r.stderr, /cannot read/); }); + test('CLI rejects AST without items', () => { writeFileSync(deed, '{}'); const r = cli('check', source, deed); assert.equal(r.status, 2); assert.match(r.stderr, /AST has no items/); }); + for (const [name, values, status, expected] of [ + ['matching values', ['x'], 0, {source_values: 1, missing: [], short: []}], + ['missing values', [], 1, {source_values: 1, missing: ['x'], short: [{value: 'x', source: 1, deed: 0}]}], + ]) test(`CLI ${name}: status and JSON`, () => { + writeFileSync(deed, JSON.stringify(strings(...values))); const r = cli('check', source, deed); + assert.equal(r.status, status); assert.equal(r.stderr, ''); assert.deepEqual(JSON.parse(r.stdout), expected); + }); +} finally { rmSync(tmp, { recursive: true, force: true }); } +console.log(`a2ml-census-test: ${passed} passed`); +JS diff --git a/scripts/tests/fix-a2ml-via-kyx-test.sh b/scripts/tests/fix-a2ml-via-kyx-test.sh index 63e77f40..60f2ae67 100644 --- a/scripts/tests/fix-a2ml-via-kyx-test.sh +++ b/scripts/tests/fix-a2ml-via-kyx-test.sh @@ -324,5 +324,34 @@ check "dangling symlink: left in place, nothing written through it" "1 0" \ r="$TMP/none"; mkdir -p "$r"; git -C "$r" init -q check "no .a2ml: exit 0, nothing to do" 0 "$(bash "$FIX" "$r" >/dev/null 2>&1; echo $?)" +# --- ambiguity and canonical-location boundaries ---------------------------- +for location in .machine_readable/ECOSYSTEM.a2ml .machine_readable/6a2/ECOSYSTEM.a2ml; do + r=$(mkrepo alternate) + mkdir -p "$r/$(dirname "$location")" + git -C "$r" mv .machine_readable/descriptiles/ECOSYSTEM.a2ml "$location" + git -C "$r" rm -q .machine_readable/descriptiles/STATE.a2ml contractiles/must/Mustfile.a2ml + git -C "$r" commit -qm alternate + STUB_KYX=ok run "$r" + check "$location: conversion-only success exits 0" 0 "$RC" + check "$location: report identifies the original source and canonical target" \ + "$location .machine_readable/descriptiles/ECOSYSTEM.deed" \ + "$(jq -r '.converted[0] | "\(.source) \(.target)"' "$REPORT/a2ml-report.json")" + check "$location: no refusal issue" 0 "$([[ -e "$REPORT/a2ml-p1-issue.md" ]] && echo 1 || echo 0)" + check "$location: outcome predicate reaches zero" 0 "$(bash "$FIX" --check "$r")" +done +r=$(mkrepo ambiguous) +cp "$r/.machine_readable/descriptiles/ECOSYSTEM.a2ml" "$r/.machine_readable/ECOSYSTEM.a2ml" +git -C "$r" add -A; git -C "$r" commit -qm duplicate +KYX='' run "$r" +check "two canonical sources: both refused before requiring kyX" "2 2" \ + "$RC $(jq '[.refused[] | select(.reasons[0]|contains("more than one ECOSYSTEM"))] | length' "$REPORT/a2ml-report.json")" +check "two canonical sources: tree untouched" "" "$(git -C "$r" status --porcelain)" +r=$(mkrepo two-choras) +cp "$r/demo_chora.deed" "$r/other_chora.deed"; git -C "$r" add -A; git -C "$r" commit -qm duplicate +KYX='' run "$r" +check "two chora deeds: refuse ambiguous identity before requiring kyX" "2 1" \ + "$RC $(jq '[.refused[] | select(.reasons[0]|contains("found 2"))] | length' "$REPORT/a2ml-report.json")" +check "two chora deeds: tree untouched" "" "$(git -C "$r" status --porcelain)" + echo "fix-a2ml-via-kyx-test: $pass passed, $fail failed" [[ "$fail" -eq 0 ]] From dc9c47feecabf8f892208e75f8db77dd9ad61c4b Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Tue, 17 Nov 2026 20:46:08 +0000 Subject: [PATCH 08/10] fix(a2ml-kyx): propagate discovery failures, preserve local edits, and count whitespace values --- scripts/fix-a2ml-via-kyx.sh | 50 ++++++++++++++++++++----- scripts/lib/a2ml-census.js | 2 +- scripts/tests/fix-a2ml-via-kyx-test.sh | 52 +++++++++++++++++++++++++- 3 files changed, 93 insertions(+), 11 deletions(-) diff --git a/scripts/fix-a2ml-via-kyx.sh b/scripts/fix-a2ml-via-kyx.sh index c2cd00fa..77acafb5 100755 --- a/scripts/fix-a2ml-via-kyx.sh +++ b/scripts/fix-a2ml-via-kyx.sh @@ -88,7 +88,15 @@ die() { printf 'fix-a2ml-via-kyx: %s\n' "$1" >&2; exit "${2:-2}"; } # The tracked .a2ml files, NUL-delimited end to end (a name with a newline in # it stays one name, and is refused below), sorted, case-insensitive suffix. a2ml_files() { - git -C "$1" ls-files -z | grep -zaiE '\.a2ml$' | LC_ALL=C sort -z || true + local -a statuses + if git -C "$1" ls-files -z | grep -zaiE '\.a2ml$' | LC_ALL=C sort -z; then + return 0 + else + statuses=("${PIPESTATUS[@]}") + [[ "${statuses[0]}" -eq 0 ]] || return "${statuses[0]}" + [[ "${statuses[1]}" -le 1 ]] || return "${statuses[1]}" + return "${statuses[2]}" + fi } check_mode=0 @@ -100,18 +108,29 @@ git -C "$REPO" rev-parse --is-inside-work-tree >/dev/null 2>&1 || die "not a git REPO="$(cd "$REPO" && git rev-parse --show-toplevel)" REPO="$(cd "$REPO" && pwd -P)" +if [[ "$check_mode" -ne 1 ]]; then + # Every refusal is recorded with jq, so it is needed before anything is + # classified, not only on kyX's path: without it there would be no report. + for tool in jq sha256sum; do + command -v "$tool" >/dev/null 2>&1 || die "$tool is required to record any outcome; nothing was changed" 69 + done + +fi + +# Materialize discovery so mapfile cannot hide a producer failure. Keep NULs +# in a file: command substitution cannot preserve them. +discovery="$(mktemp "${TMPDIR:-/tmp}/a2ml-discovery.XXXXXX")" +trap 'rm -f -- "$discovery"' EXIT +a2ml_files "$REPO" > "$discovery" || die "tracked .a2ml discovery failed; nothing was changed" "$?" +mapfile -d '' -t FILES < "$discovery" +rm -f -- "$discovery" +trap - EXIT + if [[ "$check_mode" -eq 1 ]]; then - a2ml_files "$REPO" | tr -cd '\0' | wc -c | tr -d ' ' + printf '%s\n' "${#FILES[@]}" exit 0 fi -# Every refusal is recorded with jq, so it is needed before anything is -# classified, not only on kyX's path: without it there would be no report. -for tool in jq sha256sum; do - command -v "$tool" >/dev/null 2>&1 || die "$tool is required to record any outcome; nothing was changed" 69 -done - -mapfile -d '' -t FILES < <(a2ml_files "$REPO") if [[ "${#FILES[@]}" -eq 0 ]]; then echo "fix-a2ml-via-kyx: no .a2ml in $REPO; nothing to do" exit 0 @@ -150,6 +169,11 @@ undo_stage() { # src local src="$1" status git -C "$REPO" reset -q -- "$TARGET" "$src" 2>> "$WORK/undo.err" || return 1 rm -f -- "$REPO/$TARGET" 2>> "$WORK/undo.err" || return 1 + # A missing source is the expected result of git rm. If it still exists, + # preserve any edits made since preflight rather than checking over them. + if [[ -e "$REPO/$src" || -L "$REPO/$src" ]]; then + git -C "$REPO" diff --quiet -- "$src" 2>> "$WORK/undo.err" || return 1 + fi git -C "$REPO" checkout -q -- "$src" 2>> "$WORK/undo.err" || return 1 # The check itself must succeed: a failed `git status` prints nothing, and # nothing would read as clean. @@ -277,11 +301,19 @@ preflight() { # src if [[ -n "$st" ]]; then echo "${choras[0]} has uncommitted changes; kyX must read the committed chora deed the PR will reference."; return fi + if [[ -L "$REPO/$src" ]]; then + echo "the source is a symlink; kyX must read content Git tracks."; return + fi st="$(git -C "$REPO" status --porcelain -- "$src" 2>&1)" \ || { echo "git status could not read the source, so its state is unknown: ${st:0:200}"; return; } if [[ -n "$st" ]]; then echo "the source has uncommitted changes; kyX would convert content the PR does not delete."; return fi + st="$(git -C "$REPO" status --porcelain -- "$TARGET" 2>&1)" \ + || { echo "git status could not read the target, so its state is unknown: ${st:0:200}"; return; } + if [[ -n "$st" ]]; then + echo "$TARGET has uncommitted changes; they must be preserved."; return + fi # Each directory on the way to the target must be absent or a real # directory: a symlink would put the deed outside the work tree. dir="$REPO" diff --git a/scripts/lib/a2ml-census.js b/scripts/lib/a2ml-census.js index 67c9ccc1..09374429 100644 --- a/scripts/lib/a2ml-census.js +++ b/scripts/lib/a2ml-census.js @@ -203,7 +203,7 @@ function tally(values) { } export function census(sourceText, ast) { - const want = a2mlValues(sourceText).filter((v) => v.trim() !== ""); + const want = a2mlValues(sourceText).filter((v) => v !== ""); // Counted, not just present: two source fields holding the same value must // be two deed values, or a field collapsed into another passes unseen. const need = tally(want); diff --git a/scripts/tests/fix-a2ml-via-kyx-test.sh b/scripts/tests/fix-a2ml-via-kyx-test.sh index 60f2ae67..37f9402e 100644 --- a/scripts/tests/fix-a2ml-via-kyx-test.sh +++ b/scripts/tests/fix-a2ml-via-kyx-test.sh @@ -135,6 +135,15 @@ check "census: the same value twice in the deed passes" 0 "$(bun "$CENSUS" check printf 'oops\n' > "$TMP/bad.a2ml" check "census: an unreadable source is exit 2, not a pass" 2 "$(bun "$CENSUS" check "$TMP/bad.a2ml" "$TMP/ast.json" >/dev/null 2>&1; echo $?)" +# Empty strings may be dropped, but whitespace is a scalar value. +printf 'a = ""\nb = " "\nc = "\\t"\n' > "$TMP/space.a2ml" +printf '{"items":[]}\n' > "$TMP/space-ast.json" +check "census: missing whitespace values are deficits, empty strings excluded" \ + '2 [" ","\t"]' \ + "$(bun "$CENSUS" check "$TMP/space.a2ml" "$TMP/space-ast.json" | jq -r '"\(.source_values) \(.missing|tojson)"' || true)" +printf '{"items":[["field","b",["string"," "]],["field","c",["string","\\t"]]]}\n' > "$TMP/space-ast.json" +check "census: matching whitespace values pass" 0 "$(bun "$CENSUS" check "$TMP/space.a2ml" "$TMP/space-ast.json" >/dev/null; echo $?)" + # --- --check (the outcome predicate) ----------------------------------------- r=$(mkrepo check) check "--check counts tracked .a2ml" 3 "$(bash "$FIX" --check "$r")" @@ -264,6 +273,31 @@ r=$(mkrepo nojq); RC=0; PATH="$TMP/nojq-bin" A2ML_REPORT_DIR="$TMP/nojq-rep" "$( check "jq missing: exit 69 before anything is classified" 69 "$RC" +# A committed source symlink is clean in Git but its referent is untracked. +r=$(mkrepo source-link) +cp "$r/.machine_readable/descriptiles/ECOSYSTEM.a2ml" "$TMP/link-source" +rm "$r/.machine_readable/descriptiles/ECOSYSTEM.a2ml" +ln -s "$TMP/link-source" "$r/.machine_readable/descriptiles/ECOSYSTEM.a2ml" +git -C "$r" add -A; git -C "$r" commit -qm symlink +STUB_KYX=ok run "$r" +check "a tracked source symlink is refused" 1 "$(jq '[.refused[] | select(.reasons[0]|test("source is a symlink"))] | length' "$REPORT/a2ml-report.json")" +check "source symlink: tree unchanged" "" "$(git -C "$r" status --porcelain)" + +for deletion in staged unstaged; do + r=$(mkrepo "target-$deletion") + cp "$TMP/ECOSYSTEM.deed" "$r/.machine_readable/descriptiles/ECOSYSTEM.deed" + git -C "$r" add -A; git -C "$r" commit -qm target + if [[ "$deletion" == staged ]]; then + git -C "$r" rm -q .machine_readable/descriptiles/ECOSYSTEM.deed + else + rm "$r/.machine_readable/descriptiles/ECOSYSTEM.deed" + fi + before="$(git -C "$r" status --porcelain)" + STUB_KYX=ok run "$r" + check "$deletion target deletion: refused" 1 "$(jq '[.refused[] | select(.reasons[0]|test("has uncommitted changes"))] | length' "$REPORT/a2ml-report.json")" + check "$deletion target deletion: preserved" "$before" "$(git -C "$r" status --porcelain)" +done + # --- failures after the gates: staging, rollback and reporting ---------------- # Wrappers first on PATH fail one command at a chosen point and pass everything # else to the real tool, so the script needs no test hooks. @@ -273,7 +307,13 @@ cat > "$WRAP/git" < "$WRAP/rm-failed"; echo "fatal: injected rm failure" >&2; exit 128; } ;; + rm) + if [[ -n "\${EDIT_SOURCE_ON_RM:-}" ]]; then + printf '# owner edit\\n' >> "\$2/.machine_readable/descriptiles/ECOSYSTEM.a2ml" + exit 128 + fi + [[ -n "\${FAIL_GIT_RM:-}" ]] && { : > "$WRAP/rm-failed"; echo "fatal: injected rm failure" >&2; exit 128; } ;; + ls-files) [[ -n "\${FAIL_GIT_LS_FILES:-}" ]] && { printf 'partial.a2ml\\0'; exit 128; } ;; status) [[ -n "\${FAIL_GIT_STATUS:-}" ]] && { echo "fatal: injected status failure" >&2; exit 128; } [[ -n "\${FAIL_GIT_STATUS_AFTER_RM:-}" && -e "$WRAP/rm-failed" ]] && { echo "fatal: injected status failure" >&2; exit 128; } ;; @@ -298,6 +338,16 @@ wrun() { # repo -> RC; the wrapped PATH, plus whatever FAIL_* the caller set REPORT="$TMP/report-$RANDOM"; mkdir -p "$REPORT"; RC=0 PATH="$WRAP:$PATH" A2ML_REPORT_DIR="$REPORT" STUB_KYX=ok bash "$FIX" "$1" > "$TMP/out" 2> "$TMP/err" || RC=$? } +r=$(mkrepo discovery-fails) +FAIL_GIT_LS_FILES=1 wrun "$r" +check "normal discovery failure propagates past mapfile" 128 "$RC" +RC=0 +FAIL_GIT_LS_FILES=1 PATH="$WRAP:$PATH" bash "$FIX" --check "$r" > "$TMP/out" 2> "$TMP/err" || RC=$? +check "--check discovery failure propagates" 128 "$RC" +check "failed discovery prints no count" "" "$(cat "$TMP/out")" +r=$(mkrepo owner-edit); EDIT_SOURCE_ON_RM=1 wrun "$r" +check "rollback with source edits exits INCONSISTENT" 70 "$RC" +check "rollback preserves source edits" '# owner edit' "$(tail -1 "$r/.machine_readable/descriptiles/ECOSYSTEM.a2ml")" r=$(mkrepo rm-fails); FAIL_GIT_RM=1 wrun "$r" check "git rm fails mid-staging: refused, not converted -> exit 2" 2 "$RC" check "git rm fails: the rollback leaves the tree exactly as it was" "" "$(git -C "$r" status --porcelain)" From 935edd2a48a3b4a9e2d2ceb61755dc36507b5036 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 20:49:40 +0000 Subject: [PATCH 09/10] fix(a2ml-kyx): a git status warning is not a change; whitespace values count Repairs two regressions between dc9c47f (CodeRabbit autofix) and 9da41e7 (CodeRabbit-generated tests); at dc9c47f the fixer suite failed 8 checks and the census suite 1. Neither suite runs in CI (both need bun, and the fixer suite a standards checkout), so CI stayed green. - The new target check captured git status stderr with 2>&1. When the target's directory does not exist, as with an ECOSYSTEM.a2ml at .machine_readable/ or .machine_readable/6a2/, git prints "warning: could not open directory" and exits 0, which read as uncommitted changes and refused every alternate-layout conversion. All three preflight status checks now keep stderr in a file: a warning is not a change, and a real failure still refuses and quotes it. - The generated census test asserted that whitespace-only values are excluded: the hole dc9c47f closed. It now requires them to be counted (a dropped " " or tab is a shortfall) and to pass when the deed has them. - Adds the missing test for the staged-deletion finding: a staged deletion of the target is refused and left as the owner staged it. Census suite 37/37, fixer suite 112/112; the 8 alternate-layout checks fail without the status change. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDhFeexmokLqLRfZNce --- scripts/fix-a2ml-via-kyx.sh | 16 +++++++++------- scripts/tests/a2ml-census-test.sh | 10 ++++++++-- scripts/tests/fix-a2ml-via-kyx-test.sh | 8 ++++++++ 3 files changed, 25 insertions(+), 9 deletions(-) diff --git a/scripts/fix-a2ml-via-kyx.sh b/scripts/fix-a2ml-via-kyx.sh index 77acafb5..b3693f31 100755 --- a/scripts/fix-a2ml-via-kyx.sh +++ b/scripts/fix-a2ml-via-kyx.sh @@ -295,22 +295,24 @@ mapfile -d '' -t choras < <(git -C "$REPO" ls-files -z -- ':(glob)*_chora.deed') preflight() { # src local src="$1" dir st # A failed `git status` prints nothing, which would read as clean: check - # that it ran before reading what it printed. - st="$(git -C "$REPO" status --porcelain -- "${choras[0]}" 2>&1)" \ - || { echo "git status could not read ${choras[0]}, so its state is unknown: ${st:0:200}"; return; } + # that it ran before reading what it printed. Its stderr is kept apart: + # a harmless warning (a missing parent directory of an absent target) is + # not a change, and a real failure is quoted from it. + st="$(git -C "$REPO" status --porcelain -- "${choras[0]}" 2> "$WORK/status.err")" \ + || { echo "git status could not read ${choras[0]}, so its state is unknown: $(head -c 200 "$WORK/status.err")"; return; } if [[ -n "$st" ]]; then echo "${choras[0]} has uncommitted changes; kyX must read the committed chora deed the PR will reference."; return fi if [[ -L "$REPO/$src" ]]; then echo "the source is a symlink; kyX must read content Git tracks."; return fi - st="$(git -C "$REPO" status --porcelain -- "$src" 2>&1)" \ - || { echo "git status could not read the source, so its state is unknown: ${st:0:200}"; return; } + st="$(git -C "$REPO" status --porcelain -- "$src" 2> "$WORK/status.err")" \ + || { echo "git status could not read the source, so its state is unknown: $(head -c 200 "$WORK/status.err")"; return; } if [[ -n "$st" ]]; then echo "the source has uncommitted changes; kyX would convert content the PR does not delete."; return fi - st="$(git -C "$REPO" status --porcelain -- "$TARGET" 2>&1)" \ - || { echo "git status could not read the target, so its state is unknown: ${st:0:200}"; return; } + st="$(git -C "$REPO" status --porcelain -- "$TARGET" 2> "$WORK/status.err")" \ + || { echo "git status could not read the target, so its state is unknown: $(head -c 200 "$WORK/status.err")"; return; } if [[ -n "$st" ]]; then echo "$TARGET has uncommitted changes; they must be preserved."; return fi diff --git a/scripts/tests/a2ml-census-test.sh b/scripts/tests/a2ml-census-test.sh index e19fb6dd..f9f5a355 100644 --- a/scripts/tests/a2ml-census-test.sh +++ b/scripts/tests/a2ml-census-test.sh @@ -42,8 +42,14 @@ test('AST names and unknown items cannot satisfy missing values', () => { source_values: 1, missing: ['needle'], short: [{ value: 'needle', source: 1, deed: 0 }], }); }); -test('empty and whitespace-only source values are excluded', () => { - assert.deepEqual(census('a = ["", " ", "\\t", "x"]', strings('x')), {source_values: 1, missing: [], short: []}); +test('only empty values are excluded; whitespace-only values are counted', () => { + // A value that is only spaces or a tab is still a value: a deed that drops + // it has lost something, so it must show as a shortfall. + assert.deepEqual(census('a = ["", " ", "\\t", "x"]', strings('x')), { + source_values: 3, missing: [' ', '\t'], + short: [{value: ' ', source: 1, deed: 0}, {value: '\t', source: 1, deed: 0}], + }); + assert.deepEqual(census('a = ["", " ", "\\t", "x"]', strings('x', ' ', '\t')).missing, []); }); test('nonempty values retain significant whitespace', () => { assert.deepEqual(census('a = " x "', strings('x')).missing, [' x ']); diff --git a/scripts/tests/fix-a2ml-via-kyx-test.sh b/scripts/tests/fix-a2ml-via-kyx-test.sh index 37f9402e..c172b716 100644 --- a/scripts/tests/fix-a2ml-via-kyx-test.sh +++ b/scripts/tests/fix-a2ml-via-kyx-test.sh @@ -371,6 +371,14 @@ check "a dangling symlink at the target is refused as existing" 1 "$(jq '[.refus check "dangling symlink: left in place, nothing written through it" "1 0" \ "$([[ -L "$r/.machine_readable/descriptiles/ECOSYSTEM.deed" ]] && echo 1 || echo 0) $([[ -e "$TMP/no-such-deed" ]] && echo 1 || echo 0)" +r=$(mkrepo staged-del); cp "$TMP/ECOSYSTEM.deed" "$r/.machine_readable/descriptiles/ECOSYSTEM.deed" +git -C "$r" add -A; git -C "$r" commit -qm deed; git -C "$r" rm -q .machine_readable/descriptiles/ECOSYSTEM.deed +STUB_KYX=ok run "$r" +check "a staged deletion of the target is refused, never overwritten by git add" 1 \ + "$(jq '[.refused[] | select(.reasons[0]|test("ECOSYSTEM.deed has uncommitted changes"))] | length' "$REPORT/a2ml-report.json")" +check "staged deletion of the target: kept as the owner staged it" "D .machine_readable/descriptiles/ECOSYSTEM.deed" \ + "$(git -C "$r" status --porcelain -- .machine_readable/descriptiles/ECOSYSTEM.deed)" + r="$TMP/none"; mkdir -p "$r"; git -C "$r" init -q check "no .a2ml: exit 0, nothing to do" 0 "$(bash "$FIX" "$r" >/dev/null 2>&1; echo $?)" From 241c5b4f7829c56f516d647f608f3145c624d5a3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 17 Nov 2026 21:50:31 +0000 Subject: [PATCH 10/10] fix(a2ml-kyx): create the target exclusively and keep owner edits to it on rollback A target the owner creates after preflight is no longer overwritten (noclobber create), and rollback removes the target only when this run created it and it still matches what was generated, in the work tree and the index; otherwise the rollback fails and the run exits 70. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01J4QvDj7XpKyxE4WbsgTsTh --- scripts/fix-a2ml-via-kyx.sh | 29 +++++++++++++++++++++----- scripts/tests/fix-a2ml-via-kyx-test.sh | 20 ++++++++++++++++++ 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/scripts/fix-a2ml-via-kyx.sh b/scripts/fix-a2ml-via-kyx.sh index b3693f31..d23e7e51 100755 --- a/scripts/fix-a2ml-via-kyx.sh +++ b/scripts/fix-a2ml-via-kyx.sh @@ -162,13 +162,31 @@ rm -f -- "$probe" WORK="$(mktemp -d "${TMPDIR:-/tmp}/a2ml-kyx-work.XXXXXX")" STAGED=() # sources whose conversion is staged in the index REPORTS_DONE=0 # set once every report is published +CREATED_TARGET=0 # set once THIS run created the target file (never a file that was already there) # Put the index and work tree back as they were before `src` was staged, and # say whether that worked: every step is checked, then the result is. undo_stage() { # src - local src="$1" status - git -C "$REPO" reset -q -- "$TARGET" "$src" 2>> "$WORK/undo.err" || return 1 - rm -f -- "$REPO/$TARGET" 2>> "$WORK/undo.err" || return 1 + local src="$1" status paths=("$1") + if [[ "$CREATED_TARGET" -eq 1 ]]; then + # The target is ours to remove only while it is still exactly what this + # run generated, in the work tree and in the index. An edit made since + # belongs to the owner: keep it and fail the rollback. + if [[ -e "$REPO/$TARGET" || -L "$REPO/$TARGET" ]]; then + if [[ -L "$REPO/$TARGET" ]] || ! cmp -s -- "$REPO/$TARGET" "$WORK/ECOSYSTEM.deed"; then + echo "$TARGET was changed after it was generated; left unchanged" >> "$WORK/undo.err"; return 1 + fi + fi + if [[ -n "$(git -C "$REPO" ls-files --cached -- "$TARGET" 2>> "$WORK/undo.err")" ]]; then + git -C "$REPO" show ":$TARGET" 2>> "$WORK/undo.err" | cmp -s -- - "$WORK/ECOSYSTEM.deed" \ + || { echo "$TARGET was changed in the index after it was generated; left unchanged" >> "$WORK/undo.err"; return 1; } + fi + paths=("$TARGET" "$src") + fi + git -C "$REPO" reset -q -- "${paths[@]}" 2>> "$WORK/undo.err" || return 1 + if [[ "$CREATED_TARGET" -eq 1 ]]; then + rm -f -- "$REPO/$TARGET" 2>> "$WORK/undo.err" || return 1 + fi # A missing source is the expected result of git rm. If it still exists, # preserve any edits made since preflight rather than checking over them. if [[ -e "$REPO/$src" || -L "$REPO/$src" ]]; then @@ -177,7 +195,7 @@ undo_stage() { # src git -C "$REPO" checkout -q -- "$src" 2>> "$WORK/undo.err" || return 1 # The check itself must succeed: a failed `git status` prints nothing, and # nothing would read as clean. - status="$(git -C "$REPO" status --porcelain -- "$TARGET" "$src" 2>> "$WORK/undo.err")" || return 1 + status="$(git -C "$REPO" status --porcelain -- "${paths[@]}" 2>> "$WORK/undo.err")" || return 1 [[ -z "$status" ]] } @@ -425,7 +443,8 @@ for src in "${eco_sources[@]}"; do # preflight makes a failure here unlikely; if one happens anyway, the tree # and index are put back (checked) and the file is refused with the reason. if ! { mkdir -p "$REPO/$(dirname "$TARGET")" \ - && cp "$out" "$REPO/$TARGET" \ + && ( set -o noclobber; cat -- "$out" > "$REPO/$TARGET" ) \ + && CREATED_TARGET=1 \ && git -C "$REPO" add -- "$TARGET" \ && git -C "$REPO" rm -q -- "$src"; } 2> "$WORK/stage.err"; then scrub "$WORK/stage.err" diff --git a/scripts/tests/fix-a2ml-via-kyx-test.sh b/scripts/tests/fix-a2ml-via-kyx-test.sh index c172b716..71022553 100644 --- a/scripts/tests/fix-a2ml-via-kyx-test.sh +++ b/scripts/tests/fix-a2ml-via-kyx-test.sh @@ -317,6 +317,17 @@ for a in "\$@"; do status) [[ -n "\${FAIL_GIT_STATUS:-}" ]] && { echo "fatal: injected status failure" >&2; exit 128; } [[ -n "\${FAIL_GIT_STATUS_AFTER_RM:-}" && -e "$WRAP/rm-failed" ]] && { echo "fatal: injected status failure" >&2; exit 128; } ;; + check-ignore) + if [[ -n "\${OWNER_CREATES_TARGET:-}" ]]; then + mkdir -p "\$2/.machine_readable/descriptiles" + printf '# owner deed\\n' > "\$2/.machine_readable/descriptiles/ECOSYSTEM.deed" + fi ;; + add) + if [[ -n "\${OWNER_EDITS_TARGET_AFTER_ADD:-}" ]]; then + "$REAL_GIT" "\$@" || exit \$? + printf '# owner edit\\n' >> "\$2/.machine_readable/descriptiles/ECOSYSTEM.deed" + exit 0 + fi ;; reset) [[ -n "\${FAIL_GIT_RESET:-}" ]] && { echo "fatal: injected reset failure" >&2; exit 128; } ;; esac done @@ -348,6 +359,15 @@ check "failed discovery prints no count" "" "$(cat "$TMP/out")" r=$(mkrepo owner-edit); EDIT_SOURCE_ON_RM=1 wrun "$r" check "rollback with source edits exits INCONSISTENT" 70 "$RC" check "rollback preserves source edits" '# owner edit' "$(tail -1 "$r/.machine_readable/descriptiles/ECOSYSTEM.a2ml")" +# A target the owner creates after preflight is never overwritten or removed. +r=$(mkrepo target-race); OWNER_CREATES_TARGET=1 wrun "$r" +check "target created after preflight: refused, not converted -> exit 2" 2 "$RC" +check "target created after preflight: the owner's file is untouched" '# owner deed' "$(cat "$r/.machine_readable/descriptiles/ECOSYSTEM.deed")" +check "target created after preflight: the source is still there, nothing staged" "" "$(git -C "$r" diff --cached --name-only)" +# A generated target the owner edits before report publication fails is kept. +r=$(mkrepo target-edit); OWNER_EDITS_TARGET_AFTER_ADD=1 FAIL_MV=1 wrun "$r" +check "target edited before a failed publish: exit INCONSISTENT" 70 "$RC" +check "target edited before a failed publish: the edit is preserved" '# owner edit' "$(tail -1 "$r/.machine_readable/descriptiles/ECOSYSTEM.deed")" r=$(mkrepo rm-fails); FAIL_GIT_RM=1 wrun "$r" check "git rm fails mid-staging: refused, not converted -> exit 2" 2 "$RC" check "git rm fails: the rollback leaves the tree exactly as it was" "" "$(git -C "$r" status --porcelain)"