diff --git a/campaigns/A2ML-KYX-CAMPAIGN.adoc b/campaigns/A2ML-KYX-CAMPAIGN.adoc new file mode 100644 index 00000000..e62760b4 --- /dev/null +++ b/campaigns/A2ML-KYX-CAMPAIGN.adoc @@ -0,0 +1,138 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 += A2ML Retirement Campaign (kyX): 2026-11-17 +:toc: +:toclevels: 2 + +== Purpose + +The owner's directive of 2026-11-17: "make sure that a2ml is sorted and +converted opportunistically and automatically when a bot sees it as a +priority one task, I need to rid the estate of this". + +A2ML is retired (owner rulings D99, D269c, D312, D313). Two pieces carry the +directive: + +* *Seeing it.* rhodibot reports every tracked `.a2ml` as a priority-1 finding: + `rhodibot.no-a2ml` (severity `retired`, `critical` on the verdict grid) in + `canon`, and `no-*.a2ml` (RSR-A2ML-001, Required in every policy pack) in the + GitHub App check. (Draft PR #690.) +* *Sorting it.* `scripts/fix-a2ml-via-kyx.sh` (recipe `recipe-a2ml-kyx`, + category `RetiredA2ML`) converts what kyX can convert losslessly and fails + loudly, with a P1 issue body, on everything else. It never hand-converts. + +== Status + +*Nothing is dispatched.* The fleet dispatch is under quarantine +(`docs/AUTOMATION-QUARANTINE.adoc`), and this change does not touch the +interlock. There is no manifest, because a manifest needs an estate scan, and +the quarantine document forbids a scan as a workaround. Running the campaign is +the owner's decision; the options are below. + +== What kyX can do today (checked 2026-11-17) + +* `metadatastician/kyx` main `731e63a`: CI green, 39 acceptance checks, every + output checked with standards `deed_lint` at a pinned commit. +* *One front end: `ECOSYSTEM.a2ml`.* Every other family is "not started" + (kyX README, Status). kyX refuses any field its spec does not list (the real + hypatia ECOSYSTEM is refused with 19 fields named), and refuses a repository + with no chora deed: it reads the id and never mints one (U1). +* *No released artifact.* A consumer builds the AffineScript compiler at kyX's + pinned commit (OCaml) and then kyX. `metadatastician/kyx` is *private*, so a + fleet runner needs read access to it. + +Dogfood on this repository (`fix-a2ml-via-kyx.sh` on a scratch clone): +*0 converted, 42 refused* across 13 families, because gitbot-fleet has no +`gitbot-fleet_chora.deed` and only its 3 ECOSYSTEM files are kyX-routable at +all. Expect the estate to look the same: until kyX grows front ends, almost +every repository gets a P1 issue and very few get a PR. + +== The fixer's gates + +Every converted file passes all four, or nothing in the tree changes for it: + +. kyX exits 0 (fail-closed field accounting under the mapping spec). +. kyX drops nothing. Spec §4 lets kyX drop empty values, comment lines and + `@directives`; by default any drop is a refusal ("not lossless"). + `A2ML_ACCEPT_SPEC_DROPS=1` accepts spec-sanctioned drops and lists each in + the PR (an owner decision, see Q-LOSSLESS below). +. The deed passes standards `deed_lint.js --ecosystem --root ` (deed.abnf + plus ecosystem.abnf). A `.k9`/`.k9.ncl` output, when a future front end emits + one, must pass `k9-validate.sh --strict`. +. The value census (`scripts/lib/a2ml-census.js`): every non-empty source value + appears in the deed (`deed_lint --ast-json`) at least as many times as in + the source, so a field collapsed into another is caught; independently of + kyX. It errs towards refusing. + +Bots open PRs, never push: the fixer stages the change and writes the PR and +P1 issue bodies; it never commits, pushes or publishes. A refusal's issue body +carries a fingerprint, so a caller can find an open issue for the same refusal +instead of opening a second one. + +== Prerequisites (each blocks a real run) + +[cols="1,3,2"] +|=== +| # | Prerequisite | Whose + +| P-1 | A kyX build a runner can fetch: a release asset `kyx.bun.js` with its SHA-256 and source commit (preferred), or a read token for the private repo. Building AffineScript on every run is minutes per repository. | owner / kyx +| P-2 | The quarantine lifted for this recipe, by a reviewed change, with PR-only token scope (branches, PRs, issues; no default-branch push, which branch protection enforces independently). | owner +| P-3 | A chora deed in each target repository (kyX never mints one). This repository has none. | per repo +| P-4 | Issue labels `P1` and `a2ml` exist where issues are opened. | per repo +|=== + +== Rollout options (owner question Q-ROLLOUT) + +Rated, most elegant and correct first; the second number is dependable and +secure/private. + +[cols="1,4,1,1"] +|=== +| Option | What | Elegance | Safety + +| A. Repo-local opportunistic job (recommended) +| `fix-fleet-enrolment.sh` adds an `a2ml` step to each repo's own fleet job: + when rhodibot's report-back carries `rhodibot.no-a2ml`, the job runs the + fixer in that repo's own CI and opens a draft PR or the P1 issue with the + repo's own `GITHUB_TOKEN`. No cross-repo credential, truly "when a bot sees + it", and enrolment is already per repo. +| 5 | 4 + +| B. Waved campaign manifest through `dispatch-runner` +| A JSONL manifest built from the rhodibot findings the report-backs already + hold (no new scan), dispatched in waves: 3 pilot repos with an ECOSYSTEM.a2ml + and a chora deed, then 25, then all. Needs P-2 for the runner. +| 4 | 4 + +| C. Report only, owner converts +| Keep rhodibot's P1 finding; the owner runs the fixer locally per repo. No + automation to qualify. +| 2 | 5 + +| D. kyX-side batch (alternative path) +| kyX grows a `--repo` batch front end and its own CI opens the PRs; the fleet + only reports. Puts conversion and its proof in one place, but moves work to + kyX. +| 4 | 3 +|=== + +Whichever is chosen, issue volume is the cost to weigh: about 500 repositories +and one front end today means hundreds of P1 issues. A variant of any option +opens *one tracking issue per family in metadatastician/kyx* and a short P1 +issue per repository that links to it. + +== Owner questions + +* *Q-ROLLOUT:* A, B, C or D above, and whether to open per-repo P1 issues now + or wait for more kyX front ends. +* *Q-LOSSLESS:* is a spec §4 drop (empty value, comment line, `@directive`) a + loss? Default: yes, refuse. Alternative: accept and list it in the PR. + +== Beyond this repository (to track) + +* kyX: a released `kyx.bun.js` (P-1); front ends for STATE, META, AGENTIC, + NEUROSYM, PLAYBOOK, CLADE and the manifest; mapping specs for ANCHOR, + rsr-profile, bot directives and the contractiles. +* standards: the vendored `rsr-criteria-v2.a2ml` here is a pinned copy of the + live standards canon; it goes when standards converts the canon (A7, P2.5). +* gitbot-fleet itself: a `gitbot-fleet_chora.deed` (P-3), then its own 42 + `.a2ml` files are the pilot. diff --git a/scripts/fix-a2ml-via-kyx.sh b/scripts/fix-a2ml-via-kyx.sh new file mode 100755 index 00000000..d23e7e51 --- /dev/null +++ b/scripts/fix-a2ml-via-kyx.sh @@ -0,0 +1,524 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell +# +# fix-a2ml-via-kyx.sh — convert a repository's retired .a2ml files with kyX, +# or fail loudly. Owner directive 2026-11-17: rid the estate of A2ML, +# opportunistically, when a bot sees it as a priority-1 task (rhodibot +# `rhodibot.no-a2ml` / RSR-A2ML-001). +# +# Conversion is kyX's front end and nothing else (D313): this script NEVER +# hand-converts, never guesses a field, and never edits a deed. For each +# tracked .a2ml it either +# +# * converts it: kyX translates it under the standards mapping spec, and the +# output must pass every gate below; the deed is staged, the .a2ml is +# staged for deletion, and a PR body is written; or +# * refuses it, with every reason, and writes a P1 issue body. Nothing in the +# tree changes for a refused file. +# +# The gates, all of them, for every converted file: +# 1. kyX exits 0. kyX is fail-closed: a table, field or value shape its +# mapping spec does not list is refused with its line (exit 1). +# 2. kyX drops nothing. kyX reports what spec §4 lets it drop (empty values, +# comment lines, @directives) on stderr; any such line is information +# that does not reach the deed, so it is a refusal. A2ML_ACCEPT_SPEC_DROPS=1 +# accepts drops the spec sanctions and lists each in the PR body instead +# (an owner decision, off by default). +# 3. The deed parses under the canon grammar: standards +# `deed_lint.js --ecosystem --root ` (deed.abnf + ecosystem.abnf). +# A .k9/.k9.ncl output, when a kyX front end emits one, must pass +# standards `k9-validate.sh --strict` (K9_VALIDATE); none does today. +# 4. No value is lost: scripts/lib/a2ml-census.js checks that every non-empty +# scalar value in the source appears in the deed (deed_lint --ast-json). +# Independent of kyX, and conservative: a value a spec rewrites reads as +# lost, so the error is always towards refusing. +# +# kyX translates ECOSYSTEM.a2ml only (metadatastician/kyx README, "Status"); +# every other family is refused here as "no kyX front end", with the mapping +# spec that will decide it when there is one. An ECOSYSTEM.a2ml is converted +# only at a canonical location (.machine_readable/descriptiles/, 6a2/, or the +# retired direct .machine_readable/ layout), into +# .machine_readable/descriptiles/ECOSYSTEM.deed, and only when the repository +# has exactly one chora deed: kyX reads the id from it and never mints one. +# +# Writes in the repository: only the staged deed and the staged deletion of +# the .a2ml it replaces. Never commits, never pushes, never opens anything: +# the caller opens a PR on a branch (never the default branch) and the P1 +# issue. Report files go to A2ML_REPORT_DIR (default: a fresh temp directory, +# printed), never into the tree: +# a2ml-report.json {converted: [...], refused: [...], ...} +# a2ml-pr.md when something was converted +# a2ml-p1-issue.md when something was refused (fingerprinted, so a caller +# can find an open issue for the same refusal) +# +# Environment (needed only when a file is routed to kyX): +# KYX path to kyX's built Bun module (_build/kyx/kyx.bun.js) +# KYX_COMMIT the 40-hex kyX commit it was built from (recorded) +# DEED_LINT path to standards 1-formats/deed/tools/deed_lint.js +# STANDARDS_COMMIT the 40-hex standards commit DEED_LINT is from (recorded) +# AFFIRMED YYYY-MM-DD for :affirmed (default: today, UTC) +# K9_VALIDATE standards 1-formats/k9/tools/k9-validate.sh (k9 outputs) +# +# Idempotent: a converted file is gone on the next run; a refused one is +# refused again with the same fingerprint. +# +# Exit: 0 no .a2ml left (converted, or none to begin with) · 2 refused and +# nothing converted (P1 issue) · 3 partial: some converted (PR), some refused +# (P1 issue) — a dispatcher must not record 2 or 3 as a clean success · 64 +# usage · 69 a file needs kyX and a tool or its provenance is missing · 70 +# INCONSISTENT: a rollback failed, so the index and work tree need a human · +# 73 the report directory is not writable (checked before anything changes). +# Any other non-zero exit after a conversion was staged rolls it back first: +# a conversion stays in the tree only together with the reports describing it. +# +# outcome-predicate: --check +# +# Usage: +# fix-a2ml-via-kyx.sh [finding.json] +# fix-a2ml-via-kyx.sh --check print the number of tracked .a2ml +set -euo pipefail + +SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CENSUS="$SELF_DIR/lib/a2ml-census.js" + +usage() { sed -n '/^# Usage:/,/^set -euo/p' "$0" | sed '$d; s/^# \{0,1\}//' >&2; exit 64; } +die() { printf 'fix-a2ml-via-kyx: %s\n' "$1" >&2; exit "${2:-2}"; } + +# The tracked .a2ml files, NUL-delimited end to end (a name with a newline in +# it stays one name, and is refused below), sorted, case-insensitive suffix. +a2ml_files() { + local -a statuses + if git -C "$1" ls-files -z | grep -zaiE '\.a2ml$' | LC_ALL=C sort -z; then + return 0 + else + statuses=("${PIPESTATUS[@]}") + [[ "${statuses[0]}" -eq 0 ]] || return "${statuses[0]}" + [[ "${statuses[1]}" -le 1 ]] || return "${statuses[1]}" + return "${statuses[2]}" + fi +} + +check_mode=0 +if [[ "${1:-}" == "--check" ]]; then check_mode=1; shift; fi +[[ $# -ge 1 && $# -le 2 ]] || usage +REPO="$1" +[[ -d "$REPO" ]] || die "not a directory: $REPO" 64 +git -C "$REPO" rev-parse --is-inside-work-tree >/dev/null 2>&1 || die "not a git work tree: $REPO" 64 +REPO="$(cd "$REPO" && git rev-parse --show-toplevel)" +REPO="$(cd "$REPO" && pwd -P)" + +if [[ "$check_mode" -ne 1 ]]; then + # Every refusal is recorded with jq, so it is needed before anything is + # classified, not only on kyX's path: without it there would be no report. + for tool in jq sha256sum; do + command -v "$tool" >/dev/null 2>&1 || die "$tool is required to record any outcome; nothing was changed" 69 + done + +fi + +# Materialize discovery so mapfile cannot hide a producer failure. Keep NULs +# in a file: command substitution cannot preserve them. +discovery="$(mktemp "${TMPDIR:-/tmp}/a2ml-discovery.XXXXXX")" +trap 'rm -f -- "$discovery"' EXIT +a2ml_files "$REPO" > "$discovery" || die "tracked .a2ml discovery failed; nothing was changed" "$?" +mapfile -d '' -t FILES < "$discovery" +rm -f -- "$discovery" +trap - EXIT + +if [[ "$check_mode" -eq 1 ]]; then + printf '%s\n' "${#FILES[@]}" + exit 0 +fi + +if [[ "${#FILES[@]}" -eq 0 ]]; then + echo "fix-a2ml-via-kyx: no .a2ml in $REPO; nothing to do" + exit 0 +fi + +for f in "${FILES[@]}"; do + case "$f" in + *$'\n'*|*$'\t'*|-*) die "refusing a tracked path with a control character or leading dash: ${f@Q}" 2 ;; + esac +done + +REPORT_DIR="${A2ML_REPORT_DIR:-}" +if [[ -z "$REPORT_DIR" ]]; then REPORT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/a2ml-kyx.XXXXXX")"; fi +mkdir -p "$REPORT_DIR" +# Physical paths on both sides: a symlink outside that points inside is inside. +REPORT_DIR="$(cd "$REPORT_DIR" && pwd -P)" +case "$REPORT_DIR/" in "$REPO"/*) die "A2ML_REPORT_DIR must be outside the repository" 64 ;; esac +for name in a2ml-report.json a2ml-pr.md a2ml-p1-issue.md; do + if [[ -L "$REPORT_DIR/$name" || ( -e "$REPORT_DIR/$name" && ! -f "$REPORT_DIR/$name" ) ]]; then + die "$REPORT_DIR/$name exists and is not a regular file; refusing to write through it" 64 + fi +done +# The reports must be writable BEFORE anything is staged: a run that stages a +# conversion and then cannot report it leaves a change nobody was told about. +probe="$(mktemp "$REPORT_DIR/.a2ml-probe.XXXXXX" 2>/dev/null)" \ + || die "cannot write to A2ML_REPORT_DIR $REPORT_DIR; nothing was changed" 73 +rm -f -- "$probe" + +WORK="$(mktemp -d "${TMPDIR:-/tmp}/a2ml-kyx-work.XXXXXX")" +STAGED=() # sources whose conversion is staged in the index +REPORTS_DONE=0 # set once every report is published +CREATED_TARGET=0 # set once THIS run created the target file (never a file that was already there) + +# Put the index and work tree back as they were before `src` was staged, and +# say whether that worked: every step is checked, then the result is. +undo_stage() { # src + local src="$1" status paths=("$1") + if [[ "$CREATED_TARGET" -eq 1 ]]; then + # The target is ours to remove only while it is still exactly what this + # run generated, in the work tree and in the index. An edit made since + # belongs to the owner: keep it and fail the rollback. + if [[ -e "$REPO/$TARGET" || -L "$REPO/$TARGET" ]]; then + if [[ -L "$REPO/$TARGET" ]] || ! cmp -s -- "$REPO/$TARGET" "$WORK/ECOSYSTEM.deed"; then + echo "$TARGET was changed after it was generated; left unchanged" >> "$WORK/undo.err"; return 1 + fi + fi + if [[ -n "$(git -C "$REPO" ls-files --cached -- "$TARGET" 2>> "$WORK/undo.err")" ]]; then + git -C "$REPO" show ":$TARGET" 2>> "$WORK/undo.err" | cmp -s -- - "$WORK/ECOSYSTEM.deed" \ + || { echo "$TARGET was changed in the index after it was generated; left unchanged" >> "$WORK/undo.err"; return 1; } + fi + paths=("$TARGET" "$src") + fi + git -C "$REPO" reset -q -- "${paths[@]}" 2>> "$WORK/undo.err" || return 1 + if [[ "$CREATED_TARGET" -eq 1 ]]; then + rm -f -- "$REPO/$TARGET" 2>> "$WORK/undo.err" || return 1 + fi + # A missing source is the expected result of git rm. If it still exists, + # preserve any edits made since preflight rather than checking over them. + if [[ -e "$REPO/$src" || -L "$REPO/$src" ]]; then + git -C "$REPO" diff --quiet -- "$src" 2>> "$WORK/undo.err" || return 1 + fi + git -C "$REPO" checkout -q -- "$src" 2>> "$WORK/undo.err" || return 1 + # The check itself must succeed: a failed `git status` prints nothing, and + # nothing would read as clean. + status="$(git -C "$REPO" status --porcelain -- "${paths[@]}" 2>> "$WORK/undo.err")" || return 1 + [[ -z "$status" ]] +} + +# A rollback that failed leaves a state this script cannot vouch for. Say so +# loudly, with its own exit code, and do nothing further. +inconsistent() { + printf 'fix-a2ml-via-kyx: INCONSISTENT: %s. The index and work tree of %s need a human (git status); nothing further was done.\n' "$1" "$REPO" >&2 + [[ -s "$WORK/undo.err" ]] && sed 's/^/fix-a2ml-via-kyx: /' "$WORK/undo.err" >&2 + exit 70 +} + +# On any exit before the reports are out, undo what was staged: a conversion +# is only left in the tree together with the PR body that describes it. +# shellcheck disable=SC2317 # invoked by the EXIT trap below +on_exit() { + local rc=$? src + trap - EXIT + if [[ "$REPORTS_DONE" -ne 1 && "${#STAGED[@]}" -gt 0 ]]; then + for src in "${STAGED[@]}"; do + if undo_stage "$src"; then + printf 'fix-a2ml-via-kyx: the run stopped before its reports were written; the staged conversion of %s was rolled back\n' "$src" >&2 + else + printf 'fix-a2ml-via-kyx: INCONSISTENT: the run stopped before its reports were written, and rolling back %s failed. The index and work tree of %s need a human (git status).\n' "$src" "$REPO" >&2 + rc=70 + fi + done + [[ "$rc" -ne 0 ]] || rc=1 + fi + rm -rf "$WORK" + exit "$rc" +} +trap on_exit EXIT + +: > "$WORK/converted.jsonl" +: > "$WORK/refused.jsonl" + +# Replace the random work directory with in a file, literally. +scrub() { + local text + text="$(<"$1")" + [[ -n "$text" ]] || return 0 + printf '%s\n' "${text//"$WORK"/}" > "$1" +} + +refuse() { # path family spec reason... + local path="$1" family="$2" spec="$3"; shift 3 + jq -nc --arg path "$path" --arg family "$family" --arg spec "$spec" \ + --args '{path:$path, family:$family, spec:(if $spec=="" then null else $spec end), reasons:$ARGS.positional}' \ + -- "$@" >> "$WORK/refused.jsonl" +} + +# family -> "family|spec|kyx(0|1)", spec empty when none; the table +# rhodibot's canon/a2ml.rs reports from, so the finding and the fixer agree. +# The delimiter is "|", not a tab: a tab is IFS whitespace, so `read` would +# collapse an empty spec and shift kyx into it. +family_of() { + local lower name + lower="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" + name="${lower##*/}" + case "$name" in + ecosystem.a2ml) printf 'ECOSYSTEM|meta-ecosystem-to-repo-deed.adoc|1\n' ;; + meta.a2ml) printf 'META|meta-ecosystem-to-repo-deed.adoc|0\n' ;; + state.a2ml) printf 'STATE|state-v1-decision.adoc|0\n' ;; + agentic.a2ml) printf 'AGENTIC|agentic-neurosym-playbook-to-repo-deed.adoc|0\n' ;; + neurosym.a2ml) printf 'NEUROSYM|agentic-neurosym-playbook-to-repo-deed.adoc|0\n' ;; + playbook.a2ml) printf 'PLAYBOOK|agentic-neurosym-playbook-to-repo-deed.adoc|0\n' ;; + clade.a2ml) printf 'CLADE|clade-to-repo-deed.adoc|0\n' ;; + 0-ai-manifest.a2ml) printf 'AI-MANIFEST|ai-manifest-decision.adoc|0\n' ;; + anchor.a2ml) printf 'ANCHOR||0\n' ;; + rsr-profile.a2ml) printf 'RSR-PROFILE||0\n' ;; + rsr-criteria-v2.a2ml) printf 'RSR-CRITERIA|rsr-criteria-to-praxis-deed.adoc|0\n' ;; + *) + if [[ "$lower" == *bot_directives/* ]]; then printf 'BOT-DIRECTIVES||0\n' + elif [[ "$name" == *file.a2ml && "$lower" == *contractiles/* ]]; then printf 'CONTRACTILE||0\n' + else printf 'UNMAPPED||0\n'; fi ;; + esac +} + +# Canonical ECOSYSTEM.a2ml sources (anything else is refused, not guessed). +is_canonical_ecosystem() { + case "$1" in + .machine_readable/descriptiles/ECOSYSTEM.a2ml|.machine_readable/6a2/ECOSYSTEM.a2ml|.machine_readable/ECOSYSTEM.a2ml) return 0 ;; + *) return 1 ;; + esac +} +TARGET=".machine_readable/descriptiles/ECOSYSTEM.deed" + +need_tools() { + local missing=() + [[ -n "${KYX:-}" && -f "${KYX:-}" ]] || missing+=("KYX (kyX's built kyx.bun.js)") + [[ "${KYX_COMMIT:-}" =~ ^[0-9a-f]{40}$ ]] || missing+=("KYX_COMMIT (40-hex)") + [[ -n "${DEED_LINT:-}" && -f "${DEED_LINT:-}" ]] || missing+=("DEED_LINT (standards deed_lint.js)") + [[ "${STANDARDS_COMMIT:-}" =~ ^[0-9a-f]{40}$ ]] || missing+=("STANDARDS_COMMIT (40-hex)") + command -v bun >/dev/null 2>&1 || missing+=("bun") + command -v jq >/dev/null 2>&1 || missing+=("jq") + if [[ "${#missing[@]}" -gt 0 ]]; then + printf 'fix-a2ml-via-kyx: a file needs kyX, and these are missing: %s\n' "${missing[*]}" >&2 + printf 'fix-a2ml-via-kyx: nothing was changed; this is an operator error, not the repository'"'"'s\n' >&2 + exit 69 + fi +} + +AFFIRMED="${AFFIRMED:-$(date -u +%Y-%m-%d)}" +[[ "$AFFIRMED" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || die "AFFIRMED must be YYYY-MM-DD" 64 +ACCEPT_DROPS="${A2ML_ACCEPT_SPEC_DROPS:-0}" +[[ "$ACCEPT_DROPS" == 0 || "$ACCEPT_DROPS" == 1 ]] || die "A2ML_ACCEPT_SPEC_DROPS must be 0 or 1" 64 + +# Exactly one chora deed at the root (D312), or none to convert against. It is +# taken from the index, not the filesystem: the PR carries only the deed and +# the deletion, so the chora kyX reads must be the committed one. +mapfile -d '' -t choras < <(git -C "$REPO" ls-files -z -- ':(glob)*_chora.deed') + +# Why `src` cannot be converted in this work tree right now, or nothing. Asked +# BEFORE kyX runs, so a failed `git add`/`git rm` cannot leave half a change. +preflight() { # src + local src="$1" dir st + # A failed `git status` prints nothing, which would read as clean: check + # that it ran before reading what it printed. Its stderr is kept apart: + # a harmless warning (a missing parent directory of an absent target) is + # not a change, and a real failure is quoted from it. + st="$(git -C "$REPO" status --porcelain -- "${choras[0]}" 2> "$WORK/status.err")" \ + || { echo "git status could not read ${choras[0]}, so its state is unknown: $(head -c 200 "$WORK/status.err")"; return; } + if [[ -n "$st" ]]; then + echo "${choras[0]} has uncommitted changes; kyX must read the committed chora deed the PR will reference."; return + fi + if [[ -L "$REPO/$src" ]]; then + echo "the source is a symlink; kyX must read content Git tracks."; return + fi + st="$(git -C "$REPO" status --porcelain -- "$src" 2> "$WORK/status.err")" \ + || { echo "git status could not read the source, so its state is unknown: $(head -c 200 "$WORK/status.err")"; return; } + if [[ -n "$st" ]]; then + echo "the source has uncommitted changes; kyX would convert content the PR does not delete."; return + fi + st="$(git -C "$REPO" status --porcelain -- "$TARGET" 2> "$WORK/status.err")" \ + || { echo "git status could not read the target, so its state is unknown: $(head -c 200 "$WORK/status.err")"; return; } + if [[ -n "$st" ]]; then + echo "$TARGET has uncommitted changes; they must be preserved."; return + fi + # Each directory on the way to the target must be absent or a real + # directory: a symlink would put the deed outside the work tree. + dir="$REPO" + for part in .machine_readable descriptiles; do + dir="$dir/$part" + if [[ -L "$dir" || ( -e "$dir" && ! -d "$dir" ) ]]; then + echo "${dir#"$REPO"/} is a symlink or not a directory; the deed is written only inside the work tree."; return + fi + done + if git -C "$REPO" check-ignore -q --no-index -- "$TARGET"; then + echo "$TARGET is gitignored here, so it could not be staged."; return + fi +} + +eco_sources=() +for f in "${FILES[@]}"; do + IFS='|' read -r fam spec kyx <<< "$(family_of "$f")" + if [[ "$kyx" != 1 ]]; then + if [[ -n "$spec" ]]; then + refuse "$f" "$fam" "$spec" "kyX has no front end for $fam yet; $spec decides where it goes. Do not hand-convert (D313)." + else + refuse "$f" "$fam" "" "no mapping spec decides $fam yet, and kyX has no front end for it. Do not hand-convert (D313)." + fi + continue + fi + if ! is_canonical_ecosystem "$f"; then + refuse "$f" "$fam" "$spec" "not at a canonical descriptile location, so it may describe another tree (a template, a vendored copy); kyX needs the chora deed of the tree it describes." + continue + fi + eco_sources+=("$f") +done + +if [[ "${#eco_sources[@]}" -gt 1 ]]; then + for f in "${eco_sources[@]}"; do + refuse "$f" ECOSYSTEM meta-ecosystem-to-repo-deed.adoc "more than one ECOSYSTEM.a2ml (${eco_sources[*]}) for one $TARGET; which is authoritative is a human decision." + done + eco_sources=() +fi + +for src in "${eco_sources[@]}"; do + spec=meta-ecosystem-to-repo-deed.adoc + if [[ "${#choras[@]}" -ne 1 ]]; then + refuse "$src" ECOSYSTEM "$spec" "kyX reads :canonical-name and the chora id from the repository's one *_chora.deed and never mints one (U1); found ${#choras[@]}." + continue + fi + if [[ -e "$REPO/$TARGET" || -L "$REPO/$TARGET" ]]; then + refuse "$src" ECOSYSTEM "$spec" "$TARGET already exists; it is never overwritten. Reconcile the two by hand-review, not by hand-conversion." + continue + fi + blocked="$(preflight "$src")" + if [[ -n "$blocked" ]]; then + refuse "$src" ECOSYSTEM "$spec" "$blocked" + continue + fi + need_tools + out="$WORK/ECOSYSTEM.deed"; rm -f "$out" "$WORK/kyx.err" "$WORK/ast.json" + rc=0 + bun "$KYX" a2ml ecosystem "$REPO/$src" --chora "$REPO/${choras[0]}" --affirmed "$AFFIRMED" --out "$out" \ + > "$WORK/kyx.out" 2> "$WORK/kyx.err" || rc=$? + # The work directory is random per run; a reason that names it would give + # the same refusal a new fingerprint every time, and a duplicate P1 issue. + scrub "$WORK/kyx.err" + mapfile -t kyx_err < "$WORK/kyx.err" + if [[ "$rc" -eq 1 ]]; then + mapfile -t why < <(grep '^kyx: refused: ' "$WORK/kyx.err" | sed 's/^kyx: refused: /kyX refused: /') + [[ "${#why[@]}" -gt 0 ]] || why=("kyX refused it (exit 1) without a reason line: ${kyx_err[*]:-}") + refuse "$src" ECOSYSTEM "$spec" "${why[@]}" + continue + elif [[ "$rc" -ne 0 || ! -s "$out" ]]; then + refuse "$src" ECOSYSTEM "$spec" "kyX failed (exit $rc), so nothing is known about the conversion: ${kyx_err[*]:-no output}" + continue + fi + mapfile -t drops < <(grep '^kyx: dropped' "$WORK/kyx.err" || true) + if [[ "${#drops[@]}" -gt 0 && "$ACCEPT_DROPS" != 1 ]]; then + refuse "$src" ECOSYSTEM "$spec" "not lossless: kyX dropped content the deed does not carry (spec §4); A2ML_ACCEPT_SPEC_DROPS is off:" "${drops[@]}" + continue + fi + if ! bun "$DEED_LINT" --ecosystem --root "$REPO" "$out" > "$WORK/lint.out" 2>&1; then + scrub "$WORK/lint.out" + mapfile -t lint < "$WORK/lint.out" + refuse "$src" ECOSYSTEM "$spec" "kyX's output does not pass standards deed_lint --ecosystem --root (canon grammar):" "${lint[@]}" + continue + fi + for k9 in "$WORK"/*.k9 "$WORK"/*.k9.ncl; do + [[ -e "$k9" ]] || continue + if [[ -z "${K9_VALIDATE:-}" ]] || ! bash "$K9_VALIDATE" --strict "$k9" > "$WORK/k9.out" 2>&1; then + refuse "$src" ECOSYSTEM "$spec" "a k9 output does not pass standards k9-validate.sh --strict (or K9_VALIDATE is unset)" + continue 2 + fi + done + if ! bun "$DEED_LINT" --ast-json "$out" > "$WORK/ast.json" 2> "$WORK/ast.err"; then + refuse "$src" ECOSYSTEM "$spec" "deed_lint --ast-json could not read kyX's output: $(scrub "$WORK/ast.err"; head -c 400 "$WORK/ast.err")" + continue + fi + crc=0 + census_json="$(bun "$CENSUS" check "$REPO/$src" "$WORK/ast.json" 2> "$WORK/census.err")" || crc=$? + if [[ "$crc" -ne 0 ]]; then + if [[ "$crc" -eq 1 ]]; then + mapfile -t lost < <(jq -r '.short[] | "value \(.value | tojson) occurs \(.source) time(s) in the source, \(.deed) in the deed"' <<< "$census_json") + refuse "$src" ECOSYSTEM "$spec" "not lossless: the value census found source values the deed carries fewer times than the source:" "${lost[@]}" + else + refuse "$src" ECOSYSTEM "$spec" "the value census could not read the source, so losslessness is unproven: $(scrub "$WORK/census.err"; head -c 400 "$WORK/census.err")" + fi + continue + fi + # Every gate passed: stage the deed and the deletion, all or nothing. The + # preflight makes a failure here unlikely; if one happens anyway, the tree + # and index are put back (checked) and the file is refused with the reason. + if ! { mkdir -p "$REPO/$(dirname "$TARGET")" \ + && ( set -o noclobber; cat -- "$out" > "$REPO/$TARGET" ) \ + && CREATED_TARGET=1 \ + && git -C "$REPO" add -- "$TARGET" \ + && git -C "$REPO" rm -q -- "$src"; } 2> "$WORK/stage.err"; then + scrub "$WORK/stage.err" + undo_stage "$src" \ + || inconsistent "staging the conversion of $src failed ($(head -c 200 "$WORK/stage.err")), and rolling it back failed too" + refuse "$src" ECOSYSTEM "$spec" "staging the conversion failed and was rolled back: $(head -c 400 "$WORK/stage.err")" + continue + fi + STAGED+=("$src") + jq -nc --arg source "$src" --arg target "$TARGET" --arg kyx "$KYX_COMMIT" --arg std "$STANDARDS_COMMIT" \ + --arg affirmed "$AFFIRMED" --argjson census "$census_json" --args \ + '{source:$source, target:$target, kyx_commit:$kyx, standards_commit:$std, affirmed:$affirmed, + values_checked:$census.source_values, spec_drops:$ARGS.positional}' \ + -- "${drops[@]}" >> "$WORK/converted.jsonl" +done + +repo_name="$(basename "$REPO")" +jq -n --arg repo "$repo_name" --arg accept "$ACCEPT_DROPS" \ + --slurpfile converted "$WORK/converted.jsonl" --slurpfile refused "$WORK/refused.jsonl" \ + '{repo:$repo, accept_spec_drops:($accept=="1"), converted:$converted, refused:$refused}' \ + > "$WORK/a2ml-report.json" + +# Each report is written to a new file in REPORT_DIR and renamed over the old +# one: a rename replaces a link instead of writing through it. +publish() { # source name + local tmp + tmp="$(mktemp "$REPORT_DIR/.$2.XXXXXX")" + cat -- "$1" > "$tmp" + mv -fT -- "$tmp" "$REPORT_DIR/$2" +} +publish "$WORK/a2ml-report.json" a2ml-report.json +rm -f -- "$REPORT_DIR/a2ml-pr.md" "$REPORT_DIR/a2ml-p1-issue.md" + +n_conv="$(jq '.converted | length' "$REPORT_DIR/a2ml-report.json")" +n_ref="$(jq '.refused | length' "$REPORT_DIR/a2ml-report.json")" + +if [[ "$n_conv" -gt 0 ]]; then + jq -r ' + "## Retire A2ML: kyX conversion\n", + "A2ML is retired (D99, D269c, D312, D313). Each file below was converted by **kyX** under its standards mapping spec. Nothing was converted by hand.\n", + "| Source (deleted) | Deed (added) | kyX | standards | values checked |", + "|---|---|---|---|---|", + (.converted[] | "| `\(.source)` | `\(.target)` | `\(.kyx_commit[0:12])` | `\(.standards_commit[0:12])` | \(.values_checked) |"), + "", + "Gates passed for every file: kyX exit 0 (fail-closed), " + (if .accept_spec_drops then "spec §4 drops accepted by A2ML_ACCEPT_SPEC_DROPS=1 and listed below" else "no content dropped" end) + ", `deed_lint --ecosystem --root` (canon grammar), and the value census (every source value is in the deed, as often as in the source).", + (if any(.converted[]; (.spec_drops|length) > 0) then ("\nDropped under spec §4:\n" + ([.converted[] | .spec_drops[] | "- " + .] | join("\n"))) else empty end), + "", + "Review it like any PR: the deed is owner-facing, and :affirmed records the date of this conversion.\n", + "_Opened by gitbot-fleet `scripts/fix-a2ml-via-kyx.sh` (opportunistic A2ML retirement)._" + ' "$REPORT_DIR/a2ml-report.json" > "$WORK/a2ml-pr.md" + publish "$WORK/a2ml-pr.md" a2ml-pr.md +fi + +if [[ "$n_ref" -gt 0 ]]; then + fp="$(jq -cS '[.refused[] | {path, reasons}]' "$REPORT_DIR/a2ml-report.json" | sha256sum | cut -c1-16)" + jq -r --arg fp "$fp" ' + "", + "## P1: retired A2ML that kyX cannot convert (\(.refused | length) file(s))\n", + "A2ML is retired (D99, D269c, D312, D313), and conversion is kyX'"'"'s front end only (D313). These files could not be converted **losslessly by kyX**, so nothing was changed. **Do not hand-convert them.**\n", + (.refused | group_by(.family)[] | + "### \(.[0].family)" + (if .[0].spec then " — `1-formats/deed/mappings/\(.[0].spec)`" else " — no mapping spec yet" end) + "\n", + (.[] | "- `\(.path)`\n" + ([.reasons[] | " - " + .] | join("\n"))), + ""), + "What unblocks each: a kyX front end for the family (metadatastician/kyx), a mapping spec where there is none, or a fix to the source that kyX names.\n", + "_Opened by gitbot-fleet `scripts/fix-a2ml-via-kyx.sh`; it will not open a second issue for the same refusal._" + ' "$REPORT_DIR/a2ml-report.json" > "$WORK/a2ml-p1-issue.md" + publish "$WORK/a2ml-p1-issue.md" a2ml-p1-issue.md +fi +REPORTS_DONE=1 + +echo "fix-a2ml-via-kyx: $n_conv converted, $n_ref refused; reports in $REPORT_DIR" +if [[ "$n_ref" -gt 0 ]]; then + jq -r '.refused[] | "REFUSED\t\(.path)\t\(.reasons[0])"' "$REPORT_DIR/a2ml-report.json" >&2 +fi +if [[ "$n_ref" -eq 0 ]]; then exit 0; fi +if [[ "$n_conv" -gt 0 ]]; then exit 3; fi +exit 2 diff --git a/scripts/fix-script-registry.json b/scripts/fix-script-registry.json index 6892bae9..7735498b 100644 --- a/scripts/fix-script-registry.json +++ b/scripts/fix-script-registry.json @@ -56,7 +56,8 @@ "RetiredDescriptilePolicy": "fix-retired-descriptile-policy.sh", "ToolchainInputMissing": "fix-rust-toolchain-input.sh", "StaleReusablePin": "fix-stale-reusable-pin.sh", - "LockSelfShaRef": "fix-lock-self-sha-refs.sh" + "LockSelfShaRef": "fix-lock-self-sha-refs.sh", + "RetiredA2ML": "fix-a2ml-via-kyx.sh" }, "by_recipe": { "recipe-actions-allow-all": "fix-actions-policy.sh", @@ -130,7 +131,8 @@ "pin_integrity/self_sha_lock_record": "fix-lock-self-sha-refs.sh", "recipe-workflow-linter-spdx-rule": "fix-workflow-linter-spdx-rule.sh", "recipe-sonarqube-token-guard": "fix-sonarqube-token-guard.sh", - "recipe-workflow-split-comment-keys": "fix-split-comment-keys.sh" + "recipe-workflow-split-comment-keys": "fix-split-comment-keys.sh", + "recipe-a2ml-kyx": "fix-a2ml-via-kyx.sh" } } } diff --git a/scripts/lib/a2ml-census.js b/scripts/lib/a2ml-census.js new file mode 100644 index 00000000..09374429 --- /dev/null +++ b/scripts/lib/a2ml-census.js @@ -0,0 +1,228 @@ +// SPDX-License-Identifier: MPL-2.0 +// SPDX-FileCopyrightText: 2026 Jonathan D.A. Jewell +// +// a2ml-census.js — the value census behind fix-a2ml-via-kyx.sh's "no +// information is lost" gate. Runs under Bun, zero dependencies. +// +// bun a2ml-census.js check SOURCE.a2ml AST.json +// +// AST.json is `bun deed_lint.js --ast-json OUT.deed` (standards). Every +// non-empty scalar VALUE in the A2ML source (strings, numbers, booleans, dates) +// must appear as a value in the deed at least as many times as in the source +// (a multiset check, so a field collapsed into another is caught). Keys and +// table names are not values (a mapping spec renames them), and comments are not values (kyX's own +// drop report covers those). Prints one JSON object +// {"source_values": N, "missing": [value, ...], +// "short": [{"value", "source": count, "deed": count}, ...]} +// and exits 0 when nothing is missing, 1 when something is, 2 on a usage or +// parse error. It is a NECESSARY condition for losslessness, checked +// independently of kyX: it errs towards reporting loss (a value a spec +// legitimately rewrites shows as missing), never towards hiding it. +// +// This is a verifier, not a converter: it never writes a deed. + +import { readFileSync } from "node:fs"; + +function fail(msg) { + process.stderr.write(`a2ml-census: ${msg}\n`); + process.exit(2); +} + +// --- A2ML (the retired TOML-like record dialect) value lexer -------------- + +export function a2mlValues(text) { + const s = text.replace(/\r\n/g, "\n"); + const out = []; + let i = 0; + let line = 1; + const peek = (k = 0) => s[i + k]; + const err = (m) => fail(`A2ML line ${line}: ${m}`); + const skipWs = (newlines) => { + for (;;) { + const c = peek(); + if (c === " " || c === "\t") i++; + else if (newlines && c === "\n") { i++; line++; } + else if (c === "#") { while (i < s.length && peek() !== "\n") i++; } + else return; + } + }; + // One escape, after its backslash; the same rules in "..." and """...""". + const escape = () => { + const e = s[i++]; + const map = { n: "\n", t: "\t", r: "\r", '"': '"', "\\": "\\", b: "\b", f: "\f" }; + if (e in map) return map[e]; + if (e === "u" || e === "U") { + const n = e === "u" ? 4 : 8; + const hex = s.slice(i, i + n); + if (!/^[0-9A-Fa-f]+$/.test(hex) || hex.length !== n) err("bad \\u escape"); + const cp = parseInt(hex, 16); + if (cp > 0x10ffff || (cp >= 0xd800 && cp <= 0xdfff)) err("\\u escape is not a Unicode scalar value"); + i += n; + return String.fromCodePoint(cp); + } + err(`bad escape \\${e}`); + }; + const basic = () => { + // after the opening quote + let v = ""; + for (;;) { + const c = s[i++]; + if (c === undefined || c === "\n") err("unterminated string"); + if (c === '"') return v; + if (c === "\\") v += escape(); + else v += c; + } + }; + const multi = (q) => { + // after the opening triple quote; a newline right after it is trimmed + if (peek() === "\n") { i++; line++; } + let v = ""; + for (;;) { + if (i >= s.length) err("unterminated multi-line string"); + if (s.startsWith(q + q + q, i)) { i += 3; return v; } + const c = s[i++]; + if (c === "\n") line++; + if (q === '"' && c === "\\") { + // A line-ending backslash is the last non-whitespace character on its + // line (TOML 1.0), so spaces or tabs may follow it before the break. + let j = i; + while (s[j] === " " || s[j] === "\t") j++; + if (s[j] === "\n") { // trim the padding, the break and the next indent + i = j; + while (peek() === "\n" || peek() === " " || peek() === "\t") { if (peek() === "\n") line++; i++; } + continue; + } + v += escape(); + } else v += c; + } + }; + const value = () => { + skipWs(false); + const c = peek(); + if (s.startsWith('"""', i)) { i += 3; out.push(multi('"')); return; } + if (s.startsWith("'''", i)) { i += 3; out.push(multi("'")); return; } + if (c === '"') { i++; out.push(basic()); return; } + if (c === "'") { + i++; + const j = s.indexOf("'", i); + const nl = s.indexOf("\n", i); + if (j < 0 || (nl >= 0 && nl < j)) err("unterminated literal string"); + out.push(s.slice(i, j)); + i = j + 1; + return; + } + if (c === "[") { + i++; + for (;;) { + skipWs(true); + if (peek() === "]") { i++; return; } + value(); + skipWs(true); + if (peek() === ",") { i++; continue; } + if (peek() === "]") { i++; return; } + err("expected , or ] in an array"); + } + } + if (c === "{") { + i++; + for (;;) { + skipWs(false); + if (peek() === "}") { i++; return; } + key(); + value(); + skipWs(false); + if (peek() === ",") { i++; continue; } + if (peek() === "}") { i++; return; } + err("expected , or } in an inline table"); + } + } + // bare scalar: bool, number, date + let j = i; + while (j < s.length && !/[\s,\]}#]/.test(s[j])) j++; + const tok = s.slice(i, j); + if (tok === "") err("expected a value"); + out.push(tok); + i = j; + }; + const key = () => { + // a bare, quoted or dotted key, then '=' + skipWs(false); + while (i < s.length && peek() !== "=") { + if (peek() === "\n") err("a key without '='"); + if (peek() === '"') { i++; basic(); continue; } + if (peek() === "'") { const j = s.indexOf("'", i + 1); if (j < 0) err("bad key"); i = j + 1; continue; } + i++; + } + if (peek() !== "=") err("expected '='"); + i++; + }; + while (i < s.length) { + skipWs(true); + if (i >= s.length) break; + if (peek() === "[" || peek() === "@") { + // A table header is a name, not a value. An `@directive(...)` line is + // kyX's drop doctrine's to report (spec §4), so the drop gate covers it. + while (i < s.length && peek() !== "\n") i++; + continue; + } + key(); + value(); + skipWs(false); + if (i < s.length && peek() !== "\n") err("trailing content after a value"); + } + return out; +} + +// --- deed AST (deed_lint.js --ast-json) value collector ------------------- + +export function deedValues(ast) { + const out = []; + const node = (n) => { + if (!Array.isArray(n) || n.length < 2) return; + const [kind, v] = n; + if (kind === "list") { for (const x of v) node(x); return; } + if (kind === "boolean") { out.push(v === "#t" ? "true" : "false"); return; } + if (typeof v === "string") out.push(v); + }; + const items = (xs) => { + for (const it of xs) { + if (it[0] === "field") node(it[2]); + else if (it[0] === "clause") items(it[2]); + } + }; + if (!ast || !Array.isArray(ast.items)) fail("AST has no items (is it deed_lint --ast-json output?)"); + items(ast.items); + return out; +} + +// A multiset count: how many times each value occurs. +function tally(values) { + const n = new Map(); + for (const v of values) n.set(v, (n.get(v) ?? 0) + 1); + return n; +} + +export function census(sourceText, ast) { + const want = a2mlValues(sourceText).filter((v) => v !== ""); + // Counted, not just present: two source fields holding the same value must + // be two deed values, or a field collapsed into another passes unseen. + const need = tally(want); + const have = tally(deedValues(ast)); + const short = []; + for (const [value, source] of need) { + const deed = have.get(value) ?? 0; + if (deed < source) short.push({ value, source, deed }); + } + return { source_values: want.length, missing: short.map((s) => s.value), short }; +} + +if (import.meta.main) { + const [cmd, src, astPath] = process.argv.slice(2); + if (cmd !== "check" || !src || !astPath) fail("usage: a2ml-census.js check SOURCE.a2ml AST.json"); + let text, ast; + try { text = readFileSync(src, "utf8"); } catch (e) { fail(`cannot read ${src}: ${e.message}`); } + try { ast = JSON.parse(readFileSync(astPath, "utf8")); } catch (e) { fail(`cannot read ${astPath}: ${e.message}`); } + const r = census(text, ast); + process.stdout.write(JSON.stringify(r) + "\n"); + process.exit(r.missing.length === 0 ? 0 : 1); +} diff --git a/scripts/tests/a2ml-census-test.sh b/scripts/tests/a2ml-census-test.sh new file mode 100644 index 00000000..f9f5a355 --- /dev/null +++ b/scripts/tests/a2ml-census-test.sh @@ -0,0 +1,114 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# Unit tests for the value census, independent of kyX and standards. +# Usage: bash scripts/tests/a2ml-census-test.sh (Bun only; normally < 5s). +set -euo pipefail +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +command -v bun >/dev/null || { echo 'NOT RUN: needs bun' >&2; exit 2; } +CENSUS="$ROOT/scripts/lib/a2ml-census.js" bun run - <<'JS' +import assert from 'node:assert/strict'; +import { mkdtempSync, writeFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { spawnSync } from 'node:child_process'; +const { a2mlValues, deedValues, census } = await import(process.env.CENSUS); +let passed = 0; +function test(name, fn) { fn(); console.log(`PASS: ${name}`); passed++; } +const field = (key, kind, value) => ['field', key, [kind, value]]; +const ast = (...items) => ({ items }); +const strings = (...values) => ast(...values.map(v => field('renamed', 'string', v))); + +const lexCases = [ + ['empty input and comments', '# ignored = "value"\r\n\t\r\n', []], + ['headers, directives and keys are not values', '[project]\n[[related.projects]]\n@drop("ignored")\n"key=with equals".name = "kept" # ignored\n', ['kept']], + ['literal strings preserve escapes and comment markers', "a = 'C:\\tmp#file'\nb = '''\nraw\\n\nline'''\n", ['C:\\tmp#file', 'raw\\n\nline']], + ['CRLF normalizes before multiline parsing', 'a = """\r\nfirst\r\nsecond"""\r\n', ['first\nsecond']], + ['nested containers collect only values in order', 'a = [{name = "x", nested = [false, 0]}, ["y",], {}]\n', ['x', 'false', '0', 'y']], + ['bare scalars retain their spelling', 'a = [-1, 1.20, 2026-11-17, true, false]\n', ['-1', '1.20', '2026-11-17', 'true', 'false']], + ['basic escapes decode', String.raw`a = "\b\t\n\f\r\"\\"`, ['\b\t\n\f\r"\\']], + ['Unicode scalar boundaries decode', String.raw`a = "\u0000\uD7FF\uE000\U0010FFFF"`, ['\0\uD7FF\uE000\u{10FFFF}']], + ['continuations trim blank lines and indentation', 'a = """\none \\\n \t\n two"""', ['one two']], +]; +for (const [name, source, expected] of lexCases) test(name, () => assert.deepEqual(a2mlValues(source), expected)); + +test('AST recursively collects clauses and lists, normalizing booleans', () => { + assert.deepEqual(deedValues(ast(['clause', 'ignored-name', [ + field('ignored-key', 'list', [['string', 'x'], ['list', [['boolean', '#t'], ['boolean', '#f']]], ['integer', '0']]), + ['clause', 'nested', [field('id', 'uuid', 'id-value')]], + ]])), ['x', 'true', 'false', '0', 'id-value']); +}); +test('AST names and unknown items cannot satisfy missing values', () => { + assert.deepEqual(census('a = "needle"', ast(field('needle', 'string', 'other'), ['unknown', 'needle'])), { + source_values: 1, missing: ['needle'], short: [{ value: 'needle', source: 1, deed: 0 }], + }); +}); +test('only empty values are excluded; whitespace-only values are counted', () => { + // A value that is only spaces or a tab is still a value: a deed that drops + // it has lost something, so it must show as a shortfall. + assert.deepEqual(census('a = ["", " ", "\\t", "x"]', strings('x')), { + source_values: 3, missing: [' ', '\t'], + short: [{value: ' ', source: 1, deed: 0}, {value: '\t', source: 1, deed: 0}], + }); + assert.deepEqual(census('a = ["", " ", "\\t", "x"]', strings('x', ' ', '\t')).missing, []); +}); +test('nonempty values retain significant whitespace', () => { + assert.deepEqual(census('a = " x "', strings('x')).missing, [' x ']); +}); +test('duplicate deficits report exact counts and stable first-seen order', () => { + assert.deepEqual(census('a = ["b", "a", "b", "a", "b"]', strings('b', 'a')), { + source_values: 5, missing: ['b', 'a'], short: [{value: 'b', source: 3, deed: 1}, {value: 'a', source: 2, deed: 1}], + }); +}); +test('surplus deed values and reordered fields are permitted', () => { + assert.deepEqual(census('a = ["x", "x", true, 0]', ast(...strings('extra', 'x', 'x', 'x').items, field('n', 'integer', '0'), field('b', 'boolean', '#t'))), {source_values: 4, missing: [], short: []}); +}); +test('census does not mutate caller AST', () => { + const input = strings('x'); const before = structuredClone(input); + census('a = "x"', input); assert.deepEqual(input, before); +}); + +// Parse failures exit the process: exercise the public CLI in isolated children. +const tmp = mkdtempSync(join(tmpdir(), 'a2ml-census-test-')); +try { + const source = join(tmp, 'source.a2ml'), deed = join(tmp, 'ast.json'); + const cli = (...args) => { + const r = spawnSync(process.execPath, [process.env.CENSUS, ...args], {encoding: 'utf8', timeout: 3000}); + assert.equal(r.error, undefined); assert.equal(r.signal, null); return r; + }; + writeFileSync(deed, JSON.stringify(strings('x'))); + const invalid = [ + ['missing assignment', 'oops\n', /key without '='/], + ['missing value', 'a =', /expected a value/], + ['unterminated basic string', 'a = "oops', /unterminated string/], + ['newline in literal', "a = 'one\ntwo'", /unterminated literal/], + ['unterminated multiline', 'a = """oops', /unterminated multi-line/], + ['unknown escape', String.raw`a = "\q"`, /bad escape/], + ['short Unicode escape', String.raw`a = "\u123"`, /bad \\u escape/], + ['surrogate escape', String.raw`a = "\uD800"`, /not a Unicode scalar/], + ['out of range escape', String.raw`a = "\U00110000"`, /not a Unicode scalar/], + ['missing array comma', 'a = ["x" "y"]', /expected , or \]/], + ['missing table comma', 'a = {b = "x" c = "y"}', /expected , or }/], + ['trailing content', 'a = "x" junk', /trailing content/], + ['line number after CRLF', '# comment\r\na = "oops\r\n', /A2ML line 2/], + ]; + for (const [name, text, error] of invalid) test(`CLI rejects ${name}`, () => { + writeFileSync(source, text); const r = cli('check', source, deed); + assert.equal(r.status, 2); assert.equal(r.stdout, ''); assert.match(r.stderr, error); + }); + for (const [name, args] of [['no arguments', []], ['wrong command', ['convert', source, deed]], ['missing AST argument', ['check', source]]]) { + test(`CLI usage: ${name}`, () => { const r = cli(...args); assert.equal(r.status, 2); assert.match(r.stderr, /usage:/); }); + } + test('CLI unreadable source', () => { const r = cli('check', join(tmp, 'absent'), deed); assert.equal(r.status, 2); assert.match(r.stderr, /cannot read/); }); + writeFileSync(source, 'a = "x"'); + test('CLI invalid JSON', () => { writeFileSync(deed, '{'); const r = cli('check', source, deed); assert.equal(r.status, 2); assert.match(r.stderr, /cannot read/); }); + test('CLI rejects AST without items', () => { writeFileSync(deed, '{}'); const r = cli('check', source, deed); assert.equal(r.status, 2); assert.match(r.stderr, /AST has no items/); }); + for (const [name, values, status, expected] of [ + ['matching values', ['x'], 0, {source_values: 1, missing: [], short: []}], + ['missing values', [], 1, {source_values: 1, missing: ['x'], short: [{value: 'x', source: 1, deed: 0}]}], + ]) test(`CLI ${name}: status and JSON`, () => { + writeFileSync(deed, JSON.stringify(strings(...values))); const r = cli('check', source, deed); + assert.equal(r.status, status); assert.equal(r.stderr, ''); assert.deepEqual(JSON.parse(r.stdout), expected); + }); +} finally { rmSync(tmp, { recursive: true, force: true }); } +console.log(`a2ml-census-test: ${passed} passed`); +JS diff --git a/scripts/tests/fix-a2ml-via-kyx-test.sh b/scripts/tests/fix-a2ml-via-kyx-test.sh new file mode 100644 index 00000000..71022553 --- /dev/null +++ b/scripts/tests/fix-a2ml-via-kyx-test.sh @@ -0,0 +1,435 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# +# Tests for scripts/fix-a2ml-via-kyx.sh and scripts/lib/a2ml-census.js. +# +# kyX is a STUB here (a Bun script driven by STUB_KYX): these tests pin the +# fixer's gates and outputs, not kyX's translation, which kyX's own acceptance +# suite covers (metadatastician/kyx tests/kyx/run.sh). The canon grammar gate +# uses the REAL standards deed_lint.js: set DEED_LINT, or STANDARDS to a +# metadatastician/standards checkout. Without it the suite does not run and +# exits 2 (NOT RUN), never 0. +# +# Usage: STANDARDS=/path/to/standards bash scripts/tests/fix-a2ml-via-kyx-test.sh +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +FIX="$ROOT/scripts/fix-a2ml-via-kyx.sh" +CENSUS="$ROOT/scripts/lib/a2ml-census.js" +DEED_LINT="${DEED_LINT:-${STANDARDS:+$STANDARDS/1-formats/deed/tools/deed_lint.js}}" +if [[ -z "$DEED_LINT" || ! -f "$DEED_LINT" ]] || ! command -v bun >/dev/null || ! command -v jq >/dev/null; then + echo "NOT RUN: needs bun, jq and the real standards deed_lint.js (set STANDARDS or DEED_LINT)" >&2 + exit 2 +fi +export DEED_LINT +export STANDARDS_COMMIT=b49a33e991f37a2249d873cbdc02c27a08f05d6d +export KYX_COMMIT=731e63ab4c3b02a6e4751de97208ea65eb8bdf6e +export AFFIRMED=2026-11-17 + +pass=0 fail=0 +TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT +ok() { echo "PASS: $1"; pass=$((pass+1)); } +no() { echo "FAIL: $1"; fail=$((fail+1)); } +check() { if [[ "$2" == "$3" ]]; then ok "$1"; else no "$1 (want ${2@Q}, got ${3@Q})"; fi; } + +CHORA=';; SPDX-License-Identifier: MPL-2.0 +(repo-deed + :schema-version "1.0.0" + :canonical-name "demo" + :repo-uuid #uuid"00000000-0000-8000-8000-0000000000a1" + (identity :owner "hyperpolymath" :primary-forge github :prefixed-name "demo"))' + +SOURCE='# SPDX-License-Identifier: MPL-2.0 +[project] +name = "demo" +purpose = "A demo used by the fixer tests." +[position-in-ecosystem] +ecosystem = "hyperpolymath" +type = "tool" +[[related-projects.projects]] +name = "standards" +relationship = "standard-source" +notes = "Reads its grammar."' + +# The deed text is data with literal quotes, exported to the stub as-is. +# shellcheck disable=SC2089,SC2090 +GOOD_DEED=';; SPDX-License-Identifier: MPL-2.0 +(descriptile-deed + :schema-version "1.0.0" + :canonical-name "demo" + :of-chora #uuid"00000000-0000-8000-8000-0000000000a1" + :affirmed "2026-11-17" :affirmed-by agent + :project "demo" + :ecosystem "hyperpolymath" + :position-type "tool" + :purpose "A demo used by the fixer tests." + (related :name "standards" :relationship standard-source :notes "Reads its grammar."))' + +# The stub: STUB_KYX = ok | refuse | drop | lossy | badgrammar | crash +cat > "$TMP/kyx.js" <<'JS' +import { writeFileSync } from "node:fs"; +const a = process.argv.slice(2); +const out = a[a.indexOf("--out") + 1]; +const mode = process.env.STUB_KYX; +const good = process.env.GOOD_DEED; +if (a[0] !== "a2ml" || a[1] !== "ecosystem" || !a.includes("--chora") || !a.includes("--affirmed")) { + process.stderr.write("kyx: usage\n"); process.exit(2); +} +if (mode === "refuse") { process.stderr.write("kyx: refused: line 3: [extra] is not in §3\nkyx: 1 reason(s); nothing written\n"); process.exit(1); } +if (mode === "crash") { process.stderr.write("kyx: cannot read source\n"); process.exit(2); } +let text = good; +if (mode === "lossy") text = good.replace('"A demo used by the fixer tests."', '"A demo."'); +if (mode === "badgrammar") text = good.replace("standard-source", "depends-on"); +writeFileSync(out, text + "\n"); +if (mode === "drop") process.stderr.write("kyx: dropped by the drop doctrine (spec §4): 1 comment line(s), 0 directive(s)\n"); +process.exit(0); +JS +# shellcheck disable=SC2090 +export KYX="$TMP/kyx.js" GOOD_DEED + +mkrepo() { # name -> path of a fresh repo with chora + sources + local r="$TMP/$1"; rm -rf "$r"; mkdir -p "$r/.machine_readable/descriptiles" "$r/contractiles/must" + git -C "$r" init -q; git -C "$r" config user.email t@t; git -C "$r" config user.name t + printf '%s\n' "$CHORA" > "$r/demo_chora.deed" + printf '%s\n' "$SOURCE" > "$r/.machine_readable/descriptiles/ECOSYSTEM.a2ml" + printf '[state]\nphase = "beta"\n' > "$r/.machine_readable/descriptiles/STATE.a2ml" + printf '# x\n' > "$r/contractiles/must/Mustfile.a2ml" + git -C "$r" add -A; git -C "$r" commit -qm init + printf '%s\n' "$r" +} +run() { # repo -> sets RC, REPORT + REPORT="$TMP/report-$RANDOM"; mkdir -p "$REPORT"; RC=0 + A2ML_REPORT_DIR="$REPORT" bash "$FIX" "$1" > "$TMP/out" 2> "$TMP/err" || RC=$? +} +staged() { git -C "$1" diff --cached --name-status | tr '\t' ' ' | LC_ALL=C sort | tr '\n' ';'; } + +# --- the census on its own --------------------------------------------------- +printf '%s\n' "$GOOD_DEED" > "$TMP/ECOSYSTEM.deed" +bun "$DEED_LINT" --ast-json "$TMP/ECOSYSTEM.deed" > "$TMP/ast.json" +printf '%s\n' "$SOURCE" > "$TMP/src.a2ml" +check "census: every source value is in the good deed" 0 "$(bun "$CENSUS" check "$TMP/src.a2ml" "$TMP/ast.json" >/dev/null; echo $?)" +printf '%s\nextra = "never mapped"\n' "$SOURCE" > "$TMP/src2.a2ml" +check "census: a value the deed lacks is reported" '["never mapped"]' "$(bun "$CENSUS" check "$TMP/src2.a2ml" "$TMP/ast.json" | jq -c .missing || true)" +printf 'a = """\nmulti\nline"""\nb = [ "x", # c\n "y" ]\nc = { d = 1, e = true }\n@directive(skip)\nf = \x27lit\x27\n' > "$TMP/src3.a2ml" +check "census: multi-line, arrays with comments, inline tables, literals; directives skipped" \ + '["multi\nline","x","y","1","true","lit"]' \ + "$(bun -e 'const {a2mlValues} = await import(process.argv[1]); console.log(JSON.stringify(a2mlValues(require("fs").readFileSync(process.argv[2],"utf8"))))' "$CENSUS" "$TMP/src3.a2ml")" +printf 'a = """\nA\\u00e9\\U0001F600"""\nb = "\\u00e9"\n' > "$TMP/src4.a2ml" +check "census: \\u and \\U escapes decode in multi-line strings as in single-line ones" \ + '["Aé😀","é"]' \ + "$(bun -e 'const {a2mlValues} = await import(process.argv[1]); console.log(JSON.stringify(a2mlValues(require("fs").readFileSync(process.argv[2],"utf8"))))' "$CENSUS" "$TMP/src4.a2ml")" +printf 'a = """\none \\ \t\n two \\\n three"""\n' > "$TMP/pad.a2ml" +check "census: a line-ending backslash with spaces or tabs after it is still a continuation" \ + '["one two three"]' \ + "$(bun -e 'const {a2mlValues} = await import(process.argv[1]); console.log(JSON.stringify(a2mlValues(require("fs").readFileSync(process.argv[2],"utf8"))))' "$CENSUS" "$TMP/pad.a2ml")" +printf 'a = """\none \\ two"""\n' > "$TMP/padbad.a2ml" +check "census: a backslash followed by spaces and more text on the line is still a bad escape (exit 2)" 2 \ + "$(bun "$CENSUS" check "$TMP/padbad.a2ml" "$TMP/ast.json" >/dev/null 2>&1; echo $?)" +printf 'a = "same"\nb = "same"\nc = "once"\n' > "$TMP/dup.a2ml" +printf '{"items":[["field","a",["string","same"]],["field","c",["string","once"]]]}\n' > "$TMP/dup-ast.json" +check "census: a value in two source fields but one deed field is reported (counted, not just present)" \ + '["same"] 2 1' \ + "$(bun "$CENSUS" check "$TMP/dup.a2ml" "$TMP/dup-ast.json" | jq -r '"\(.missing | tojson) \(.short[0].source) \(.short[0].deed)"' || true)" +printf '{"items":[["field","a",["string","same"]],["field","b",["string","same"]],["field","c",["string","once"]]]}\n' > "$TMP/dup-ast2.json" +check "census: the same value twice in the deed passes" 0 "$(bun "$CENSUS" check "$TMP/dup.a2ml" "$TMP/dup-ast2.json" >/dev/null; echo $?)" +printf 'oops\n' > "$TMP/bad.a2ml" +check "census: an unreadable source is exit 2, not a pass" 2 "$(bun "$CENSUS" check "$TMP/bad.a2ml" "$TMP/ast.json" >/dev/null 2>&1; echo $?)" + +# Empty strings may be dropped, but whitespace is a scalar value. +printf 'a = ""\nb = " "\nc = "\\t"\n' > "$TMP/space.a2ml" +printf '{"items":[]}\n' > "$TMP/space-ast.json" +check "census: missing whitespace values are deficits, empty strings excluded" \ + '2 [" ","\t"]' \ + "$(bun "$CENSUS" check "$TMP/space.a2ml" "$TMP/space-ast.json" | jq -r '"\(.source_values) \(.missing|tojson)"' || true)" +printf '{"items":[["field","b",["string"," "]],["field","c",["string","\\t"]]]}\n' > "$TMP/space-ast.json" +check "census: matching whitespace values pass" 0 "$(bun "$CENSUS" check "$TMP/space.a2ml" "$TMP/space-ast.json" >/dev/null; echo $?)" + +# --- --check (the outcome predicate) ----------------------------------------- +r=$(mkrepo check) +check "--check counts tracked .a2ml" 3 "$(bash "$FIX" --check "$r")" +check "the outcome predicate is declared" 1 "$(grep -cx '# outcome-predicate: --check' "$FIX")" + +# --- a clean conversion, plus refusals for what kyX cannot do ---------------- +r=$(mkrepo ok); STUB_KYX=ok run "$r" +check "partial: ECOSYSTEM converted, STATE and Mustfile refused -> exit 3" 3 "$RC" +check "the deed is staged and the source deletion staged" \ + "A .machine_readable/descriptiles/ECOSYSTEM.deed;D .machine_readable/descriptiles/ECOSYSTEM.a2ml;" "$(staged "$r")" +check "the staged deed is kyX's output, byte for byte" "$(printf '%s\n' "$GOOD_DEED")" "$(cat "$r/.machine_readable/descriptiles/ECOSYSTEM.deed")" +check "nothing is committed" 1 "$(git -C "$r" rev-list --count HEAD)" +check "report: one converted, with provenance" "$KYX_COMMIT" "$(jq -r '.converted[0].kyx_commit' "$REPORT/a2ml-report.json")" +check "report: STATE refused as no front end" STATE "$(jq -r '.refused[] | select(.path|endswith("STATE.a2ml")) | .family' "$REPORT/a2ml-report.json")" +check "report: Mustfile refused as a contractile with no spec" CONTRACTILE "$(jq -r '.refused[] | select(.path|endswith("Mustfile.a2ml")) | .family' "$REPORT/a2ml-report.json")" +check "report: a contractile has no spec (null), not a shifted field" null "$(jq -c '.refused[] | select(.path|endswith("Mustfile.a2ml")) | .spec' "$REPORT/a2ml-report.json")" +check "the issue says a contractile has no mapping spec yet, and links no spec" "1 0" \ + "$(grep -c '^### CONTRACTILE — no mapping spec yet' "$REPORT/a2ml-p1-issue.md") $(grep -c 'mappings/0' "$REPORT/a2ml-p1-issue.md" || true)" +check "a PR body is written" 1 "$(grep -c 'kyX conversion' "$REPORT/a2ml-pr.md")" +check "a P1 issue body is written, fingerprinted" 1 "$(grep -c '^