decompose: cut-ledger (split from #293, 2/6)

CHANGELOG.md

@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
 
 ## Unreleased
 
+- new(cut-ledger): **`bin/cut-ledger.sh` writes the cut ledger of one decompose slice — one row per crossing edge with its resolution (resolved / stub / handoff / retire / platform) and evidence — and exits 1 while any edge has none** (split from [#293](https://github.com/MendixMau/mxcli-project-toolkit/pull/293), 2/6). Reads the source app's catalog `refs` table (or a `--refs` TSV), a slice file (modules and single documents), first-match rules with mandatory evidence, `mx check` CE1613 names the catalog never recorded (check-only rows), the module exporter's JSON (modules it counts with no ledger row are printed), and verbatim `--extra` rows. #293 shipped it with no fixture; new `tests/wave2/test-cut-ledger.sh` (44 checks) runs on a captured line subset of #293's field run (43 of 346 crossing refs, one key per rule) and reproduces that run's own 57 ledger rows column for column, the pre-fix exporter note, UNRESOLVED → exit 1, refusals → exit 2 (now also an unreadable `--rules`/`--catalog`/`--exporter-json`, which escaped as exceptions with the UNRESOLVED code 1, and a `--refs` line with the wrong field count, which was silently dropped), and `--catalog` at single-tree and `app/` paths. Header corrected to the run's real counts (346 refs → 308 rows; it said 199). — MendixMau (#293 field run on a public template app)
 - new(cut-score), fix(app-facts): **`bin/cut-score.sh` scores a proposed split of an existing app into several apps; `app-facts.sh` no longer needs the sqlite3 CLI** (split from [#293](https://github.com/MendixMau/mxcli-project-toolkit/pull/293), 1/6). Given `--cut "A,B|C|D"` (or one app per level of `app-layer-map.json`) it lists every module edge that crosses an app boundary from `analysis/app-facts/dependencies.json`, names the lane each ref kind would need (OData external entity, published action or REST, or none), and sums a clean / costly / blocked verdict per partition with the denominator printed ("7 of 7 own edges"); an aggregate over a foreign entity is read from `.mxcli/catalog.db` (single-tree or `app/`), and reads NOT DETECTED with a reason when it cannot be. Adapted to master: the catalog's `build_mode` label is recorded, not tested (#259 — v0.24+ labels a full catalog `source`), and an empty `activities_data` reads NOT DETECTED instead of "no aggregates". `app-facts.sh` runs its guard queries through Python's sqlite3 when the CLI is missing (absent on some cloud containers and always on Git Bash), stripping the `\r` native Windows Python prints. Rows whose aggregate cannot be detected name the actual reason (no catalog / empty table / unreadable). `tests/wave2/test-cut-score.sh` 70/70 on a captured four-module fixture (14 new: label, empty table, row reasons, own modules derived from edges, both layouts); `test-app-report.sh` 109/109; `test-catalog-edge-guard.sh` 22/22, new case forcing the Python path with CRLF output (fails without the `tr`). — MendixMau (#293 field runs on public template apps)
 - new(sync-project): **`bin/sync-project.sh <project> --only-bin <script.sh> [--only-bin …]` installs only the named `project-bin/` scripts into the project's `bin/` and touches nothing else; `--upgrade-bin` is now documented as what it is — a full sync that may also overwrite a modified copy.** A field run used `--upgrade-bin exec.sh --upgrade-bin _common.sh --upgrade-bin test-scratch.sh` as "install just these scripts" and got a full sync: 244 tracked files changed, one deleted, ~30 added, git hooks reinstalled. `--only-bin` decides everything before writing anything (a name outside the install manifest, NOINSTALL scripts included, a path-like or empty name, a locally modified copy without `--upgrade-bin <name>`/`--force`, or an older copy the full sync's `_common.sh` refresh hold would keep, exits 1 with the project untouched; names are never glob-expanded), backs up an overwritten copy, and prints every file it wrote. The remedy lines in `app-restart.sh`, `test-scratch.sh`, `test-stack-up.sh`, `doctor.sh` and `harvest-learnings.sh` now name `--only-bin`. Fixture `tests/wave2/test-sync-only-bin.sh` (40 checks, single- and two-tree): one `git status` change, hooks and skills untouched, while a full `--upgrade-bin` sync of the same project installs other scripts and the hooks. — field run, issue #390
 - new(bug-lookup): **`bin/bug-lookup.sh` ranks the ledger against a whole pasted error message.** One word (a CE code, a BUG id, a keyword) keeps the full-entry path unchanged; several words print the top 3 entries, each with header, score and one excerpt line, and `--full N` prints the N-th in full. Scoring is one POSIX awk pass: distinct query terms weighted by IDF, CE codes and BUG ids x3, header terms x1.5, an exact BUG id +100. A "no confident match" banner is printed below 40% term-weight coverage or for a CE code no entry mentions. `--exact` keeps the old literal phrase match for a quoted multi-word keyword. Measured on 40 labelled dev queries (the set the weights were tuned on): top-3 9% before (the literal match finds almost nothing in a pasted message) vs 94% now, top-1 89%, confident-wrong 8%. On 18 held-out queries written before the ranker ran on them: top-1 92%, top-3 100%, confident-wrong 17%. All 3 wrong answers are no-match rows without a CE code: the 40% coverage floor caught none of the 4 such rows across both sets, and coverage does not separate them from correct answers. Only the unknown-CE rule makes "no match" work today. The fixture skips, with a note, any no-match row whose CE code has since entered the ledger. About 45 ms per query vs about 950 ms (mawk; about 300 ms on BWK awk); at most 179 words printed. The banner also fires on 3 correct dev answers. Identical output under gawk, mawk and BWK awk 20121220/20180827/20231127; macOS's 20200816 build itself is not tested. All arguments are joined (`"$*"`; it read only `$1` before), so an unquoted multi-word query is ranked too. The banner exits 0 (it still lists the nearest 3); 1 is no match. The script header no longer caches the ledger's size. The routing row and CLAUDE.md mention the pasted message (+9 words on the mdl and gate stubs), and the `mxcli-bugs` row drops its stale "32k words". Fixture `tests/wave2/test-bug-lookup.sh` (floors on both sets; the PASS count depends on the awks installed; 9 checks fail on master). Field evidence: the 2026-10-15 golden-set run. — maintainer request, MendixMau

bin/cut-ledger.sh

@@ -0,0 +1,76 @@
+#!/bin/bash
+# cut-ledger.sh: the cut ledger of one Decompose slice — one row per crossing edge, each with a
+#                resolution and its evidence, or the run fails.
+#
+# usage: bin/cut-ledger.sh (--catalog <catalog.db> | --refs <refs.tsv>) --slice <slice.txt>
+#                          --rules <rules.tsv> [--check-log <mx-check.txt>]...
+#                          [--exporter-json <export.json>]... [--extra <rows.tsv>]...
+#                          [--dump-refs <refs.tsv>] -o <ledger.tsv>
+#
+# WHAT THIS IS. The Stage 5 slice-loop instrument of the decompose-and-rehost plan (step "replace each
+# crossing edge"; the plan is split out of #293 and lands separately). bin/cut-score.sh scores a
+# proposed cut at module level;
+# this lists the edges of the cut you are actually executing, at document level, and refuses to
+# finish while any of them has no decision. "Done" for a slice is not "mx check is green" — a stub
+# turns an edge green and carries none of its behaviour — it is "every edge has a row that says
+# resolved, stub, handoff or retire, with the evidence".
+#
+# WHAT IT READS.
+#   --catalog   the mxcli catalog of the SOURCE app (.mxcli/catalog.db beside the .mpr, written by
+#               `mxcli -p <app>.mpr graph-report`, which project-bin/app-facts.sh runs as the
+#               mandatory Stage 0 facts step), refs table only. Or --refs: a TSV with the header
+#               SourceType SourceName TargetType TargetName RefKind (what --dump-refs writes; the
+#               committed fixture form, since a catalog is tens of MB).
+#   --slice     one name per line: a module (moves whole) or Module.Doc (a document that moves with
+#               the slice although its module stays, e.g. the Synchronization flows of one entity).
+#   --rules     tab-separated: direction (out|in|check|*), pattern (fnmatch on the far-side
+#               document, or 'OwnSide -> FarSide'), resolution, evidence. First match wins.
+#               Resolutions: resolved[:how] · stub · handoff · retire · platform. A row with no
+#               matching rule is UNRESOLVED and the run exits 1. A rule with no evidence is refused.
+#   --check-log mx check output on the slice package (the exporter's self-check, Mendix 9.24 `mx check`
+#               on the package's project.mpr). Every CE1613 name it reports that the catalog did not
+#               record becomes a `check-only` row — the catalog does not see widget actions (App
+#               events), layout placeholders, or images.
+#   --exporter-json  the module-MPK exporter's --json output (export-module-mpk.py, split out of #293
+#               and landing separately; any JSON with crossing_refs_by_module works): each module it counts
+#               that has no out-edge row is printed as `exporter-only` (a false positive or a
+#               catalog blind spot — read which).
+#   --extra     rows no catalog holds, verbatim (direction, kind, source_doc, target_doc, refs,
+#               resolution, evidence): second-order edges of a module brought along to satisfy an
+#               edge, app-level settings the package scrubs, Java hand-off edits.
+#
+# WHAT IT WRITES. <ledger.tsv>: edge, direction, kind, source_doc, target_doc, refs, resolution,
+# evidence. Out = the slice uses something that stays behind; in = something that stays behind uses
+# the slice (the monolith-side work of the strangler). System is auto-resolved as platform.
+#
+# WHAT IT CANNOT SEE. Whether the resolution works: that is the target's mx check and mxbuild, and
+# for a stub it never is — the evidence column says what the stub does not carry. Java and
+# JavaScript bodies naming another module, and references by string (deep links, REST paths).
+#
+# Exit: 0 every row has a resolution · 1 at least one UNRESOLVED row (and only that) · 2 bad arguments,
+#       an unreadable input (missing file, not a catalog, malformed JSON) or a malformed --refs line
+#
+# Field run: public field-service template app 9.24.2 -> its NativeNote slice into Mendix 11.13.0
+# (2026-10-12): 346 crossing refs -> 308 ledger rows, 0 UNRESOLVED, 0.15 s. A captured subset of that
+# run's inputs and ledger is the fixture: tests/wave2/fixtures/cut-ledger/ (test-cut-ledger.sh).
+#
+# LAYOUT. Every input is an explicit path (no .mpr discovery), so single-tree vs two-tree (`app/`)
+# only changes the --catalog argument: <project>/.mxcli/catalog.db or <project>/app/.mxcli/catalog.db.
+#
+# portability-ok: bash 3.2, Python 3 resolved through portable.sh
+
+set -u
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+. "$HERE/lib/portable.sh"
+require_py
+
+case "${1:-}" in
+  -h|--help|"") sed -n '2,9p' "$0" | sed 's/^# \{0,1\}//'; [ -z "${1:-}" ] && exit 2; exit 0 ;;
+esac
+
+"$PY" -I "$HERE/lib/cut-ledger.py" "$@"
+rc=$?
+[ "$rc" -gt 1 ] && exit 2
+exit "$rc"