feat(exec): opt-in --on-copy — exec and gate on a scratch copy, promote by rename (#350)

CHANGELOG.md

@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
 
 ## Unreleased
 
+- new(exec): **`exec.sh --on-copy` (or `EXEC_ON_COPY=1`), opt-in: the script and the mxbuild gate run on a scratch copy, and the live model is written only by a rename-based promotion after a verified pass** ([#350](https://github.com/MendixMau/mxcli-project-toolkit/issues/350)). By default the live model holds an unverified write for the length of the gate (22-65 s per apply in the field), and keeps it when exec.sh is killed. Promotion is refused while Studio Pro holds the model (even under `FORCE_EXEC`), when the live model changed since the copy (cksum manifest), or when the copy is on another filesystem. A swap that fails part-way renames back, else restores the snapshot. On any other outcome the live model is untouched and the copy is kept under `.mpr-snapshots/.on-copy/` with a resume hint (newest 3 kept). Files exec writes beside the model (javasource/, javascriptsource/, theme/...; found by checksum) are promoted with it only if their live copy still matches its pre-copy checksum; a write anywhere else refuses. A single-tree checkout copies only the `.mpr` and the app directories, never sources/, analysis/, docs/ or binaries. A promotion killed half-way leaves `.mxtk-promote.*` names: every exec.sh path (the default one included) and `test-scratch.sh` then refuse until `./bin/exec.sh --recover-promote` finishes a committed promotion or renames the live model back (idempotent; a commit marker decides which; `restore-mpr.sh` only as the last resort). The default path is otherwise unchanged. The copy routine moved from `test-scratch.sh` into `_common.sh` (`mxtk_copy_model_tree`, `mxtk_promote_leftovers`). Fixture `tests/wave2/test-exec-on-copy.sh` (241 checks, both layouts; a kill in each of 8 promotion windows, recovered twice, is the pre-exec or the promoted model, never a mix). Bench `tests/wave2/bench-exec-on-copy.sh` (stub mxbuild 2 s, an input): with exec.sh killed mid-gate the live model stayed byte-identical in 0/10 default runs and 10/10 on-copy runs at 500, 5,000 and 20,000 units. The broken window fell from about 2.1 s (clean) and 5.5-25.4 s (failed gate) to a 15-42 ms swap, at a cost of +1.4 s, +3.7 s and +10.9 s wall time per exec. Field run ([#390](https://github.com/MendixMau/mxcli-project-toolkit/issues/390), macOS arm64, Mendix 11.15, 549 units): promotion with Studio Pro closed and the Studio Pro refusal both PASS; the Ctrl-C recovery half was not run. It found the copy dominating (87-134 s against a ~120 ms swap): the copy now clones where the filesystem can (APFS `cp -c`, Linux reflink), never copies build caches (`node_modules`, `.cache`), and checksums side files in one batch; on the bench (550 units plus a realistic app tree: 15,000-file node_modules, 250 MB of jars and widgets) the on-copy overhead fell from +79.5 s to +1.7 s per exec. A Studio Pro refusal under `--on-copy` / `--recover-promote` now writes a BUILD-LOG row. Statement-level resume from the kept copy is a follow-up. — external review (#350), MendixMau
 - new(tests/behaviour): **a standing behaviour suite for agent-read text: arms M (base ref), T (candidate ref, or the base minus one skill for an ablation) and N (no pack), two drafting tasks with planted STOP-row requests and five retrieval probes, scored mechanically.** `tests/behaviour/build-packs.sh` builds a pack from any ref in a temporary worktree; `score.sh` adds `read_all` (the agent quoted the pack's `PACK-END` line back), `tool_calls` and `tokens`, and treats `mxcli check` as optional (NA without a binary). `RUNBOOK.md` has the triggers (every PR that changes baseline-routed text or an agent stub; N and ablations at each release cut), 5 reps x 2 tasks plus 5 probes x 3 reps, the rule (T no worse than M on every scored metric, probes T >= M - 1), and partial reads counted and re-run, never dropped. `docs/RELEASE-PROCESS.md` §4a names it as the way to run the Behaviour A/B. Field runs committed under `tests/behaviour/results/`: the 2026-10-15 stub-diet A/B (#369/#378/#380, 42 reps, 3 partial reads on the 4,822-line pack) and the first no-pack run (page trap refused 0/5 without a pack against 5/5 with either; mf passes check 1/5 against 5/5; every arm, N included, also inherited the host session's project CLAUDE.md (a generic mxcli/MDL guide), so N measures 'no toolkit rules', not 'no guidance'). Golden input: four real agent outputs, `tests/wave2/test-behaviour-score.sh`. — maintainer request (MendixMau)
 - process(pr-template, release-process): **every Direct-lane PR and every review fills a five-row Impact block: Process, Dependencies, Context cost, Behaviour, Runtime.** "none" or "NOT RUN: <reason>" is legal, a blank row is not. Context cost is pasted from `bin/render-routing.sh --check` on base and head, never estimated; a missing or estimated row is a review finding. The Behaviour A/B (`process/preflight-skill-baseline-2026-08-31.md`) is required for baseline-routed text and agent stubs, recommended for situational skills, NOT RUN with a reason for process docs review agents read, n/a for scripts only; Context cost may be "none" when no routed or always-loaded file is touched. `docs/RELEASE-PROCESS.md` §4a holds the rule and a bad/good example; `CONTRIBUTING.md`'s reviewer list points there. No CI check: `bin/check-pr-discipline.sh` reads the git diff, not the PR body. Field evidence: three PRs reviewed on 2026-10-15 with no process cost stated (#375 added a human step per re-baseline, #384 adds 1–11 s per exec), both found by chance. — maintainer request, MendixMau
 - learn(learned-detection-gaps, testing-shape, ui-loop): **a class on a snippet call never reaches the DOM; a 403 bounce and a widget-pinned session probe both read as sign-outs; app restarts matched a "signs me out" report by timing.** Detection-gaps row: `Class` on a `SNIPPETCALL` is green through check, exec, mxbuild, lint and DESCRIBE, and absent in the browser ([#336](https://github.com/MendixMau/mxcli-project-toolkit/issues/336)). Testing-shape §4 gains two false-red rows: `login.html?messageCode=403` with the session kept ([#338](https://github.com/MendixMau/mxcli-project-toolkit/issues/338)), and instruments pinned to one grid name or a typed menu list (1/7 and 7/34 false FAILs after a redesign, [#339](https://github.com/MendixMau/mxcli-project-toolkit/issues/339)). ui-loop: say so before a restart while a human uses the app ([#337](https://github.com/MendixMau/mxcli-project-toolkit/issues/337)) — three-app field run, 2026-10-14

project-bin/_common.sh

@@ -658,6 +658,175 @@ mxtk_mxbuild_why() {
   "$_py" -c "import json;d=json.load(open('$(native_path "$1")'));print('; '.join((e.get('message','') if isinstance(e,dict) else str(e)) for e in d.get('errors',[]) or [])[:300])" 2>/dev/null || true
 }
 
+# mxtk_copy_model_tree <model-dir> <dest-dir> [<mpr-basename> | --app] — copy the model tree into
+# an existing <dest-dir>, for a scratch copy.
+#   Two args (bin/test-scratch.sh, #278): the whole <model-dir> through tar, leaving out .git,
+#   .mpr-snapshots, deployment, .mxcli (the LIVE app's run --local record — a copy carrying it
+#   points app-restart.sh at the live app) and node_modules.
+#   <mpr-basename> (bin/exec.sh --on-copy on a single-tree checkout, #350): an ALLOW-LIST — the .mpr
+#   with its -wal/-shm, plus the directories in $MXTK_APP_DIRS that exist; never sources/,
+#   analysis/, docs/ or binaries. A directory mxbuild needs and the list lacks fails the gate ON THE
+#   COPY, never the live model; MXTK_APP_DIRS adds it.
+#   --app (exec.sh --on-copy on a two-tree app/): every top-level entry but the two-arg exclusions.
+#   In both on-copy forms, directories named in $MXTK_COPY_SKIP (build caches) are never copied,
+#   and an entry is CLONED where the filesystem can (macOS APFS `cp -c`, Linux reflink), else tarred.
+#   WHY (field run, #390): a 549-unit app took 87-134 s to copy against a ~120 ms swap. Clone vs tar
+#   is probed once per copy on the .mpr itself (read-only); MXTK_COPY_MODE=tar|clone overrides.
+#   Every path preserves mtimes, which --on-copy relies on. Returns non-zero if any entry failed.
+MXTK_APP_DIRS="${MXTK_APP_DIRS:-mprcontents widgets javasource javascriptsource theme themesource userlib vendorlib resources nativemobile mlsource}"
+MXTK_COPY_SKIP="${MXTK_COPY_SKIP:-node_modules .cache .mendix-cache .gradle}"
+mxtk_copy_how() {  # $1 = a file in the source, $2 = dest dir: prints clone-mac | clone-gnu | tar
+  case "${MXTK_COPY_MODE:-}" in
+    tar) echo tar; return ;;
+    clone) [ "$(mxtk_platform)" = macos ] && echo clone-mac || echo clone-gnu; return ;;
+  esac
+  if [ "$(mxtk_platform)" = macos ] && cp -c "$1" "$2/.mxtk-cowprobe" 2>/dev/null; then echo clone-mac
+  elif cp --reflink=always "$1" "$2/.mxtk-cowprobe" 2>/dev/null; then echo clone-gnu
+  else echo tar; fi
+  rm -f "${2:?}/.mxtk-cowprobe"
+}
+mxtk_copy_model_tree() {
+  if [ -z "${3:-}" ]; then
+    tar -C "$1" --exclude=./.git --exclude=./.mpr-snapshots --exclude=./deployment \
+        --exclude=./.mxcli --exclude=./node_modules -cf - . | tar -C "$2" -xf -
+    return
+  fi
+  local _src="$1" _dst="$2" _e _how _x="" _s _skip _rc=0 _list
+  for _s in $MXTK_COPY_SKIP; do _x="$_x --exclude=$_s"; done
+  if [ "$3" = --app ]; then
+    _list="$(ls -A "$_src" | grep -vxE '\.git|\.mpr-snapshots|deployment|\.mxcli|node_modules')"
+    _how="$(mxtk_copy_how "$_src/$(ls "$_src" | grep -m1 '\.mpr$')" "$_dst")"
+  else
+    _list="$(for _e in "$3" "$3-wal" "$3-shm" $MXTK_APP_DIRS; do [ -e "$_src/$_e" ] && echo "$_e"; done)"
+    _how="$(mxtk_copy_how "$_src/$3" "$_dst")"
+  fi
+  while IFS= read -r _e; do
+    [ -n "$_e" ] || continue
+    _skip=""   # an entry holding a build cache is tarred (with the cache excluded), never cloned
+    if [ -d "$_src/$_e" ] && [ "$_how" != tar ]; then
+      for _s in $MXTK_COPY_SKIP; do [ -n "$(find "$_src/$_e" -name "$_s" -prune -print 2>/dev/null | head -1)" ] && _skip=1; done
+    fi
+    case "$_how${_skip:+-skip}" in
+      clone-mac) cp -cRp "$_src/$_e" "$_dst/$_e" ;;
+      clone-gnu) cp -a --reflink=auto "$_src/$_e" "$_dst/$_e" ;;
+      *) false ;;
+    esac || {
+      rm -rf "${_dst:?}/${_e:?}"
+      # shellcheck disable=SC2086
+      tar -C "$_src" $_x -cf - "./$_e" | tar -C "$_dst" -xf - || _rc=1
+    }
+  done <<MXTKLIST
+$_list
+MXTKLIST
+  return "$_rc"
+}
+
+# ── Interrupted exec.sh --on-copy promotions (#350, #384 review) ─────────────────────────────
+# oc_promote (exec.sh) leaves these names beside the model for promotion PID N, in this order:
+#   .mxtk-promote.N.manifest   written FIRST: "L 0|1" (live had mprcontents/), "C 0|1" (the copy
+#                              had), then "E <rel>" / "N <rel>" per side file (existed live / new)
+#   .mxtk-promote.N.mpr / .mprcontents       staged new model (never live)
+#   <rel>.mxtk-promote.N                     a staged side file (never live)
+#   .mxtk-promote.N.old.mpr                  hard link of the live .mpr, taken before the swap
+#   .mxtk-promote.N.old.mprcontents          the live mprcontents/, moved aside by the swap
+#   <rel>.mxtk-promote.N.old                 hard link of a live side file, taken before its swap
+#   .mxtk-promote.N.committed                every rename done and verified; only cleanup is left
+# A kill between the two mprcontents/ renames leaves NO mprcontents/, so every model writer and
+# copier (exec.sh on every path, test-scratch.sh) refuses while any of these exist.
+
+# mxtk_promote_leftovers <model-dir> — prints the PIDs of interrupted promotions; 0 iff any.
+mxtk_promote_leftovers() {
+  local f n pids=""
+  for f in "$1"/.mxtk-promote.*; do
+    [ -e "$f" ] || continue
+    n="${f##*/.mxtk-promote.}"; n="${n%%.*}"
+    case " $pids " in *" $n "*) ;; *) pids="$pids $n" ;; esac
+  done
+  [ -n "$pids" ] || return 1
+  echo $pids
+}
+
+# mxtk_promote_recover <model-dir> <mpr-basename> <pid> [<handback-dir>] — finish or undo one
+# interrupted promotion. Idempotent: every step is guarded by what is on disk, so a second run (or
+# a run after a killed run) is a no-op or finishes the job. Never deletes a live file: anything it
+# takes out of the live tree goes back to <handback-dir> (the kept copy) or into .mxtk-promote.N.trash,
+# and the trash is removed only after every rename succeeded.
+#   .committed present -> the live model IS the promoted one: remove the .old leftovers only.
+#   no .committed      -> roll back to the pre-promotion live model, by rename.
+# Returns 0 done · 1 a rename failed (nothing deleted; re-run) · 2 ambiguous (nothing touched).
+mxtk_promote_recover() {
+  local md="$1" mb="$2" p="$3" hb="${4:-}" t s f k rel lv=1 cv=1 ok=1
+  t="$md/.mxtk-promote.$p"; s="mxtk-promote.$p"
+  _pr_out() {  # $1 = path in the live tree, $2 = its rel path in the copy: hand back, else trash
+    if [ -n "$hb" ] && [ ! -e "$hb/$2" ]; then mkdir -p "$(dirname "$hb/$2")" && mv "$1" "$hb/$2" && return 0; fi
+    mkdir -p "$t.trash/$(dirname "$2")" && mv "$1" "$t.trash/$2.$$"
+  }
+  if [ ! -f "$t.manifest" ]; then
+    for f in "$t".*; do
+      [ -e "$f" ] || continue
+      # .committed alone: a kill between the committed cleanup's two last rms (manifest first) —
+      # the promotion was complete, only its marker is left.
+      case "$f" in "$t.fsprobe"|"$t.trash"|"$t.committed") ;; *)
+        echo "✗ promotion $p: $f exists but its manifest does not — cannot tell what it is. Nothing touched."
+        return 2 ;; esac
+    done
+    rm -rf "${t:?}.fsprobe" "${t:?}.trash" "${t:?}.committed"; return 0
+  fi
+  if [ -f "$t.committed" ]; then
+    rm -rf "${t:?}.old.mprcontents" "${t:?}.old.mpr" "${t:?}.trash" || return 1
+    while read -r k rel; do
+      case "$k" in E|N) rm -f "${md:?}/${rel:?}.${s:?}.old" ;; esac
+    done < "$t.manifest"
+    rm -f "${t:?}.manifest" "${t:?}.fsprobe" && rm -f "${t:?}.committed"
+    echo "  promotion $p had committed: the live model is the promoted one; leftovers removed."
+    return 0
+  fi
+  lv="$(sed -n 's/^L //p' "$t.manifest")"; cv="$(sed -n 's/^C //p' "$t.manifest")"
+  if [ -d "$t.old.mprcontents" ] && [ -d "$md/mprcontents" ] && [ -e "$t.new.mprcontents" ]; then
+    echo "✗ promotion $p: mprcontents/, its .old and its .new all exist — cannot tell which is live. Nothing touched."
+    return 2
+  fi
+  # 1. Side files: put each original back (a hard link to the live file means it was never swapped).
+  while read -r k rel; do
+    case "$k" in E|N) ;; *) continue ;; esac
+    if [ -f "$md/$rel.$s.old" ]; then
+      if [ "$md/$rel.$s.old" -ef "$md/$rel" ]; then mkdir -p "$t.trash" && mv "$md/$rel.$s.old" "$t.trash/side.$$" || ok=0
+      else
+        [ -f "$md/$rel" ] && { _pr_out "$md/$rel" "$rel" || ok=0; }
+        mv -f "$md/$rel.$s.old" "$md/$rel" || ok=0
+      fi
+    elif [ "$k" = N ] && [ ! -f "$md/$rel.$s" ] && [ -f "$md/$rel" ]; then
+      _pr_out "$md/$rel" "$rel" || ok=0   # added by the promotion; did not exist before it
+    fi
+    [ -f "$md/$rel.$s" ] && { _pr_out "$md/$rel.$s" "$rel" || ok=0; }
+  done < "$t.manifest"
+  # 2. The .mpr.
+  if [ -f "$t.old.mpr" ]; then
+    if [ "$t.old.mpr" -ef "$md/$mb" ]; then mkdir -p "$t.trash" && mv "$t.old.mpr" "$t.trash/mpr.$$" || ok=0
+    else
+      if [ -n "$hb" ] && [ ! -e "$hb/$mb" ]; then ln "$md/$mb" "$hb/$mb" 2>/dev/null || cp -p "$md/$mb" "$hb/$mb"; fi
+      mv -f "$t.old.mpr" "$md/$mb" || ok=0
+    fi
+  fi
+  [ -f "$t.mpr" ] && { _pr_out "$t.mpr" "$mb" || ok=0; }
+  # 3. mprcontents/.
+  if [ -d "$t.old.mprcontents" ]; then
+    if [ -d "$md/mprcontents" ]; then mv "$md/mprcontents" "$t.new.mprcontents" || ok=0; fi
+    [ "$ok" = 1 ] && { mv "$t.old.mprcontents" "$md/mprcontents" || ok=0; }
+  elif [ "$lv" = 0 ] && [ "$cv" = 1 ] && [ ! -d "$t.mprcontents" ] && [ -d "$md/mprcontents" ]; then
+    mv "$md/mprcontents" "$t.new.mprcontents" || ok=0   # a v1 live model: the promotion added it
+  fi
+  [ -d "$t.new.mprcontents" ] && { _pr_out "$t.new.mprcontents" mprcontents || ok=0; }
+  [ -d "$t.mprcontents" ] && { _pr_out "$t.mprcontents" mprcontents || ok=0; }
+  if [ "$ok" != 1 ] || { [ "$lv" = 1 ] && [ ! -d "$md/mprcontents" ]; }; then
+    echo "✗ promotion $p: a rename failed — nothing was deleted. Fix the cause and run it again."
+    return 1
+  fi
+  rm -rf "${t:?}.trash" && rm -f "${t:?}.fsprobe" "${t:?}.manifest"
+  echo "  promotion $p had not committed: rolled back by rename; the live model is the one before it."
+  return 0
+}
+
 # project_name — the .mpr basename without extension, for user-facing messages.
 project_name() {
   local mpr