[field 2026-10-17] Gate mxbuild races the live mxcli run --watch on deployment/ #441

Field report from a field migration project (Next.js + PostgreSQL to Mendix 11.15, 12 modules), Stage 5, 2026-10-16 to 2026-10-17. mxcli v0.27.0. Related: #171, #285.

What we were doing

Stage 5 of the conversion, a parallel build. The lead session replayed numbered MDL scripts on the real model one at a time with bash bin/exec.sh <script>.mdl (snapshot, mxcli exec, then the gate: a full mxbuild --target=deploy, then the lint gate, then one BUILD-LOG row). In the first phases a single agent authored scripts. From about 22:57 local, three phases were authored in parallel on private copies of the app while the lead kept replaying on the real model. Throughout, an app was served with ./mxcli run --local --watch -p App.mpr for the look loop. Other Mendix projects also built on the same machine (14 cores, load average up to 68.75).

Commands in use (exact shape):

./mxcli run --local --watch -p App.mpr
bash bin/exec.sh mdlsource/<NN-script-name>.mdl [--commit-first]
mxbuild --java-home=<jdk> --write-errors=<file> --target=deploy App.mpr   # run by _common.sh as the gate
bash bin/verify-model.sh                                                    # standalone model verify that writes the model stamp

Evidence base: docs/BUILD-LOG.md had 115 data rows at the time of writing (exec ledger, local time +0800). Row counts by text: 57 LINT ROSE, 19 gate could not run, 10 UNVERIFIED, 7 --commit-first: git commit refused, 5 uncommitted model changes, 3 mxcli check failed pre-exec.

Log excerpt (BUILD-LOG rows, verbatim)

| 2026-10-16T19:18:27+0800 | `06-domain-ModuleA.mdl` | fail | ❌ gate could not run | mxbuild exit 1 · change: logic · approval: auto |
| 2026-10-16T19:24:44+0800 | `08-domain-ModuleB.mdl` | fail | ❌ gate could not run | mxbuild exit 1 · change: logic · approval: auto |
| 2026-10-16T19:29:32+0800 | `10-domain-ModuleC.mdl` | fail | ❌ gate could not run | mxbuild exit 1 · change: logic · approval: auto |
| 2026-10-16T19:39:42+0800 | `14-domain-ModuleD.mdl` | fail | ❌ gate could not run | mxbuild exit 1 · change: logic · approval: auto |
| 2026-10-16T19:23:10+0800 | `07-domain-ModuleE.mdl` | unverified | ⚠️ applied, UNVERIFIED | gate unverified (mxbuild exit 3: Compilation of the app bundle failed.) |
| 2026-10-16T20:05:26+0800 | `18-layouts.mdl` | unverified | ⚠️ applied, UNVERIFIED | gate unverified (mxbuild exit 3: An error occurred while compiling Theme files) |

Same scripts passed clean on rerun with no model change (for example 18-layouts failed UNVERIFIED at 20:05:26 and passed at 20:06:36). Watcher side, .mxcli/run.log, same minute:

Directory not empty : '.../deployment/web'
IO_PathNotFound
Compilation of the app bundle failed
An error occurred while compiling Theme files

Gate side: exit 1 right after Synchronize with file system, then a clean verify-model.

Symptom

Verbatim from the exec ledger (BUILD-LOG detail):

❌ gate could not run
mxbuild exit 1

Logs show exit 1 right after Synchronize with file system, then a clean verify-model. The watcher log (.mxcli/run.log) at the same moment shows:

Directory not empty : '.../deployment/web'
IO_PathNotFound
Compilation of the app bundle failed
An error occurred while compiling Theme files

Exit 3 variants have the same cause. No errors file is written, so exec.sh reports "gate could not run" and keeps the write.

Repro

1. In a project run mxcli run --local --watch.
2. Repeatedly run toolkit exec.sh with a tiny valid script.
3. Both mxbuild processes (the watcher's rebuild and the gate's --target=deploy) write the same deployment/ dir. One dies, intermittently.

Expected vs Actual

• Expected: the gate returns pass or fail on the model.
• Actual: no errors file, "gate could not run", the write is kept, and the next exec is blocked by the dirty-model guards.

Frequency

• 17 of 30 failure events in the deep-dive slice were this race.
• 0 of 22 gates failed this way in watcher-free private copies running the same scripts.
• Ledger view: 18 gate-not-run runs (mxbuild exit 1, no errors) and 10 applied-unverified runs (mxbuild exit 3), 28 of 110 exec runs.
• Heavy machine load raises the odds but is not required.

Cost

About 2 to 4 minutes per event (gate, rerun, blocked guards), roughly 45 minutes total, plus 5 probe runs that were only retries. The wider "cause A" total was about 48 minutes of a 349 minute session.

Proposed fix

• exec.sh / _common.sh: detect a live mxcli run --watch on the same mpr (run-state.json, pid) and stop or pause it for the gate, or build the gate into a separate output dir so it never touches the watched deployment/. If mxbuild has no such flag, file an mxcli issue: "gate build should not write the folder run --local --watch serves".
• Retry once (after about 10 s) on mxbuild exit 1 or 3 when problems[] has no CE code. Log both attempts.
• Update skills/ui-loop.md and project-bin/test-stack-up.sh, which currently recommend running the watcher beside exec.

Files

• bin/exec.sh
• bin/_common.sh (mxtk_mxbuild_error_count, mxbuild invocation)
• project-bin/test-stack-up.sh
• skills/ui-loop.md
• skills/iterative-build-loop.md