diff --git a/CHANGELOG.md b/CHANGELOG.md index 20342404..5ff201ad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,47 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. ## Unreleased +- learn(rehost): GenAI Showcase rehost target taken from `mx check` 42 x CE1613 to 0 errors and mxbuild BUILD SUCCEEDED on mxcli `nightly-80-gbd45c47e` (`pipelines/mendix-mpr/samples/genai-showcase-11.12.0/nightly/BUILD-FIX.md`, `build-fix.mdl`, replayed from scratch) — the 42 was a floor (phases hide each other: 42 -> 115 -> 0), mxbuild also needs module JAR dependencies, resolved jars and the source's Java sources (not MDL), and the green build still has 11 stubbed microflows, 8 dropped mapping documents, 36 removed icon widgets and about 450 of 482 failed-exec documents absent — MendixMau (GenAI Showcase field run) +- fix(skills): **baseline tier is at its 80,000-word cap on master (79,997), so our branch's three additions to always-on files moved to on-demand skills.** The CRLF-in-string-literals gotcha now lives in `skills/rehost-app.md` (where describe files are handled); the "Service access" section already exists as P2 and the external-entity limits in `skills/mendix-service-seam.md`; the two Decompose/Rehost entry-mode rows in `skills/conversion-runbook.md` are dropped (the routing table covers both skills). `security-is-not-a-later-script.md`, `learned-mdl-preflight.md` and `conversion-runbook.md` are back to master's text. — MendixMau +- learn(rehost): GenAI Showcase re-run on mxcli `nightly-80-gbd45c47e` with the E1–E3 fix — 36/36 modules landed (was 29), 2395 of 2776 documents landed (was 850), failed-exec 482 (was 1993); most of the gain is the toolkit fix, the nightly adds U37/U22/U16; target does not load (one ContentParams association path stored as an attribute name), after that one parameter is replaced `mx check` gives 42 × CE1613 in 5 named classes, BUILD FAILED; journeys NOT RUN (live LLM credentials). Captures `pipelines/mendix-mpr/samples/genai-showcase-11.12.0/nightly/`, README field-run row, 7 new mxcli defects in `contrib/inbox/2026-10-13-genai-nightly-findings.md` — mendix/genai-showcase-app; mendixlabs/mxcli nightly +- fix(rehost): `lib/exec-resume.py` defers the statement that failed, not the document the error names — on a forward reference (`X is defined later in this script`) it used to defer X and resume after it, silently skipping everything in between (E5: 5 GenAI documents lost on the nightly run, e.g. entity `ExampleMicroflows.Ticket`); it now defers the first statement referencing X and resumes from the earliest statement the error names. The deferred retry loop counts progress as fewer deferred statements, not a clean exec of nothing (E6: passes 3–6 re-tried the same 758 statements for ~60 min). Proven on the captured `exec-ExampleMicroflows.round4.txt`; Banking regression `mx check` 0 — mendix/genai-showcase-app; mendixlabs/mxcli nightly +- process(report): `process/mxcli-nightly-retest-2026-10-13.html` — maintainer-facing retest of every mxcli defect from the Decompose/Rehost field runs against `nightly-80-gbd45c47e`: 68 tested, 15 fixed, 8 changed, 39 still broken, 6 not testable, 5 withdrawn; 18 of 19 upstream fixes verified done — mendixlabs/mxcli nightly +- fix(rehost): `pipelines/mendix-mpr` no longer drops a module silently (GenAI FINDINGS E1–E3) — `lib/quarantine-mdl.py` maps `✗ … [CODE]` lint errors by their `at Q.N` line or the name in the message, and `statement N:` errors by number verified against the named document; `lib/mdlsplit.py` closes on an indented `);` before a column-0 keyword, annotation or doc comment, so its count now equals mxcli's on all 36 GenAI modules in executed (topo) order (AmazonBedrockConnector 357 → 360); an unmappable error falls back to `lib/exec-each.py` (one statement at a time, each failure deferred with its verbatim error); one `-- deferred:` line per statement instead of nested ones; timings in Python, not `date +%s.%N`/`bc` (absent on macOS), plus `captures/phases.tsv`. On the run-4 captures all 7 formerly dropped modules now quarantine 1–11 statements instead of 0; Banking regression still 155/155, `mx check` 0, BUILD SUCCEEDED — mendix/genai-showcase-app; mendixlabs/mxcli nightly +- fix(rehost): `pipelines/mendix-mpr` no longer drops a module silently (GenAI FINDINGS E1–E3) — `lib/quarantine-mdl.py` maps `✗ … [CODE]` lint errors by their `at Q.N` line or the name in the message, and `statement N:` errors by number verified against the named document; `lib/mdlsplit.py` closes on an indented `);` before a column-0 keyword, so its count now equals mxcli's on all 36 GenAI modules (AmazonBedrockConnector 357 → 360); an unmappable error falls back to `lib/exec-each.py` (one statement at a time, each failure deferred with its verbatim error); one `-- deferred:` line per statement instead of nested ones; timings in Python, not `date +%s.%N`/`bc` (absent on macOS), plus `captures/phases.tsv`. On the run-4 captures all 7 formerly dropped modules now quarantine 1–11 statements instead of 0; Banking regression still 155/155, `mx check` 0, BUILD SUCCEEDED — mendix/genai-showcase-app; mendixlabs/mxcli nightly +- fix(skills): `decompose-app.md`, `rehost-app.md` — mxcli defects fixed on main after 0.26.0 (`nightly-80-gbd45c47e`; U5, U14, U16, U22, U29, U34, U37 re-verified on template copies) now read as version-conditional workarounds under one `[fixed after 0.26.0]` tag, never deleted; Stage 0 restates why 9.x goes through 10.24 (MPK export only, once the 9.x microflow fix ships; `mxcli run` on 9.x still fails, U44); CHANGED U28 noted; plan gets an ASSUMED nightly line — mendixlabs/mxcli nightly; Siemens FSM template; mendix/genai-showcase-app +- process(report): `process/decompose-and-rehost-report-2026-10-12.html` rewritten as one story — monolith-to-apps idea and Decompose pipeline, Rehost vs MPK hybrid, increments 1–8, speed and skills effect, plan changes, mxcli defects and nightly retest (15 FIXED of 50), tokens (74 workers, 7,883,334); credit: Siemens FSM template field runs. +- learn(decompose): increment 8 Dispatch & Workforce run from the skills — Scheduler, RosterManagement, Technician, Skill, TimeRegistration into an 11.13 app beside a fresh 10.24 monolith copy that publishes `dispatchcore/v1` (OData) and three REST actions: build-green, stub rows 271 → 3, journey read/write/auth passes twice, ≈ 1 h 55 build work vs 4 h 50 for increments 5–7; sample `pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-dispatch/` (target architecture, ledger, generic `react-widgets.py`, `seam-rebind.py`, `bson-rebind-v2.py`), new defects in `contrib/inbox/2026-10-13-increment-8-dispatch-findings.md` — Siemens FSM template +- fix(skills): `rehost-app.md` — done now needs **journey parity on the main flows** (same Playwright journeys on source and rehosted app, same data effects in both databases, idempotent; or `journeys NOT RUN` with a reason), because describe-identical is not behaviour-identical (CE1613 inherited-attribute sort); adds a configuration checklist of what a rehost carries, re-enters or hands off, each hand-off with an owner, and a crisp Rehost-vs-MPK paragraph — mendix/genai-showcase-app, Siemens FSM template +- process(report): `process/rehost-vs-mpk-2026-10-13.html` — one-page answer on how quick module-to-MDL-to-module Rehost is and how it compares with an MPK move, from the 2026-10-12 Banking, GenAI and FSM field runs — Mendix migration field work +- new(skills): `decompose-app.md` Stage 1b **Target architecture (gated)** — landscape, capability → module → app map (N/N), one owning app per crossing entity, contract and auth per seam, core-first cut order, mobile partition, non-goals, put to the user before any cut, MPK export or seam work; reuses `architecture-blueprint.md`; decided 2026-10-13 after seven increments found these late (NativeNote business fit, 120 out-stubs, seam patterns invented mid-build). Credit: Siemens FSM template +- fix(skills): **level-3 strangle recipe from increment 7** — `decompose-app.md` gains Stage 5b Strangle (move entity → same-name external entities → re-create associations → drop modules → `mx check` phases CE1613, CE0161/CE0463, CE2729 → rebind XPath text; describe → exec ruled out for 25 of 38 pages), level 3 = one journey per crossing kind with both databases checked and a second idempotent run (plan criterion amended 2026-10-13), Stage 4 in-edge measure and rebind/pull/rewired/association kinds, trial-licence 402 trap; `mendix-service-seam.md` P1 AutoNumber read grant (capture 25), recipe B′ key copy, CE0161/CE6634 and publish limits, offline push-chain rebind, P13 monolith-side cascades. Credit: Siemens FSM template. +- learn(decompose): increment 7 strangled — AssetManagement, Part, WarehouseManagement and Tool deleted from the FSM 10.24 monolith, which now reads the Assets & Parts slice over OData (AssetsAPIv2, same-name external entities after `move entity`) and writes over REST; in-edge hand-offs 499 → 157 (retire 204, resolved 175), both apps mx check 0 / BUILD SUCCEEDED, ledger 713 rows exit 0, cross-app journey PASS 16/16 twice, increment 6 journey PASS; 8 new mxcli defects to the inbox (`pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-assets-strangled/`). Credit: Siemens FSM template. +- process(report): **`process/decompose-and-rehost-report-2026-10-12.html`** — product-head report on the Decompose and Rehost work: approach, app-to-MDL results, seven plan items rebuilt differently, timings, 3,010,761 measured subagent tokens (Opus 1,907,879 / Sonnet 923,951 / Fable 178,931), proven vs not — Siemens FSM template and GenAI Showcase field runs + +- fix(skills): **`skills/mendix-service-seam.md` P2 regains the auth guidance from the Mendix docs check** — basic auth is the minimal proven route (TLS only), token-based OIDC client credentials for production, Mendix SSO deprecated 1 May 2026, consumer matches the publisher's method. — MendixMau +- new(skills): `mendix-service-seam.md` — pattern/anti-pattern catalogue for OData and REST between two Mendix apps (P1–P13 with verbatim CE codes and capture paths, external-entity limits moved here from `decompose-app.md`, recipes "replace a stub by an external entity" and "cross-app join via a published mirror", completion criteria with denominators); `decompose-app.md` takes increment 6's skill feedback 1–10 (PublicID key, Stage 2 access-rule XPath pre-check, level 2 hand-offs, `rest`/`page-odata`/`local-owner` resolutions, 11.x `HeadersMicroflow`, MPR v2 backup); service-access section in `security-is-not-a-later-script.md`, sentinel guard / two-store secret / password policy in `learned-constants-and-secrets.md`, §6c pointer in `architecture-blueprint.md`; routed `ondemand` — Siemens FSM template +- learn(decompose): increment 6 core services — the FSM 10.24 monolith publishes its core (OData CoreAPI, REST CoreActions) and the Assets & Parts slice consumes it: seam-live for the non-Mobile core, stub rows 120 → 32 (non-Mobile 89 → 1 + 20 reasoned hand-offs, Mobile 31 unchanged), both apps mx check 0 / BUILD SUCCEEDED, ledger 709 rows exit 0, new slice → monolith journey PASS; 8 new mxcli defects in `contrib/inbox/2026-10-12-increment-6-core-services-findings.md` (`pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-assets-core/`) — Siemens FSM template +- fix(rehost): **message definitions are not a Rehost hand-off** — `create message definition collection` exists in mxcli 0.26.0 and an export mapping on one checks and executes on a blank 11.13.0 app (found while reducing the staged upstream issues; U8 withdrawn). The GenAI run failed on ordering: the collection was never emitted before its mappings. Corrected in `skills/rehost-app.md`, the GenAI findings and the plan's done-criterion. — MendixMau +- fix(skills): **`skills/decompose-app.md` Stage 5 checked against current Mendix docs** — adds what re-pointing a consumer at an external entity really costs (local entity must own the association and XPath cannot traverse it; commits become Send External Object; sum/avg/min/max and OQL do not work over OData), that OData can write too and when to choose a REST action anyway, no delete-and-recreate under one key, the system-of-record flip and dual-write window, and token-based service auth over basic auth for production. Source: docs.mendix.com refguide (external entities, published OData entity, consumed OData services). — MendixMau +- process(bugs): **`bug-logs/mxcli-upstream-report-2026-10-12.md`** — draft upstream report for mxcli 0.26.0: 53 distinct defects deduplicated from ~75 raw findings of six inbox drops (24 blocks round-trip, 16 silent drift, 13 usability), each with verbatim error, repro and workaround; NOT filed, awaiting review. Credit: Siemens FSM template, mendix/genai-showcase-app. +- new(skills): **`skills/decompose-app.md` and `skills/rehost-app.md`** — the judgement half of the Decompose and Rehost tracks, written from increments 1–5 (`process/decompose-and-rehost-plan-2026-10-12.md`). Decompose: 9.x → 10.24 on a copy first, cuts on business fit AND coupling (NativeNote was cheapest and a poor slice), cut order by dependency direction (Assets & Parts kept 120 out-stubs because the core stayed unpublished), MPK move, `cut-ledger.sh`, OData/REST seam with a mandatory error handler, three CONFIRMED levels. Rehost: Rehost vs MPK, the check/quarantine/exec/resume loop, ledger verdicts, done = build-green with named hand-offs (the build gate decides, not the ledger). Routed as ondemand rows in `skill-routing.tsv`; runbook Entry Modes gains one row each as variants of Change-an-existing-app and Migration — Siemens FSM template, mendix/genai-showcase-app +- learn(decompose): increment 5 web slice — Assets & Parts (4 FSM modules) by MPK into a `mxcli new` 11.13 app: build-green (mx check 0, BUILD SUCCEEDED, 703-row ledger, 0 unresolved, 120 stubs); OData + REST seam with Production security and one cross-app Playwright journey PASS, seam-live not reached (1 of 536 in-edges rewired); 9 mxcli findings to the inbox (`pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-assets/`) — Siemens FSM template +- process(native): **mxcli 0.26 native-mobile bug hunt — all 12 `Native*` modules of the FSM template round-tripped (1142 documents described); 8 new defects, 6 confirmed**, three with minimal repros (empty `index ( desc)`, native Repeater `content` slot, flows refused on a 9.24.2 target) — `contrib/inbox/2026-10-12-native-mxcli-bughunt.md`, `pipelines/mendix-mpr/samples/fsm-template-9.24.2/native-bughunt/` (Siemens FSM template) +- process(decompose): **plan amended — a Mendix 9 monolith is upgraded to 10.24 on a copy as the first Decompose stage** (user decision). mxcli 0.26 cannot author microflows in 9.x and `mxcli run` cannot run 9.x apps (increment 4 seam probe), so the monolith side of every seam needs 10.x. Credit: Siemens FSM template. — MendixMau +- process(decompose): **Increment 4, the first seam call: one record created in the FSM monolith landed in the NativeNote slice's Mendix 11.13 database over published REST** — `pipelines/mendix-mpr/samples/fsm-template-9.24.2/seam-probe/` (FINDINGS.md, verbatim captures, the MDL that ran). The monolith ran as the 10.24 copy. The 9.24.2 runtime boots headless (Java 11, manual M2EE), but mxcli 0.26 cannot create or alter a microflow in a 9.x project, and `mxcli run` forces JDK 21. 8 mxcli defects in `contrib/inbox/2026-10-12-increment-4-seam-probe-findings.md`. Credit: Siemens FSM template. +- process(rehost): **Increment 2 measured on the GenAI Showcase (Mendix 11.12.0, 36 modules regenerated): 850 of 2776 documents landed (579 identical, 271 drift), 1993 failed-exec, 43 not-emitted; `mx check` 14 × CE1613 after a repair, mxbuild `BUILD FAILED`; 36 min wall time with reused describes.** Most of the loss is the extractor's own: `lib/quarantine-mdl.py` cannot map lint-form check errors (`✗ … [CODE]` + `at Q.N`) or mxcli's `statement N` numbering (mdlsplit counts 357 where mxcli counts 360), and `check_exec` then drops the rest of the module without deferring it. That cost 7 whole modules and 1520 rows, and it is the next extractor fix. mxcli wrote 3 microflows with a null `import from mapping` entity, so the model would not load until they were dropped and re-created. Build blockers that mxcli cannot express: Java action type parameters (`.Entity`), message definitions, rules (`Unknown type: rule`), and an inherited-attribute sort that is describe-identical yet CE1613. Regression parity is not reachable headlessly; plan records build-green-with-hand-offs as the ASSUMED Rehost done-criterion. Captures, ledger and verdict in `pipelines/mendix-mpr/samples/genai-showcase-11.12.0/`; 17 mxcli defects in `contrib/inbox/2026-10-12-rehost-increment-2-genai-mxcli-findings.md`. Credit: mendix/genai-showcase-app. — MendixMau +- process(decompose): **web-only slice candidates for the FSM template** — `pipelines/mendix-mpr/samples/fsm-template-9.24.2/web-slice-candidates.md`: 26 web modules, role x module matrix, six capability candidates scored with `bin/cut-score.sh` on one cut; Insights then Assets & Parts rank cheapest, Service Desk is the heaviest — Siemens FSM template +- process(decompose): **increment 3 findings written up** — `pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-nativenote/FINDINGS.md` (gates verbatim, 308-row cut ledger by resolution, the 26 behaviour stubs, timings, 9 mxcli defects, monolith-side strangler steps, three-level slice done-criterion: build-green / seam-live / strangled); plan marks increment 3 DONE at build-green and records the done-criterion as ASSUMED. Credit: Siemens FSM template. — MendixMau +- new(decompose): **First real slice cut out of the FSM monolith (increment 3): `NativeNote` + its Synchronization_Note flows moved into a fresh Mendix 11.13.0 app by the hybrid route, plus `bin/cut-ledger.sh`, the slice's edge ledger.** 9.24 copy → `mx convert` 10.24 → `export-module-mpk.py` → `marketplace install --file`; 66/66 NativeNote documents describe-identical on the target; crossing edges in MDL (stub entities/flows marked `CUT STUB`, tiny docs copied, 26 Synchronization documents moved by describe → exec); `mx check` 0, mxbuild BUILD SUCCEEDED after Utils' three known Java hand-offs. `bin/cut-ledger.sh` lists every crossing edge at document level from the mxcli catalog (plus mx check names the catalog misses and rows no catalog holds), matches each against a rules file, and exits 1 while any edge has no resolution: field run 346 refs → 308 rows, 0 unresolved (resolved 166, stub 26, handoff 98, retire 15). Fix: `export-module-mpk.py` counted substring matches as crossing edges (`Note×54`, `Technician×2` were `NativeNote.`/`NativeTechnician.`); the match now needs a name boundary, package bytes unchanged. Found: mxcli cannot create a native navigation profile, so a native slice targets the Blank Native Mobile App starter via `mx create-project`, not `mxcli new`; seven more mxcli defects in `contrib/inbox/2026-10-12-increment-3-nativenote-findings.md`. Captures, ledger and executed MDL in `pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-nativenote/`. Credit: Siemens FSM template. — MendixMau +- fix(rehost): **`pipelines/mendix-mpr/rehost-extract.sh` executes a module even when mxcli rejects part of it.** First full run on the GenAI Showcase got 1 of 36 modules into the target: `describe java action` emits the parameter comma after its `--` comment (syntax error on 10 modules, 25 more failed as a cascade), `check` is all-or-nothing per file, and `exec` stops at the first unwritable statement. New `lib/fix-describe-output.py` repairs the comma; `lib/quarantine-mdl.py` moves the statements check rejected (plus their same-module closure) into `.deferred.mdl`; `lib/exec-resume.py` defers the statement exec named and carries on; the extractor loops check → quarantine → exec → resume per module, then retries all deferred statements in fixed-point passes once later modules exist, and the leftovers land in the ledger with the verbatim error. `REHOST_REUSE_MDL=1` skips the 30-minute describe phase; a missing target module no longer kills step 4. Also: grants are their own topo units, a same-name java action orders before its flow, column-0 `$$;` ends a statement. Banking fixture still `Check passed!`; GenAICommons now executes fully with 11 statements deferred. Credit: mendix/genai-showcase-app. — MendixMau +- fix(rehost): **`pipelines/mendix-mpr/rehost-extract.sh` survives a 48-module app (GenAI Showcase, Mendix 11.12.0).** `-c describe module … with all` on mxcli 0.26.0 cuts its output at 10 000 lines without saying so — the extractor now drops the incomplete last statement, says so, and describes every omitted document (nanoflows, snippets, JS actions, JSON structures, mappings, task queues, pages the whole-module emitter skips) one by one from `project-tree --json`. New `lib/mdlsplit.py` splits statements aware of string literals, `$$` bodies and comments (multi-line JSON literals at column 0 had been cut into fragments and reordered); `lib/topo-order-mdl.py` orders every document type, not just flows, and breaks a cycle by forcing one unit instead of abandoning the rest; `lib/compare-mdl.py` reports `NOT re-executable` widgets and mxcli #923 decisions as `not-expressible`. Modules replay in reference order (`REHOST_KEEP_ORDER=1` opts out); per-module timings in `captures/timings.tsv`. Banking fixture still passes. Credit: mendix/genai-showcase-app. — MendixMau +- process(overview): **`process/decompose-and-rehost-overview-2026-10-12.html` — one-page visual summary of the Decompose and Rehost tracks: why, how, stage lists, status cards, the self-improvement loop (inbox → triage → skills/bug-logs/bin, preflight gotchas, changelog credit), the two skills planned after increment 3 (`skills/decompose-app.md`, `skills/rehost-app.md`) and the open decisions.** Companion to the plan; nothing new is built by it. — MendixMau +- fix(decompose): **`bin/cut-score.sh` first run on a real monolith (FSM template 9.24.2, 42 own modules, 312 edges, 4508 refs): the per-partition blocker cell and the stdout summary are capped** — one cut produced 80 blocking rows and the cell carried every why-sentence, 10 KB in one table cell and a 16 KB stdout line. Verbatim `dependencies.json`, `app-layer-map.json` and the findings are in `pipelines/mendix-mpr/samples/fsm-template-9.24.2/`: all 16 ref kinds hit the lane table, but `call` conflates server-side and client-side (nanoflow, JavaScript action) calls — ~30 % of the blocker mass on the web|mobile seam is mis-laned — and the `sync` ref kind (offline profile → entity) never reaches `dependencies.json`, which hides a hard blocker. Best 3-app landscape found: Planner (25) | Mobile + Synchronization + Utils (14) | retire 3 test modules. Fixture test 56/56. Credit: Siemens FSM template. — MendixMau +- fix(app-facts): **`project-bin/app-facts.sh` no longer hard-requires the `sqlite3` CLI.** Its seven one-column guard queries run through the CLI when present and through Python's `sqlite3` module otherwise; the facts themselves always came from Python. Field run 2026-10-12 in a cloud container where apt has no `sqlite3` package (FSM template). — MendixMau +- new(decompose): **`pipelines/mendix-mpr/export-module-mpk.py` — headless "export module to .mpk" for an MPR v1 app, and the MPK route tested end to end without Studio Pro.** Writes Mendix's module-package format (studied on Marketplace Encryption 9.0.3/11.1.2 and on `mx create-module-package`): the source schema and `_MetaData` verbatim, the module's unit subtree byte for byte, the project root/conversion units and the four project documents `mx module-import` requires (a first version without them failed with `Sequence contains no elements`, exit 117), with demo users/passwords, constants, credentials, certificates and navigation scrubbed; plus `javasource/javascriptsource/themesource/resources/`, RequiredLib-marked jars and referenced widget `.mpk`s. Refuses MPR v2 and non-reference schemas. Field run 2026-10-12 on the FSM template (Mendix 9.24.2): FeatureToggle, Utils and Visualizations installed with `mxcli marketplace install --file` into 10.24 and 11.13 targets; every residual `mx check` error (2/7/134) a crossing edge, none version or format; Utils with its edges satisfied reached `mx check` 0 and mxbuild BUILD SUCCEEDED on 11.13 after three Java hand-off edits (Mendix 9 runtime API in user code, orphan action files, a jar marker naming a former module). Findings that change the Decompose plan: Mendix 10/11 `mx` already has `create-module-package` (it refuses errorful modules and regenerates every unit and entity ID; this exporter and mxcli's install preserve them, measured 9.24 → 11.13); Mendix's import window forces a 9.x → 11.x move through `mx convert` to 10.24; 9.24 `mx` has no `module-import`. Verbatim captures in `pipelines/mendix-mpr/samples/mpk-export-probe/captures/`. Credit: Siemens FSM template (Marketplace) as the field fixture. — MendixMau +- new(rehost): **`pipelines/mendix-mpr/` — the Rehost extractor (increment 1a of `process/decompose-and-rehost-plan-2026-10-12.md`): read an existing `.mpr`, emit each own module as MDL, replay it into a fresh target of the same Mendix version, write a fidelity ledger.** `rehost-extract.sh` uses the MDL statement `describe module with all` (the `describe module` subcommand emits only module + roles on mxcli v0.26.0), reorders flows into call order with `lib/topo-order-mdl.py` because describe output is in catalog order and mxcli's own `check` refuses it (MDL-ORDER01), appends the image collections `with all` omits, and keeps CRLF inside string literals byte-for-byte (a normalised annotation makes `exec` refuse the splice, MDL-V1-REBUILD). Ledger: one row per catalog document, denominator in the header; demo-user passwords (`'***'`) and project security level are hand-off rows. Field run 2026-10-12 on ako/mxcli-banking (Mendix 11.13.0): 155/155 documents, 154 identical on target describe, 1 doc-comment drift, `mx check` 0 errors, mxbuild green, `diff --format struct` empty, 35 s. Same day the Stage 0 read probe passed on the Marketplace FSM template (Mendix 9.24.2, MPR v1, 64 MB, 45 own modules): verbatim captures in `samples/fsm-template-9.24.2/`. mxcli findings and MDL learnings from both increments in `contrib/inbox/2026-10-12-decompose-rehost-increment-1-mxcli-findings.md`; the CRLF rule is promoted into `skills/learned-mdl-preflight.md`. Credit: ako/mxcli-banking (Apache-2.0) as the round-trip fixture; Siemens FSM template as the Mendix 9 probe. — MendixMau +- new(decompose): **`bin/cut-score.sh` — Stage 3 cut scoring for the Decompose track (increment 1b of `process/decompose-and-rehost-plan-2026-10-12.md`).** Takes `analysis/app-facts/dependencies.json` and a `--cut "Core,Util|Orders|Billing|Reporting"` (default: one app per `app-layer-map.json` level), lists every module edge that would cross an app boundary with the lane its ref kind must become — OData external entity (acceptable), published OData action / REST (costly), none (blocker: bidirectional call pairs, generalize/workflow/role crossings); Business Events is excluded by licence. Aggregates over a foreign entity are read from the mxcli catalog (`activities_data`: a RetrieveAction on a foreign entity plus an AggregateListAction in the same microflow) and reported NOT DETECTED with the reason when there is no catalog, never blank. Per partition: totals, blockers, verdict clean/costly/blocked; the denominator ("7 of 7 own edges, 12 of 12 refs") is always printed, unplaced modules land in `_unassigned` as a named gap. Writes `analysis/cut-score.html` + `.json`. Field run 2026-10-12 on a Mendix 10.24.0 fixture built by MDL (mxcli v0.26.0, Linux, no Studio Pro) with a planted Orders<->Billing tangle and a Reporting->Billing COUNT/SUM: both found. Fixture `tests/wave2/fixtures/cut-score/` (verbatim dependencies.json, layer map, reduced catalog) and `tests/wave2/test-cut-score.sh` (56 checks). Not yet wired into a skill or the routing table — that is the plan's next increment. — MendixMau +- process(plan): **`process/decompose-and-rehost-plan-2026-10-12.md` — two new pipelines proposed: Decompose (split a Mendix monolith by scored cut edges, strangler per slice) and Rehost (read a Mendix 9/10 `.mpr` with `describe module`, regenerate as MDL into Mendix 11, with a fidelity ledger).** Research-backed: Mendix multi-app constraints, mxcli v0.26.0 capabilities probed in a cloud container, candidate public targets (FSM template 9.24.2, GenAI Showcase 48 modules). Nothing built yet; increments and open questions listed. — MendixMau - learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run : N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau - learn(learned-detection-gaps, module-review): **two UI defects every model gate passes: a lazy-loaded combo box filter hides its preset, and a theme that pins row heights by position lets the next row cover an inserted widget** ([#299](https://github.com/MendixMau/mxcli-project-toolkit/issues/299)). Both passed check, mxbuild, lint and the scripted checks on a 127-module production app and were found only in the running app. Two register rows with the first oracle and the pre-build check (pluggable properties that change *when* data loads; grep the theme for fixed `height` on positional row classes before an insert), and two LOOK lines in `module-review.md` §4c with denominators (`N of N filters show their preset on first load`, `N of N widgets inserted into an existing container fully visible`). — jopterhorst - fix(app-restart, test-stack-up): **a port held by another process before `run --local` is named and reported as "instrument blocked" (exit 4), never killed** ([#262](https://github.com/MendixMau/mxcli-project-toolkit/issues/262)). Studio Pro had started its own runtime on the app/admin ports a restart needed; `mxcli run --local` then failed with "Authentication failed" (the admin API answering was Studio Pro's), the next journey ran against Studio Pro's runtime, and four more runs were NOT RUN. `app-restart.sh` now checks the app port and the admin port (record's `adminPort`, else app + 10) before stopping anything — a holder outside the recorded process tree exits 4 with its name, pid and parent — and again after the stop, where anything still holding a port is by definition not ours. `test-stack-up.sh` runs the same check wherever it would tell you to start `run --local` (`--no-docker`, no Docker daemon, `--check` with a leftover record), on the record's ports, `RUN_LOCAL_APP_PORT`/`RUN_LOCAL_ADMIN_PORT`, or mxcli's defaults 8080/8090. Holder lookup: lsof, then ss, then netstat + tasklist on Windows; unnamed holders still block when nothing of ours is running. Fixtures: `test-app-restart.sh` +2 cases (14 assertions, all fail on the old script), `test-stack-up-local.sh` T7 (4 of 6 fail on the old script). — AlvinnYeo diff --git a/README.md b/README.md index 50f41d55..ed4eabb5 100644 --- a/README.md +++ b/README.md @@ -523,6 +523,7 @@ Every mxcli project has a `.ai-context/skills/` directory (bundled by `mxcli ini | Generating a new project's CLAUDE.md — baseline routing plus project-specific facts | `skills/bootstrap-project.md` | | Setting up or resuming an mxcli project in a cloud/ephemeral container — the one-time setup order (mxcli download → mxcli init → init-project.sh → sources decision → push) and the commit-and-push loop that survives container reclaim | `skills/cloud-dev-environment.md` | | Changing an EXISTING Mendix app — adding a feature, altering a flow, restructuring a module — when it has no BRDs, no architecture doc and no wireframes: the knowledge base comes from the live model (Path D), stages 2–4 run over what changes plus what it touches, nothing more, and the Track B regression baseline is the precondition; audit-only stays in existing-app-assurance | `skills/existing-app-change.md` | +| Splitting an existing Mendix monolith into several apps — strangler, "move module X into its own app", carving out a slice, microservices from one .mpr: cut choice on business fit AND coupling (bin/cut-score.sh), cut order by dependency direction, MPK move, every crossing edge on bin/cut-ledger.sh, OData/REST seam, and the build-green / seam-live / strangled level each slice reached | `skills/decompose-app.md` | | Setting up or wiring a COMPANY BRAIN — the private tier between the toolkit and a project for own skills, conventions, lint rules, MDL snippets and approved MPKs; and deciding whether something goes to the toolkit, the company brain or docs/brain/ | `skills/company-brain.md` | | Cutover and retrospective — promoting proven patterns back into the toolkit | `skills/close-the-loop.md` | | Before citing ANY behavioural claim about the harness, the Mendix runtime or a test tool as evidence — a claim not in the register may not be cited | `skills/measured-claims.md` | @@ -537,6 +538,7 @@ Every mxcli project has a `.ai-context/skills/` directory (bundled by `mxcli ini | Reviewing what the extraction actually produced — the Stage 1 surface, and the file the Stage 1 gate looks for. Renders a code-extracted and a document knowledge base alike, so a requirements-driven project gets the surface too; prints no zero that a second record does not agree with | `bin/extraction-report.sh` | | Assessing or planning a migration up front, before any pipeline is chosen | `skills/assess-migration.md` | | Running the extraction pipeline | `skills/migration-pipeline.md` | +| Rehosting or regenerating an existing Mendix .mpr as MDL into a fresh app — "port this app to a clean Mendix 11 project", re-platform a whole app, rebuild it from its own model: Rehost vs MPK move, pipelines/mendix-mpr/rehost-extract.sh and its check/quarantine/exec/resume loop, reading the fidelity ledger, done = build-green with named hand-offs | `skills/rehost-app.md` | | Migrating from a stack that has no dedicated pipeline | `skills/migrate-general.md` | | Migrating an OutSystems app | `skills/migrate-outsystems.md` | | Understanding OutSystems 11 source | `skills/source-os11.md` | @@ -635,6 +637,7 @@ Every mxcli project has a `.ai-context/skills/` directory (bundled by `mxcli ini | Task | Skill to load | |---|---| +| Mendix-to-Mendix service seam — one Mendix app's OData/REST service consumed by another Mendix app: consuming its OData as external entities, replacing a stub by an external entity, joining data that lives in the other Mendix app (mirror keyed by the consumer's key), or wiring the seam between two apps after a split; keys (PublicID AutoNumber), machine-role-only access (200/401/404), REST actions with error handlers, external-entity limits | `skills/mendix-service-seam.md` | | Building a REST integration (consumed or published) for the first time on a project — the checks that avoid a rebuild after the first live call | `skills/rest-integration-first-time-right.md` | | Adding a constant, installing a marketplace module, deploying to a new environment, or needing a Mendix PAT — where an encryption key, API credential, endpoint or access token gets its value, why a free node has only one channel, and why a token missing from the files you grepped is not missing | `skills/learned-constants-and-secrets.md` | diff --git a/ROUTING.md b/ROUTING.md index 2c55707c..d4927034 100644 --- a/ROUTING.md +++ b/ROUTING.md @@ -53,6 +53,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Generating a new project's CLAUDE.md — baseline routing plus project-specific facts | `skills/bootstrap-project.md` | ba | P | ondemand | | Setting up or resuming an mxcli project in a cloud/ephemeral container — the one-time setup order (mxcli download → mxcli init → init-project.sh → sources decision → push) and the commit-and-push loop that survives container reclaim | `skills/cloud-dev-environment.md` | all | P | ondemand | | Changing an EXISTING Mendix app — adding a feature, altering a flow, restructuring a module — when it has no BRDs, no architecture doc and no wireframes: the knowledge base comes from the live model (Path D), stages 2–4 run over what changes plus what it touches, nothing more, and the Track B regression baseline is the precondition; audit-only stays in existing-app-assurance | `skills/existing-app-change.md` | ba,architect | P,0 | ondemand | +| Splitting an existing Mendix monolith into several apps — strangler, "move module X into its own app", carving out a slice, microservices from one .mpr: cut choice on business fit AND coupling (bin/cut-score.sh), cut order by dependency direction, MPK move, every crossing edge on bin/cut-ledger.sh, OData/REST seam, and the build-green / seam-live / strangled level each slice reached | `skills/decompose-app.md` | lead,architect,mdl | 0,3,5,6 | ondemand | | Setting up or wiring a COMPANY BRAIN — the private tier between the toolkit and a project for own skills, conventions, lint rules, MDL snippets and approved MPKs; and deciding whether something goes to the toolkit, the company brain or docs/brain/ | `skills/company-brain.md` | all | - | ondemand | | Cutover and retrospective — promoting proven patterns back into the toolkit | `skills/close-the-loop.md` | all | 7 | ondemand | | Before citing ANY behavioural claim about the harness, the Mendix runtime or a test tool as evidence — a claim not in the register may not be cited | `skills/measured-claims.md` | all | - | ondemand | @@ -74,6 +75,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Reviewing what the extraction actually produced — the Stage 1 surface, and the file the Stage 1 gate looks for. Renders a code-extracted and a document knowledge base alike, so a requirements-driven project gets the surface too; prints no zero that a second record does not agree with | `bin/extraction-report.sh` | ba | 1 | ondemand | | Assessing or planning a migration up front, before any pipeline is chosen | `skills/assess-migration.md` | ba | 0 | ondemand | | Running the extraction pipeline | `skills/migration-pipeline.md` | ba | 0,1 | ondemand | +| Rehosting or regenerating an existing Mendix .mpr as MDL into a fresh app — "port this app to a clean Mendix 11 project", re-platform a whole app, rebuild it from its own model: Rehost vs MPK move, pipelines/mendix-mpr/rehost-extract.sh and its check/quarantine/exec/resume loop, reading the fidelity ledger, done = build-green with named hand-offs | `skills/rehost-app.md` | lead,ba,mdl | 0,1,5,6 | ondemand | | Migrating from a stack that has no dedicated pipeline | `skills/migrate-general.md` | ba | 0 | ondemand | | Migrating an OutSystems app | `skills/migrate-outsystems.md` | ba | 0 | ondemand | | Understanding OutSystems 11 source | `skills/source-os11.md` | ba | 0,1 | ondemand | @@ -189,6 +191,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Always relevant for | Load this | Agent(s) | Stage(s) | Tier | |---|---|---|---|---| +| Mendix-to-Mendix service seam — one Mendix app's OData/REST service consumed by another Mendix app: consuming its OData as external entities, replacing a stub by an external entity, joining data that lives in the other Mendix app (mirror keyed by the consumer's key), or wiring the seam between two apps after a split; keys (PublicID AutoNumber), machine-role-only access (200/401/404), REST actions with error handlers, external-entity limits | `skills/mendix-service-seam.md` | lead,architect,mdl | 3,5 | ondemand | | Building a REST integration (consumed or published) for the first time on a project — the checks that avoid a rebuild after the first live call | `skills/rest-integration-first-time-right.md` | mdl | 4,5 | ondemand | | Adding a constant, installing a marketplace module, deploying to a new environment, or needing a Mendix PAT — where an encryption key, API credential, endpoint or access token gets its value, why a free node has only one channel, and why a token missing from the files you grepped is not missing | `skills/learned-constants-and-secrets.md` | mdl,architect,gate | 4,5,6 | ondemand | diff --git a/agents/architect-agent.md b/agents/architect-agent.md index 5e6927a3..2ee12510 100644 --- a/agents/architect-agent.md +++ b/agents/architect-agent.md @@ -57,6 +57,8 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `project-bin/check-prototype-links.js` | Before wireframes pass to the build loop, and with --brd before a BRD is signed off: dead #/route links, orphan screens, controls with no data-bind and no data-cut, BRD routes no screen has, screens no use case walks (design-artifacts.md Step 3c, brd-validation.md check 8) | | `skills/cloud-dev-environment.md` | Setting up or resuming an mxcli project in a cloud/ephemeral container | | `skills/existing-app-change.md` | Changing an EXISTING Mendix app | +| `skills/decompose-app.md` | Splitting an existing Mendix monolith into several apps | +| `skills/mendix-service-seam.md` | Mendix-to-Mendix service seam | | `project-bin/app-facts.sh` | Collecting the facts an app dossier is written from: forces a full catalog build, module edges through the real module column, strongly connected components, describes every loop-containing microflow and parses the loop bodies; exit 2 on a stale, fast-mode or schema-incomplete catalog, never a verdict | | `skills/app-analysis.md` | Building or refreshing the standing dossier of an EXISTING app before changing it: inventory, module dependency shape, loop risk patterns, each section with a verdict and a fault where nothing was measured (a standing document; nothing else in the toolkit reads it yet) | | `skills/layering-review.md` | Checking whether an EXISTING app's module boundaries still hold: the layer map (bin/app-layer-map.sh) computes the order the modules would stack in and draws only the edges that point back up, so one tangle of N mutually reachable modules becomes a named list of edges with a weight and a ref kind; Stage 3 in existing-app-change mode, and the honest blast radius for a slice | diff --git a/agents/ba-agent.md b/agents/ba-agent.md index 92ef81ee..32edb3e5 100644 --- a/agents/ba-agent.md +++ b/agents/ba-agent.md @@ -74,6 +74,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `skills/layering-review.md` | Checking whether an EXISTING app's module boundaries still hold: the layer map (bin/app-layer-map.sh) computes the order the modules would stack in and draws only the edges that point back up, so one tangle of N mutually reachable modules becomes a named list of edges with a weight and a ref kind; Stage 3 in existing-app-change mode, and the honest blast radius for a slice | | `skills/assess-migration.md` | Assessing or planning a migration up front, before any pipeline is chosen | | `skills/migration-pipeline.md` | Running the extraction pipeline | +| `skills/rehost-app.md` | Rehosting or regenerating an existing Mendix .mpr as MDL into a fresh app | | `skills/migrate-general.md` | Migrating from a stack that has no dedicated pipeline | | `skills/migrate-outsystems.md` | Migrating an OutSystems app | | `skills/source-os11.md` | Understanding OutSystems 11 source | diff --git a/agents/mdl-agent.md b/agents/mdl-agent.md index a4b47bc4..07ee7b4e 100644 --- a/agents/mdl-agent.md +++ b/agents/mdl-agent.md @@ -70,7 +70,10 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `project-bin/context-pack.sh` | Dispatching a build step to a helper | | `bug-logs/mxcli-bugs.md` | Reading a whole class of tool defects (a retest, a new mxcli release, an audit) | | `skills/cloud-dev-environment.md` | Setting up or resuming an mxcli project in a cloud/ephemeral container | +| `skills/decompose-app.md` | Splitting an existing Mendix monolith into several apps | +| `skills/mendix-service-seam.md` | Mendix-to-Mendix service seam | | `skills/microflow-loop-antipatterns.md` | Reading what loop bodies do (LOOP_TQ, deferred commit, nested loop, REST in loop, transaction control per item, scheduled-event reachability) from described MDL; the catalog holds top-level activities only and cannot see inside a loop | +| `skills/rehost-app.md` | Rehosting or regenerating an existing Mendix .mpr as MDL into a fresh app | | `skills/iterative-build-loop.md` | Building a module with mxcli | | `skills/mdl-cookbook-microflows.md` | Writing MDL microflow scripts | | `skills/build/mdl/oneshot-mdl-method.md` | Writing a single MDL script that takes a project from nothing to a working vertical slice | diff --git a/bin/cut-ledger.sh b/bin/cut-ledger.sh new file mode 100755 index 00000000..dfe5083d --- /dev/null +++ b/bin/cut-ledger.sh @@ -0,0 +1,69 @@ +#!/bin/bash +# cut-ledger.sh: the cut ledger of one Decompose slice — one row per crossing edge, each with a +# resolution and its evidence, or the run fails. +# +# usage: bin/cut-ledger.sh (--catalog | --refs ) --slice +# --rules [--check-log ]... +# [--exporter-json ]... [--extra ]... +# [--dump-refs ] -o +# +# WHAT THIS IS. The Stage 5 slice-loop instrument of process/decompose-and-rehost-plan-2026-10-12.md +# (§3, step 2: "replace each crossing edge"). bin/cut-score.sh scores a proposed cut at module level; +# this lists the edges of the cut you are actually executing, at document level, and refuses to +# finish while any of them has no decision. "Done" for a slice is not "mx check is green" — a stub +# turns an edge green and carries none of its behaviour — it is "every edge has a row that says +# resolved, stub, handoff or retire, with the evidence". +# +# WHAT IT READS. +# --catalog the mxcli catalog of the SOURCE app (.mxcli/catalog.db beside the .mpr, written by +# `mxcli -p .mpr graph-report`, which project-bin/app-facts.sh runs as the +# mandatory Stage 0 facts step), refs table only. Or --refs: a TSV with the header +# SourceType SourceName TargetType TargetName RefKind (what --dump-refs writes; the +# committed fixture form, since a catalog is tens of MB). +# --slice one name per line: a module (moves whole) or Module.Doc (a document that moves with +# the slice although its module stays, e.g. the Synchronization flows of one entity). +# --rules tab-separated: direction (out|in|check|*), pattern (fnmatch on the far-side +# document, or 'OwnSide -> FarSide'), resolution, evidence. First match wins. +# Resolutions: resolved[:how] · stub · handoff · retire · platform. A row with no +# matching rule is UNRESOLVED and the run exits 1. A rule with no evidence is refused. +# --check-log mx check output on the slice package (the exporter's self-check, Mendix 9.24 `mx check` +# on the package's project.mpr). Every CE1613 name it reports that the catalog did not +# record becomes a `check-only` row — the catalog does not see widget actions (App +# events), layout placeholders, or images. +# --exporter-json pipelines/mendix-mpr/export-module-mpk.py --json output: each module it counts +# that has no out-edge row is printed as `exporter-only` (a false positive or a +# catalog blind spot — read which). +# --extra rows no catalog holds, verbatim (direction, kind, source_doc, target_doc, refs, +# resolution, evidence): second-order edges of a module brought along to satisfy an +# edge, app-level settings the package scrubs, Java hand-off edits. +# +# WHAT IT WRITES. : edge, direction, kind, source_doc, target_doc, refs, resolution, +# evidence. Out = the slice uses something that stays behind; in = something that stays behind uses +# the slice (the monolith-side work of the strangler). System is auto-resolved as platform. +# +# WHAT IT CANNOT SEE. Whether the resolution works: that is the target's mx check and mxbuild, and +# for a stub it never is — the evidence column says what the stub does not carry. Java and +# JavaScript bodies naming another module, and references by string (deep links, REST paths). +# +# Exit: 0 every row has a resolution · 1 at least one UNRESOLVED row · 2 bad arguments +# +# Field run: FSM template 9.24.2 -> NativeNote slice into Mendix 11.13.0 (increment 3, 2026-10-12): +# 199 crossing refs -> ledger in pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-nativenote/. +# +# portability-ok: bash 3.2, Python 3 resolved through portable.sh + +set -u + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck disable=SC1091 +. "$HERE/lib/portable.sh" +require_py + +case "${1:-}" in + -h|--help|"") sed -n '2,9p' "$0" | sed 's/^# \{0,1\}//'; [ -z "${1:-}" ] && exit 2; exit 0 ;; +esac + +"$PY" -I "$HERE/lib/cut-ledger.py" "$@" +rc=$? +[ "$rc" -gt 1 ] && exit 2 +exit "$rc" diff --git a/bin/cut-score.sh b/bin/cut-score.sh new file mode 100755 index 00000000..e8ce4545 --- /dev/null +++ b/bin/cut-score.sh @@ -0,0 +1,576 @@ +#!/bin/bash +# cut-score.sh: score a proposed decomposition of an existing app into several apps: +# analysis/cut-score.html + analysis/cut-score.json +# +# WHAT THIS IS. The Stage 3 instrument of process/decompose-and-rehost-plan-2026-10-12.md. Given a +# CUT — a partition of the own modules into would-be apps — it lists every module-level edge that +# would cross an app boundary, names the integration LANE each ref kind would have to become +# (an OData external entity, a published OData action or REST call, or nothing at all because the +# platform has no lane for it), and sums that per partition into a verdict: clean / costly / +# blocked. Business Events is not a lane here: the customer this was written for holds no Event +# Broker licence, so a crossing can only become OData or REST, or it blocks. +# +# WHAT IT READS. analysis/app-facts/dependencies.json, written by project-bin/app-facts.sh (the +# mandatory Stage 0 facts step) — the edges array with its per-ref-kind counts, and +# scope.own_modules. Optionally .mxcli/catalog.db, which app-facts.sh leaves behind (graph-report +# builds it in full mode): that is the only place an AGGREGATE over a foreign entity can be seen, +# because dependencies.json records that a microflow retrieves Billing.Invoice but not what it +# then does with the list. With no catalog the aggregate column reads NOT DETECTED and says why; +# it is never silently blank. The default cut (one app per layer-map level) reads +# analysis/app-layer-map.json, written by bin/app-layer-map.sh; without it, --cut is required. +# +# WHAT THIS IS NOT. Not a judge. The lane table below is platform fact (an external entity cannot +# own an association, cannot be aggregated server-side, cannot be the subject of a workflow; a +# synchronous call across apps is a published action or a REST operation and the transaction no +# longer spans), and the verdict is arithmetic over that table. Whether a costly partition is +# worth it is skills/layering-review.md and the Stage 3 interview, not this file. +# +# WHAT IT CANNOT SEE, and prints on the page: Java and JavaScript actions, published REST called +# by name, workflow references, page URLs (same blind spots as app-layer-map.sh); an aggregate +# over a list that came back from a cross-partition CALL (the catalog has no variable linkage, so +# the aggregate is attributed to the retrieve in the same microflow, never to the call); the +# volume of data an OData hop would move at runtime. +# +# Usage: +# bin/cut-score.sh [project-dir] --cut "Core,Util|Orders|Billing|Reporting" +# bin/cut-score.sh [project-dir] # default cut: one app per layer-map level +# bin/cut-score.sh --facts DIR --catalog DB --cut ... -o OUT +# bin/cut-score.sh ... --json # the same numbers to stdout, no HTML +# +# A module in scope.own_modules that --cut does not name lands in a partition called "_unassigned" +# and is reported as a gap; a name in --cut that is not an own module is reported as a gap too. +# Neither is ever inferred into a neighbouring partition. +# +# Exit: 0 rendered · 2 no dependencies.json, no cut (no --cut and no app-layer-map.json), or +# malformed --cut +# +# Field run: lab checkout lab/decompose (CutFixture, Mendix 10.24.0, mxcli v0.26.0, 2026-10-12) — +# 7 own edges, 6 crossing on the four-way cut; found the Orders<->Billing call tangle as the +# blocker and the Reporting->Billing retrieve+aggregate from the catalog. +# +# portability-ok: bash 3.2, Python 3 resolved through portable.sh + +set -u + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck disable=SC1091 +. "$HERE/lib/portable.sh" +require_py + +PROJECT=""; FACTS=""; OUT=""; JSON=0; CUT=""; CATALOG=""; LAYERMAP="" +while [ $# -gt 0 ]; do + case "$1" in + --facts) FACTS="$2"; shift 2 ;; + --catalog) CATALOG="$2"; shift 2 ;; + --layer-map) LAYERMAP="$2"; shift 2 ;; + --cut) CUT="$2"; shift 2 ;; + -o|--out) OUT="$2"; shift 2 ;; + --json) JSON=1; shift ;; + -h|--help) sed -n '2,50p' "$0"; exit 0 ;; + -*) echo "cut-score: unknown option $1" >&2; exit 2 ;; + *) PROJECT="$1"; shift ;; + esac +done +[ -n "$PROJECT" ] || PROJECT="." +PROJECT="$(cd "$PROJECT" 2>/dev/null && pwd)" || { echo "cut-score: no such directory" >&2; exit 2; } +[ -n "$FACTS" ] || FACTS="$PROJECT/analysis/app-facts" +[ -n "$OUT" ] || OUT="$PROJECT/analysis/cut-score.html" +[ -n "$LAYERMAP" ] || LAYERMAP="$PROJECT/analysis/app-layer-map.json" +if [ -z "$CATALOG" ]; then + # The catalog sits beside the .mpr: single-tree (project root) or two-tree (app/). + for c in "$PROJECT/.mxcli/catalog.db" "$PROJECT/app/.mxcli/catalog.db"; do + [ -f "$c" ] && { CATALOG="$c"; break; } + done + [ -n "$CATALOG" ] || CATALOG="$PROJECT/.mxcli/catalog.db" +fi + +if [ ! -f "$FACTS/dependencies.json" ]; then + echo "cut-score: no $FACTS/dependencies.json. Run project-bin/app-facts.sh in the project first" >&2 + exit 2 +fi +if [ -z "$CUT" ] && [ ! -f "$LAYERMAP" ]; then + echo "cut-score: no --cut given and no $LAYERMAP to take the default cut from." >&2 + echo " Run bin/app-layer-map.sh first (one app per level) or pass --cut \"A,B|C|D\"." >&2 + exit 2 +fi + +FACTS="$FACTS" OUT="$OUT" JSON="$JSON" CUT="$CUT" CATALOG="$CATALOG" LAYERMAP="$LAYERMAP" \ + "$PY" - <<'PY' +import json, os, sys, html, datetime, collections, sqlite3 + +F = os.environ["FACTS"]; OUT = os.environ["OUT"]; JSON = os.environ["JSON"] == "1" +CUT = os.environ["CUT"]; CATALOG = os.environ["CATALOG"]; LAYERMAP = os.environ["LAYERMAP"] + +def load(path, required=False): + if not os.path.exists(path): + if required: + print("cut-score: %s is missing" % path, file=sys.stderr); sys.exit(2) + return None + try: + return json.load(open(path, encoding="utf-8")) + except ValueError as e: + if required: + print("cut-score: %s is not valid JSON: %s" % (path, e), file=sys.stderr); sys.exit(2) + return None + +dep = load(os.path.join(F, "dependencies.json"), required=True) +man = load(os.path.join(F, "manifest.json")) or {} + +gaps = [] # every fact this page wanted and did not get; printed, never swallowed +if not man: + gaps.append("manifest.json is missing or unreadable, so the page cannot say when the facts were collected") + +edges_raw = dep.get("edges") or [] +if not edges_raw: + gaps.append("dependencies.json carries no edges array: nothing can cross a boundary that nothing crosses") +own = sorted(set(x for x in (dep.get("scope", {}).get("own_modules") or []) if x)) +if not own: + own = sorted({e.get("from") for e in edges_raw} | {e.get("to") for e in edges_raw} - {None}) + gaps.append("scope.own_modules is absent, so the module set was derived from the edges; " + "a module with no edge at all is missing from this page") + +# ---- the cut ---------------------------------------------------------------------------------- +partitions = collections.OrderedDict() # name -> [modules] +cut_source = "" +if CUT.strip(): + cut_source = "--cut" + for i, part in enumerate(CUT.split("|")): + mods = [m.strip() for m in part.split(",") if m.strip()] + if not mods: + print("cut-score: --cut has an empty partition at position %d" % (i + 1), file=sys.stderr); sys.exit(2) + name = "+".join(mods) if len(mods) <= 2 else "%s+%d more" % (mods[0], len(mods) - 1) + if name in partitions: + name = "%s#%d" % (name, i + 1) + partitions[name] = mods +else: + lm = load(LAYERMAP, required=True) + levels = lm.get("levels_by_module") or {} + if not levels: + print("cut-score: %s has no levels_by_module" % LAYERMAP, file=sys.stderr); sys.exit(2) + cut_source = "default: one app per level of %s" % LAYERMAP + by_level = collections.defaultdict(list) + for m, lv in levels.items(): + by_level[int(lv)].append(m) + for lv in sorted(by_level): + partitions["L%d" % lv] = sorted(by_level[lv]) + +part_of = {} +for name, mods in partitions.items(): + for m in mods: + if m in part_of: + gaps.append("module %s is named in two partitions (%s and %s); the first wins" % (m, part_of[m], name)) + continue + part_of[m] = name +unknown = sorted(m for m in part_of if m not in own) +if unknown: + gaps.append("named in the cut but not an own module of this app (typo, marketplace, or dropped): %s" + % ", ".join(unknown)) +unassigned = sorted(m for m in own if m not in part_of) +if unassigned: + partitions["_unassigned"] = unassigned + for m in unassigned: + part_of[m] = "_unassigned" + gaps.append("own modules the cut does not place: %s — scored as a partition of their own named " + "_unassigned; decide where they go, nothing here guesses" % ", ".join(unassigned)) + +# ---- lane table (platform fact) --------------------------------------------------------------- +# kind -> (lane, cost, class, why) +LANES = { + "associate": ("odata-external-entity", "medium", "acceptable", + "the referenced entity becomes an external entity in the referencing app; the local " + "side keeps the association (an external entity cannot own one); the data stays with the owner"), + "retrieve": ("odata-external-entity", "medium", "acceptable", + "a database retrieve becomes an OData read; XPath is limited to what the service exposes"), + "datasource": ("odata-external-entity", "medium", "acceptable", + "a page datasource over a foreign entity reads an external entity; no client-side XPath beyond the service"), + "xpath": ("odata-external-entity", "medium", "acceptable", + "an XPath constraint over a foreign attribute must be expressible as an OData filter"), + "member": ("odata-external-entity", "low", "acceptable", + "attribute access on a foreign entity rides on the external entity; exposed attributes only"), + "parameter": ("odata-external-entity", "low", "acceptable", + "a parameter typed on a foreign entity retypes to the external entity"), + "return": ("odata-external-entity", "low", "acceptable", + "a return typed on a foreign entity retypes to the external entity"), + "type": ("odata-external-entity", "low", "acceptable", + "a type reference to a foreign entity retypes to the external entity"), + "create": ("odata-external-entity (write)", "high", "costly", + "creating a foreign object needs a writable external entity (OData v4) or an action in the owning app"), + "change": ("odata-external-entity (write)", "high", "costly", + "changing a foreign object needs a writable external entity or an action in the owning app"), + "commit": ("odata-external-entity (write)", "high", "costly", + "committing a foreign object needs a writable external entity; the transaction no longer spans both apps"), + "delete": ("odata-external-entity (write)", "high", "costly", + "deleting a foreign object needs a writable external entity or an action in the owning app"), + "call": ("odata-action-or-rest", "high", "costly", + "an in-process microflow call becomes a published OData action or a REST operation; " + "synchronous, no shared transaction, error handling changes"), + "action": ("odata-action-or-rest", "high", "costly", + "a widget action on a foreign microflow becomes a published action or REST call"), + "show_page": ("deep-link", "high", "costly", + "a page in another app cannot be shown in-process; the navigation becomes a URL"), + "home_page": ("deep-link", "high", "costly", "a home page in another app becomes a URL"), + "menu_item": ("deep-link", "high", "costly", "a menu item into another app becomes a URL"), + "layout": ("copy", "low", "acceptable", "a layout cannot be shared across apps; copy it or take Atlas"), + "mapping": ("copy", "low", "acceptable", "an import/export mapping cannot be shared; copy it"), + "value": ("copy", "low", "acceptable", "a constant/enum value reference; copy the document"), + "generalize": ("none", "blocker", "blocker", + "inheritance cannot cross an app boundary: an external entity has no specializations"), + "workflow": ("none", "blocker", "blocker", "a workflow cannot reference documents in another app"), + "user_task": ("none", "blocker", "blocker", "a user task cannot target a page or microflow in another app"), + "security_role": ("none", "blocker", "blocker", "a module role cannot be granted by another app's user roles"), + "role": ("none", "blocker", "blocker", "a role reference cannot cross an app boundary"), +} +UNKNOWN = ("unclassified", "unknown", "unclassified", + "this ref kind is not in the lane table; it is counted in the denominator and must be read by hand") +COST_RANK = {"low": 1, "medium": 2, "high": 3, "blocker": 4, "unknown": 3} + +# ---- crossing edges --------------------------------------------------------------------------- +rows = [] +total_edges = 0; total_weight = 0 +calls_between = collections.defaultdict(int) # (P,Q) -> call weight +for e in edges_raw: + a, b = e.get("from"), e.get("to") + if not a or not b or a == b: + continue + kinds = e.get("kinds") or {} + w_edge = int(e.get("edges") or sum(kinds.values()) or 0) + total_edges += 1; total_weight += w_edge + pa, pb = part_of.get(a), part_of.get(b) + if pa is None or pb is None: + gaps.append("edge %s -> %s has an endpoint outside the own-module set; skipped" % (a, b)) + continue + if pa == pb: + continue + if not kinds: + kinds = {"(kinds not recorded)": w_edge} + for kind, n in sorted(kinds.items()): + lane, cost, cls, why = LANES.get(kind, UNKNOWN) + if kind not in LANES: + gaps.append("ref kind %r on %s -> %s is not in the lane table" % (kind, a, b)) + rows.append({"from": a, "to": b, "from_partition": pa, "to_partition": pb, "kind": kind, + "weight": int(n), "lane": lane, "cost": cost, "class": cls, "why": why, + "aggregate": "NOT DETECTED"}) + if kind in ("call", "action"): + calls_between[(pa, pb)] += int(n) + +crossing_edges = len({(r["from"], r["to"]) for r in rows}) +crossing_weight = sum(r["weight"] for r in rows) + +# ---- bidirectional call pairs at partition level: the blocker --------------------------------- +bidir = [] +seen = set() +for (pa, pb), w in sorted(calls_between.items()): + if (pb, pa) in calls_between and (pb, pa) not in seen: + seen.add((pa, pb)) + bidir.append({"a": pa, "b": pb, "a_to_b_calls": w, "b_to_a_calls": calls_between[(pb, pa)], + "edges": sorted({"%s -> %s" % (r["from"], r["to"]) for r in rows + if r["kind"] in ("call", "action") + and {r["from_partition"], r["to_partition"]} == {pa, pb}})}) +bidir_pairs = {frozenset((x["a"], x["b"])) for x in bidir} +for r in rows: + if r["kind"] in ("call", "action") and frozenset((r["from_partition"], r["to_partition"])) in bidir_pairs: + r["lane"] = "none"; r["cost"] = "blocker"; r["class"] = "blocker" + r["why"] = ("part of a bidirectional call pair between %s and %s: each side would publish an action the " + "other consumes, and neither can deploy first" % (r["from_partition"], r["to_partition"])) + +# ---- aggregate detection from the catalog ----------------------------------------------------- +aggregate = {"status": "NOT DETECTED", "reason": "", "catalog": CATALOG, "hits": []} +if not os.path.exists(CATALOG): + aggregate["reason"] = ("no catalog at %s; app-facts.sh leaves one beside the .mpr after graph-report, " + "pass --catalog to point at it" % CATALOG) +else: + try: + db = sqlite3.connect("file:%s?mode=ro" % CATALOG, uri=True) + names = {r[0] for r in db.execute("select name from sqlite_master where type='table'")} + if "activities_data" not in names: + aggregate["reason"] = "catalog has no activities_data table; this mxcli wrote a catalog this script cannot read" + else: + cols = {r[1] for r in db.execute("pragma table_info(activities_data)")} + need = {"MicroflowQualifiedName", "ModuleName", "ActionType", "EntityRef"} + if not need <= cols: + aggregate["reason"] = "activities_data lacks columns %s" % ", ".join(sorted(need - cols)) + else: + mode = db.execute("select Value from catalog_meta where Key='build_mode'").fetchone() \ + if "catalog_meta" in names else None + if mode and mode[0] != "full": + aggregate["reason"] = "catalog build_mode is %r, activities are not trustworthy; rerun app-facts.sh" % mode[0] + else: + per_mf = collections.defaultdict(lambda: {"module": None, "aggregates": 0, "retrieves": []}) + for mf, mod, at, ent in db.execute( + "select MicroflowQualifiedName, ModuleName, ActionType, EntityRef from activities_data " + "where ActionType in ('AggregateListAction','RetrieveAction')"): + d = per_mf[mf]; d["module"] = mod + if at == "AggregateListAction": + d["aggregates"] += 1 + elif ent: + d["retrieves"].append(ent) + hits = [] + for mf, d in sorted(per_mf.items()): + if d["aggregates"] == 0 or not d["retrieves"]: + continue + pa = part_of.get(d["module"]) + for ent in sorted(set(d["retrieves"])): + emod = ent.split(".")[0] + pb = part_of.get(emod) + if pa and pb and pa != pb: + hits.append({"microflow": mf, "from": d["module"], "to": emod, "entity": ent, + "aggregates": d["aggregates"], "from_partition": pa, "to_partition": pb}) + aggregate["hits"] = hits + aggregate["status"] = "DETECTED" if hits else "none found" + aggregate["reason"] = ("basis: a microflow with a RetrieveAction on a foreign entity and an " + "AggregateListAction in the same microflow; the catalog stores no " + "variable linkage, so an aggregate over a list returned by a foreign CALL " + "is invisible here") + for h in hits: + for r in rows: + if r["from"] == h["from"] and r["to"] == h["to"] and r["kind"] == "retrieve": + r["aggregate"] = "yes (%s, %d aggregate%s)" % (h["microflow"], h["aggregates"], + "" if h["aggregates"] == 1 else "s") + r["cost"] = "high"; r["class"] = "costly" + r["why"] = ("the retrieved list is aggregated in %s: over an external entity that pulls " + "the whole set across the wire or moves the aggregation into the owning app" + % h["microflow"]) + db.close() + except sqlite3.Error as e: + aggregate["reason"] = "catalog could not be read: %s" % e +if aggregate["status"] == "NOT DETECTED": + gaps.append("aggregate detection: NOT DETECTED — %s" % aggregate["reason"]) +for r in rows: + if r["aggregate"] == "NOT DETECTED": + if aggregate["status"] == "NOT DETECTED": + r["aggregate"] = "NOT DETECTED (no catalog)" + elif r["kind"] in ("retrieve", "datasource"): + r["aggregate"] = "no" if r["kind"] == "retrieve" else "n/a (page)" + else: + r["aggregate"] = "n/a" + +# ---- per partition ---------------------------------------------------------------------------- +parts = collections.OrderedDict() +for name, mods in partitions.items(): + out_rows = [r for r in rows if r["from_partition"] == name] + in_rows = [r for r in rows if r["to_partition"] == name] + blockers = [r for r in out_rows + in_rows if r["class"] == "blocker"] + costly = [r for r in out_rows if r["class"] in ("costly", "unclassified")] + acceptable = [r for r in out_rows if r["class"] == "acceptable"] + # blocked: a blocker touches this partition in either direction (a tangle has no side). + # costly: something going OUT needs a published action, REST, a write, a deep link, or carries + # an aggregate. clean: nothing goes out, or only external-entity reads/types/copies do. + if blockers: + verdict = "blocked" + elif costly: + verdict = "costly" + else: + verdict = "clean" + parts[name] = { + "modules": mods, + "out_edges": len({(r["from"], r["to"]) for r in out_rows}), "out_weight": sum(r["weight"] for r in out_rows), + "in_edges": len({(r["from"], r["to"]) for r in in_rows}), "in_weight": sum(r["weight"] for r in in_rows), + "lanes": dict(collections.Counter(r["lane"] for r in out_rows)), + "blockers": sorted({"%s -> %s (%s): %s" % (r["from"], r["to"], r["kind"], r["why"]) for r in blockers}), + "costly": sorted({"%s -> %s (%s)" % (r["from"], r["to"], r["kind"]) for r in costly}), + "acceptable": sorted({"%s -> %s (%s)" % (r["from"], r["to"], r["kind"]) for r in acceptable}), + "aggregates_out": [h for h in aggregate["hits"] if h["from_partition"] == name], + "verdict": verdict, + } + +overall = "blocked" if any(p["verdict"] == "blocked" for p in parts.values()) else \ + "costly" if any(p["verdict"] == "costly" for p in parts.values()) else "clean" + +summary = { + "generated": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "facts": F, "catalog": CATALOG, "cut_source": cut_source, + "cut": [{"name": n, "modules": m} for n, m in partitions.items()], + "own_modules": len(own), "own_edges": total_edges, "own_edge_weight": total_weight, + "crossing_edges": crossing_edges, "crossing_weight": crossing_weight, "crossing_rows": len(rows), + "denominator": "%d of %d own edges cross a partition boundary, carrying %d of %d refs" % ( + crossing_edges, total_edges, crossing_weight, total_weight), + "rows": rows, "bidirectional_partition_pairs": bidir, "aggregate": aggregate, + "partitions": parts, "verdict": overall, "gaps": gaps, + "lanes_excluded": ["business-events (no Event Broker licence)"], +} + +if JSON: + print(json.dumps(summary, indent=2, sort_keys=True)); sys.exit(0) + +# ---- HTML ------------------------------------------------------------------------------------- +E_ = html.escape +app_name = man.get("mpr") or os.path.basename(os.path.dirname(os.path.dirname(F))) or "app" + +CSS = """ +:root{--bg:#fbfbfd;--fg:#16181d;--mut:#606878;--line:#e2e5ec;--card:#fff; +--ok:#1a7f52;--warn:#a2650a;--bad:#b3261e;--crit:#7a1710;--acc:#2b5bd7} +*{box-sizing:border-box} +body{margin:0;background:var(--bg);color:var(--fg); +font:14px/1.5 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif} +.wrap{max-width:1300px;margin:0 auto;padding:22px 20px 60px} +header h1{font-size:23px;margin:0 0 3px} +.sub{color:var(--mut);margin:0 0 16px;font-size:12.5px} +section{background:var(--card);border:1px solid var(--line);border-radius:10px;padding:16px 18px;margin:0 0 16px} +h2{font-size:16px;margin:0 0 10px} +h3{font-size:13.5px;margin:18px 0 7px;text-transform:uppercase;letter-spacing:.05em;color:var(--mut)} +p{margin:0 0 9px} +.lede{font-size:14.5px} +.kpis{display:flex;flex-wrap:wrap;gap:9px;margin:12px 0 4px} +.kpi{border:1px solid var(--line);border-radius:8px;padding:8px 12px;min-width:112px;background:var(--card)} +.kpi b{display:block;font-size:21px;line-height:1.15} +.kpi span{color:var(--mut);font-size:11.5px} +.kpi.k-bad b{color:var(--bad)} .kpi.k-ok b{color:var(--ok)} .kpi.k-warn b{color:var(--warn)} +.note{border-radius:8px;padding:9px 12px;margin:10px 0;font-size:13px} +.note.fault{background:#fdecea;border:1px solid #f3b7b1;color:var(--crit)} +.note.warn{background:#fdf5e4;border:1px solid #efd9a4;color:var(--warn)} +.legend{color:var(--mut);font-size:12.2px;margin-top:10px} +.tw{overflow-x:auto;margin:6px 0} +table{border-collapse:collapse;width:100%;font-size:12.8px} +th,td{text-align:left;padding:5px 9px;border-bottom:1px solid var(--line);vertical-align:top} +th{color:var(--mut);font-weight:600;font-size:11.5px;text-transform:uppercase;letter-spacing:.04em} +td.why{white-space:normal;color:var(--mut);font-size:12px} +code{font:12px/1.4 ui-monospace,SFMono-Regular,Menlo,monospace;background:#f1f2f6;padding:1px 4px;border-radius:4px} +.v{display:inline-block;padding:1px 8px;border-radius:10px;font-size:11.5px;font-weight:600;color:#fff} +.v.clean{background:var(--ok)} .v.costly{background:var(--warn)} .v.blocked{background:var(--bad)} +.c-blocker{color:var(--bad);font-weight:600} .c-costly{color:var(--warn);font-weight:600} .c-acceptable{color:var(--ok)} .c-unclassified{color:var(--bad)} +.empty{color:var(--mut);font-style:italic} +@media(prefers-color-scheme:dark){ +:root{--bg:#131519;--fg:#e8eaee;--mut:#98a0b0;--line:#2b2f38;--card:#191c21} +.note.fault{background:#33191a;border-color:#6b2b26;color:#ffb3a8} +.note.warn{background:#2e2716;border-color:#6a5527;color:#f0cf8a} +code{background:#23262d}} +""" + +def table(headers, body_rows, cls=None): + if not body_rows: + return '

none

' + out = ['
%s' % "".join("" % E_(h) for h in headers)] + for r in body_rows: + cells = [] + for i, c in enumerate(r): + if isinstance(c, tuple): + cells.append('' % (c[1], c[0])) + else: + cells.append("%s" % (' class="why"' if cls and cls[i] == "why" else "", E_(str(c)))) + out.append("%s" % "".join(cells)) + out.append("
%s%s
") + return "".join(out) + +def vbadge(v): + return '%s' % (v, v) + +# The per-partition cell names each blocking edge once, without its "why": that sentence is on the +# edge's own row below. On a real monolith (FSM 9.24.2, 2026-10-12: 80 blocking rows on one cut) the +# full text made one cell of 10 KB and the stdout summary a 16 KB line nobody could read. +def edge_only(b): + return b.split(": ", 1)[0] + +def blockers_cell(blockers, cap=12): + if not blockers: + return "-" + out = ["%d blocking edge%s" % (len(blockers), "" if len(blockers) == 1 else "s")] + out += [E_(edge_only(b)) for b in blockers[:cap]] + if len(blockers) > cap: + out.append("… %d more, all in the rows below" % (len(blockers) - cap)) + return "
".join(out) + +p = [] +p.append('

Cut score: %s

%s%s · facts %s · rendered %s · ' + 'the judgement for this page is skills/layering-review.md and the Stage 3 interview

' % ( + E_(app_name), E_(man.get("mpr") or "mpr not named"), + (" · " + E_(man.get("mxcli"))) if man.get("mxcli") else "", + E_(man.get("generated") or "collection date unknown"), E_(summary["generated"]))) + +for g in gaps: + p.append('
%s
' % E_(g)) + +p.append('

Verdict %s

' % vbadge(overall)) +p.append('

Cut (%s): %s.

' % (E_(cut_source), " | ".join( + "%s = %s" % (E_(n), E_(", ".join(m))) for n, m in partitions.items()))) +p.append('

%s. Lanes available: OData external entity, published OData action or REST. ' + 'Excluded: Business Events (no Event Broker licence).

' % E_(summary["denominator"])) +kp = [("%d" % len(partitions), "partitions", ""), + ("%d / %d" % (crossing_edges, total_edges), "crossing edges / own edges", ""), + ("%d / %d" % (crossing_weight, total_weight), "crossing refs / own refs", ""), + ("%d" % len(bidir), "bidirectional call pairs", "k-bad" if bidir else "k-ok"), + ("%d" % sum(1 for r in rows if r["class"] == "blocker"), "blocker rows", "k-bad" if any(r["class"] == "blocker" for r in rows) else "k-ok"), + (("%d" % len(aggregate["hits"])) if aggregate["status"] != "NOT DETECTED" else "n/d", + "cross-partition aggregates" + ("" if aggregate["status"] != "NOT DETECTED" else " (NOT DETECTED)"), + "k-warn" if aggregate["hits"] or aggregate["status"] == "NOT DETECTED" else "k-ok")] +p.append('
%s
' % "".join('
%s%s
' % (c, E_(v), E_(l)) for v, l, c in kp)) +p.append('
') + +p.append('

Per partition

') +p.append(table(["Partition", "Modules", "Out edges (refs)", "In edges (refs)", "Lanes out", "Blockers", "Verdict"], + [[n, ", ".join(d["modules"]), "%d (%d)" % (d["out_edges"], d["out_weight"]), + "%d (%d)" % (d["in_edges"], d["in_weight"]), + ", ".join("%s x%d" % (k, v) for k, v in sorted(d["lanes"].items())) or "-", + (blockers_cell(d["blockers"]), "why"), + (vbadge(d["verdict"]), "")] for n, d in parts.items()])) +if bidir: + p.append('

Bidirectional call pairs (blocker)

') + p.append(table(["A", "B", "A->B calls", "B->A calls", "Edges"], + [[b["a"], b["b"], b["a_to_b_calls"], b["b_to_a_calls"], "; ".join(b["edges"])] for b in bidir])) +p.append('
') + +p.append('

Every crossing edge (%d rows over %d module edges)

' % (len(rows), crossing_edges)) +p.append(table(["From", "To", "Partitions", "Ref kind", "Refs", "Lane", "Cost", "Class", "Aggregate", "Why"], + [[r["from"], r["to"], "%s -> %s" % (r["from_partition"], r["to_partition"]), r["kind"], r["weight"], + r["lane"], r["cost"], (E_(r["class"]), "c-" + r["class"]), r["aggregate"], r["why"]] for r in + sorted(rows, key=lambda r: (-COST_RANK.get(r["cost"], 0), r["from"], r["to"], r["kind"]))], + cls=[None, None, None, None, None, None, None, None, None, "why"])) +p.append('
') + +p.append('

Aggregates over foreign entities: %s

' % E_(aggregate["status"])) +p.append('

%s

' % E_(aggregate["reason"])) +p.append('

catalog: %s

' % E_(CATALOG)) +if aggregate["hits"]: + p.append(table(["Microflow", "From", "Retrieves", "Aggregates in microflow", "Partitions"], + [[h["microflow"], h["from"], h["entity"], h["aggregates"], + "%s -> %s" % (h["from_partition"], h["to_partition"])] for h in aggregate["hits"]])) +p.append('
') + +p.append('

Method, and what this page cannot see

    ') +for line in [ + "Source: analysis/app-facts/dependencies.json (edges with per-kind counts, scope.own_modules) and, for " + "aggregates, the mxcli catalog beside the .mpr. Nothing here re-reads the model.", + "A crossing edge is a module-to-module edge whose endpoints sit in different partitions. Edges inside a " + "partition are not listed; they are in the denominator above.", + "Lanes are platform facts, not judgement: an external entity cannot own an association, be specialised, " + "be aggregated server-side, or be the subject of a workflow; a call across apps is a published action or " + "REST operation with no shared transaction. Business Events is excluded by licence.", + "Blocker = bidirectional call pair between two partitions, or a generalize/workflow/security-role crossing. " + "Costly = a call, a write to a foreign object, a page shown across apps, or a retrieve that feeds an aggregate. " + "Acceptable = association, read, type or copy over an external entity (the owner keeps the data). " + "A partition is blocked if a blocker touches it in either direction, costly if anything costly leaves it, " + "clean if nothing or only acceptable edges leave it.", + "Aggregate detection attributes an AggregateListAction to a RetrieveAction in the same microflow. It cannot " + "see an aggregate over a list returned by a cross-partition call, nor how many objects an OData hop would move.", + "Not visible at all: Java and JavaScript actions, published REST called by name, workflow references, page URLs. " + "An edge that only exists through one of those is missing here, which is why a cut is confirmed in Studio Pro " + "before it becomes a slice.", +]: + p.append("
  • %s
    • " % E_(line)) +p.append('

An agent reading this later should take the JSON, not the page: ' + 'bin/cut-score.sh <project> --cut "..." --json emits every row with its lane, cost and aggregate flag.

') +p.append('
') + +doc = ("" + "" + "Cut score: %s
%s
" + % (E_(app_name), CSS, "\n".join(p))) + +d = os.path.dirname(OUT) +if d and not os.path.isdir(d): + os.makedirs(d) +open(OUT, "w", encoding="utf-8").write(doc) +json_out = os.path.splitext(OUT)[0] + ".json" +open(json_out, "w", encoding="utf-8").write(json.dumps(summary, indent=2, sort_keys=True)) +print("cut-score: %s" % OUT) +print("cut-score: %s" % json_out) +print(" verdict %s; %s; %d bidirectional call pair(s); aggregates %s (%d)" % ( + overall, summary["denominator"], len(bidir), aggregate["status"], len(aggregate["hits"]))) +for n, d_ in parts.items(): + print(" %-28s %-8s out %d(%d) in %d(%d)%s" % (n, d_["verdict"], d_["out_edges"], d_["out_weight"], + d_["in_edges"], d_["in_weight"], + (" blockers: %d (%s%s)" % (len(d_["blockers"]), + "; ".join(edge_only(b) for b in d_["blockers"][:3]), + " …" if len(d_["blockers"]) > 3 else "")) if d_["blockers"] else "")) +for g in gaps: + print(" gap: %s" % g) +PY diff --git a/bin/gate-check.sh b/bin/gate-check.sh index d24a021c..4a4a5d8b 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -1766,13 +1766,13 @@ stage_protocol_paths() { case "$1" in # P) echo "skills/interview-protocol.md skills/grill-mode.md skills/agent-roles.md skills/bootstrap-project.md skills/cloud-dev-environment.md skills/existing-app-change.md skills/app-analysis.md skills/mendix-epics-api.md skills/corpus-extraction-integrity.md skills/platform-link.md" ;; - 0) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-scope.md skills/source-triage.md skills/image-transcription.md skills/small-project-tier.md skills/existing-app-change.md skills/app-analysis.md skills/layering-review.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/assess-migration.md skills/migration-pipeline.md skills/migrate-general.md skills/migrate-outsystems.md skills/source-os11.md skills/os-xml-schema.md skills/source-node-express-react.md skills/document-discovery.md skills/extractor-quality-loop.md skills/qa-loop-goal-pattern.md skills/mendix-epics-api.md skills/corpus-extraction-integrity.md skills/gate-check-file-locations.md" ;; - 1) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-extraction.md skills/agent-roles.md skills/image-transcription.md skills/small-project-tier.md skills/migration-pipeline.md skills/source-os11.md skills/os-xml-schema.md skills/source-node-express-react.md skills/document-discovery.md skills/extractor-quality-loop.md skills/kb-generation.md skills/corpus-extraction-integrity.md" ;; + 0) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-scope.md skills/source-triage.md skills/image-transcription.md skills/small-project-tier.md skills/existing-app-change.md skills/decompose-app.md skills/app-analysis.md skills/layering-review.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/assess-migration.md skills/migration-pipeline.md skills/rehost-app.md skills/migrate-general.md skills/migrate-outsystems.md skills/source-os11.md skills/os-xml-schema.md skills/source-node-express-react.md skills/document-discovery.md skills/extractor-quality-loop.md skills/qa-loop-goal-pattern.md skills/mendix-epics-api.md skills/corpus-extraction-integrity.md skills/gate-check-file-locations.md" ;; + 1) echo "skills/interview-protocol.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-extraction.md skills/agent-roles.md skills/image-transcription.md skills/small-project-tier.md skills/migration-pipeline.md skills/rehost-app.md skills/source-os11.md skills/os-xml-schema.md skills/source-node-express-react.md skills/document-discovery.md skills/extractor-quality-loop.md skills/kb-generation.md skills/corpus-extraction-integrity.md" ;; 2) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-brd.md skills/checkpoints/checkpoint-architecture.md skills/agent-roles.md skills/image-transcription.md skills/small-project-tier.md skills/kb-generation.md skills/brd-generation.md skills/brd-validation.md" ;; - 3) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-design.md skills/agent-roles.md skills/small-project-tier.md skills/mendix-best-practices-index.md skills/layering-review.md skills/architecture-blueprint.md skills/modularize-domain.md skills/design-artifacts.md skills/brd-to-build-plan.md skills/workflow-structure-rules.md skills/learned-mdl-cannot-express.md" ;; + 3) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-design.md skills/agent-roles.md skills/small-project-tier.md skills/mendix-best-practices-index.md skills/decompose-app.md skills/mendix-service-seam.md skills/layering-review.md skills/architecture-blueprint.md skills/modularize-domain.md skills/design-artifacts.md skills/brd-to-build-plan.md skills/workflow-structure-rules.md skills/learned-mdl-cannot-express.md" ;; 4) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-build.md skills/agent-roles.md skills/small-project-tier.md skills/module-brief.md skills/module-folder-convention.md skills/brd-to-build-plan.md skills/coverage-ledger.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/learned-constants-and-secrets.md" ;; - 5|build-ready) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/agent-roles.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/microflow-loop-antipatterns.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/e2e-locators-that-lie.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; - 6) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/agent-roles.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/e2e-locators-that-lie.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 5|build-ready) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/agent-roles.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/decompose-app.md skills/mendix-service-seam.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/e2e-locators-that-lie.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 6) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/agent-roles.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/decompose-app.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/e2e-locators-that-lie.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; 7) echo "skills/interview-protocol.md skills/change-intake.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/close-the-loop.md skills/share-demo-package.md skills/handoff-to-studio-pro.md skills/platform-link.md skills/teamserver-alignment.md skills/deploy-to-sandbox.md" ;; *) echo "" ;; # diff --git a/bin/lib/cut-ledger.py b/bin/lib/cut-ledger.py new file mode 100644 index 00000000..9155dd98 --- /dev/null +++ b/bin/lib/cut-ledger.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +"""cut-ledger.py — engine of bin/cut-ledger.sh (read that header; this file has no CLI of its own +beyond what the wrapper passes). Python 3 stdlib only.""" +import argparse +import fnmatch +import json +import os +import re +import sqlite3 +import sys + +COLS = ["edge", "direction", "kind", "source_doc", "target_doc", "refs", "resolution", "evidence"] +RESOLUTIONS = ("resolved", "stub", "handoff", "retire", "platform") +REFS_COLS = ["SourceType", "SourceName", "TargetType", "TargetName", "RefKind"] + + +def die(msg): + print(msg, file=sys.stderr) + sys.exit(2) + + +def doc_of(name): + """Qualified document name: the first two dotted parts (Module.Doc[.Member…] -> Module.Doc).""" + p = name.split(".") + return ".".join(p[:2]) if len(p) >= 2 else name + + +def read_slice(path): + mods, docs = set(), set() + for line in open(path, encoding="utf-8"): + line = line.split("#", 1)[0].strip() + if not line: + continue + (docs if "." in line else mods).add(line) + if not mods and not docs: + die("cut-ledger: slice file %s names nothing" % path) + return mods, docs + + +def in_slice(name, mods, docs): + return name.split(".")[0] in mods or doc_of(name) in docs + + +def read_refs(args): + rows = [] + if args.catalog: + db = sqlite3.connect("file:%s?mode=ro" % args.catalog, uri=True) + if not db.execute("select 1 from sqlite_master where name='refs'").fetchone(): + die("cut-ledger: %s has no refs table (not an mxcli catalog; run graph-report)" % args.catalog) + for r in db.execute("select %s from refs" % ", ".join(REFS_COLS)): + rows.append(dict(zip(REFS_COLS, r))) + else: + with open(args.refs, encoding="utf-8") as f: + head = f.readline().rstrip("\n").split("\t") + if head != REFS_COLS: + die("cut-ledger: %s header must be %s" % (args.refs, "\t".join(REFS_COLS))) + for line in f: + v = line.rstrip("\n").split("\t") + if len(v) == len(REFS_COLS): + rows.append(dict(zip(REFS_COLS, v))) + return rows + + +def read_rules(path): + rules = [] + for n, line in enumerate(open(path, encoding="utf-8"), 1): + if not line.strip() or line.lstrip().startswith("#"): + continue + v = line.rstrip("\n").split("\t") + if len(v) < 4: + die("cut-ledger: %s:%d needs 4 tab-separated fields (direction, pattern, resolution, evidence)" % (path, n)) + d, pat, res, ev = v[0].strip(), v[1].strip(), v[2].strip(), v[3].strip() + if d not in ("out", "in", "check", "*"): + die("cut-ledger: %s:%d direction %r is not out|in|check|*" % (path, n, d)) + if res.split(":")[0] not in RESOLUTIONS: + die("cut-ledger: %s:%d resolution %r is not one of %s (a ':detail' suffix is allowed)" % (path, n, res, "|".join(RESOLUTIONS))) + if not ev: + die("cut-ledger: %s:%d has no evidence; a resolution without evidence is a claim" % (path, n)) + rules.append((d, pat, res, ev, n)) + return rules + + +def match(rules, direction, other_doc, own_doc): + """First rule whose direction fits and whose pattern matches the far-side doc. + A pattern 'A -> B' matches own side A and far side B (both fnmatch).""" + for d, pat, res, ev, n in rules: + if d not in ("*", direction): + continue + if "->" in pat: + a, b = [x.strip() for x in pat.split("->", 1)] + if fnmatch.fnmatchcase(own_doc, a) and fnmatch.fnmatchcase(other_doc, b): + return res, ev, n + elif fnmatch.fnmatchcase(other_doc, pat): + return res, ev, n + return None + + +CHECK_NAME = re.compile(r"\[error\] \[CE1613\] \"The selected [^']*'([A-Za-z0-9_]+\.[^']+)' no longer exists") + + +def main(): + ap = argparse.ArgumentParser(prog="cut-ledger") + src = ap.add_mutually_exclusive_group(required=True) + src.add_argument("--catalog") + src.add_argument("--refs") + ap.add_argument("--slice", required=True) + ap.add_argument("--rules", required=True) + ap.add_argument("--check-log", action="append", default=[]) + ap.add_argument("--exporter-json", action="append", default=[]) + ap.add_argument("--extra", action="append", default=[]) + ap.add_argument("--dump-refs") + ap.add_argument("-o", "--out", required=True) + args = ap.parse_args() + + mods, docs = read_slice(args.slice) + refs = read_refs(args) + rules = read_rules(args.rules) + + crossing = [r for r in refs if in_slice(r["SourceName"], mods, docs) != in_slice(r["TargetName"], mods, docs)] + if args.dump_refs: + with open(args.dump_refs, "w", encoding="utf-8") as f: + f.write("\t".join(REFS_COLS) + "\n") + for r in sorted(crossing, key=lambda r: (r["SourceName"], r["TargetName"], r["RefKind"])): + f.write("\t".join(r[c] for c in REFS_COLS) + "\n") + + # one ledger row per (direction, kind, source doc, target doc); refs = how many catalog refs it carries + agg = {} + for r in crossing: + direction = "out" if in_slice(r["SourceName"], mods, docs) else "in" + key = (direction, r["RefKind"], doc_of(r["SourceName"]), doc_of(r["TargetName"])) + agg[key] = agg.get(key, 0) + 1 + + rows, used_rules = [], set() + for (direction, kind, s, t), n in sorted(agg.items(), key=lambda kv: (kv[0][0] != "out", kv[0][3], kv[0][2], kv[0][1])): + own, other = (s, t) if direction == "out" else (t, s) + if other.split(".")[0] == "System" and direction == "out": + res, ev = "platform", "System module exists in every Mendix app" + else: + m = match(rules, direction, other, own) + if m: + res, ev, rn = m + used_rules.add(rn) + else: + res, ev = "UNRESOLVED", "no rule in %s matches %s" % (os.path.basename(args.rules), other) + rows.append([direction, kind, s, t, str(n), res, ev]) + + # mx check names the catalog never recorded (widget actions, layout placeholders, images, ...) + covered = {r[3] for r in rows if r[0] == "out"} | {doc_of(r[3]) for r in rows if r[0] == "out"} + check_extra = {} + for log in args.check_log: + for line in open(log, encoding="utf-8", errors="replace"): + m = CHECK_NAME.search(line) + if not m: + continue + name = m.group(1) + if in_slice(name, mods, docs) or doc_of(name) in covered: + continue + check_extra[doc_of(name)] = check_extra.get(doc_of(name), 0) + 1 + for d, n in sorted(check_extra.items()): + m = match(rules, "check", d, "") or match(rules, "out", d, "") + if m: + res, ev, rn = m + used_rules.add(rn) + else: + res, ev = "UNRESOLVED", "named by mx check, not in the catalog, no rule matches" + rows.append(["out", "check-only", "(mx check)", d, str(n), res, ev]) + + # edges no catalog holds (second-order edges of a module brought along, app-level settings, + # Java hand-offs): direction, kind, source_doc, target_doc, refs, resolution, evidence — verbatim + for ex in args.extra: + for n, line in enumerate(open(ex, encoding="utf-8"), 1): + if not line.strip() or line.lstrip().startswith("#"): + continue + v = line.rstrip("\n").split("\t") + if len(v) != 7 or v[5].split(":")[0] not in RESOLUTIONS or not v[6].strip(): + die("cut-ledger: %s:%d needs 7 fields with a known resolution and evidence" % (ex, n)) + rows.append(v) + + # reconcile against the exporter's per-module counts: a module it names with no ledger row is reported + exporter_notes = [] + out_mods = {} + for r in rows: + if r[0] == "out": + out_mods[r[3].split(".")[0]] = out_mods.get(r[3].split(".")[0], 0) + 1 + for ej in args.exporter_json: + txt = open(ej, encoding="utf-8").read() + j = json.loads(txt[txt.index("{"):txt.rindex("}") + 1]) + for mod, units in sorted(j.get("crossing_refs_by_module", {}).items()): + if mod not in out_mods: + exporter_notes.append("%s: exporter counts %d unit(s) naming it, ledger has 0 out-edges to it" % (mod, units)) + + with open(args.out, "w", encoding="utf-8") as f: + f.write("\t".join(COLS) + "\n") + for i, r in enumerate(rows, 1): + f.write("\t".join(["E%03d" % i] + r) + "\n") + + tot = {} + for r in rows: + k = r[5].split(":")[0] + tot[k] = tot.get(k, 0) + 1 + unused = [n for d, p, res, ev, n in rules if n not in used_rules] + n_extra = len(rows) - len(agg) - len(check_extra) + print("cut-ledger: %d crossing refs -> %d ledger rows (%d out, %d in from the catalog; %d check-only; %d extra) -> %s" % ( + len(crossing), len(rows), sum(1 for k in agg if k[0] == "out"), sum(1 for k in agg if k[0] == "in"), + len(check_extra), n_extra, args.out)) + print(" by resolution: " + ", ".join("%s %d" % (k, tot[k]) for k in sorted(tot))) + for note in exporter_notes: + print(" exporter-only: " + note) + if unused: + print(" rules never used (lines): " + ", ".join(map(str, unused))) + if tot.get("UNRESOLVED"): + print(" UNRESOLVED rows: %d — every crossing edge needs a rule (resolved, stub, handoff, retire)" % tot["UNRESOLVED"]) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bin/lib/skill-routing.tsv b/bin/lib/skill-routing.tsv index 200b80e4..f3d9493b 100644 --- a/bin/lib/skill-routing.tsv +++ b/bin/lib/skill-routing.tsv @@ -122,6 +122,8 @@ bootstrap-project skills/bootstrap-project.md Generating a new project's CLAUDE. cloud-dev-environment skills/cloud-dev-environment.md Setting up or resuming an mxcli project in a cloud/ephemeral container — the one-time setup order (mxcli download → mxcli init → init-project.sh → sources decision → push) and the commit-and-push loop that survives container reclaim all P ondemand spine existing-app-assurance skills/existing-app-assurance.md Auditing or regression/e2e-testing an EXISTING app — no intake, no stages, no gates test,review 6 ondemand verify existing-app-change skills/existing-app-change.md Changing an EXISTING Mendix app — adding a feature, altering a flow, restructuring a module — when it has no BRDs, no architecture doc and no wireframes: the knowledge base comes from the live model (Path D), stages 2–4 run over what changes plus what it touches, nothing more, and the Track B regression baseline is the precondition; audit-only stays in existing-app-assurance ba,architect P,0 ondemand spine +decompose-app skills/decompose-app.md Splitting an existing Mendix monolith into several apps — strangler, "move module X into its own app", carving out a slice, microservices from one .mpr: cut choice on business fit AND coupling (bin/cut-score.sh), cut order by dependency direction, MPK move, every crossing edge on bin/cut-ledger.sh, OData/REST seam, and the build-green / seam-live / strangled level each slice reached lead,architect,mdl 0,3,5,6 ondemand spine +mendix-service-seam skills/mendix-service-seam.md Mendix-to-Mendix service seam — one Mendix app's OData/REST service consumed by another Mendix app: consuming its OData as external entities, replacing a stub by an external entity, joining data that lives in the other Mendix app (mirror keyed by the consumer's key), or wiring the seam between two apps after a split; keys (PublicID AutoNumber), machine-role-only access (200/401/404), REST actions with error handlers, external-entity limits lead,architect,mdl 3,5 ondemand build/integration app-facts project-bin/app-facts.sh Collecting the facts an app dossier is written from: forces a full catalog build, module edges through the real module column, strongly connected components, describes every loop-containing microflow and parses the loop bodies; exit 2 on a stale, fast-mode or schema-incomplete catalog, never a verdict architect,review 0,6 ondemand architecture app-analysis skills/app-analysis.md Building or refreshing the standing dossier of an EXISTING app before changing it: inventory, module dependency shape, loop risk patterns, each section with a verdict and a fault where nothing was measured (a standing document; nothing else in the toolkit reads it yet) ba,architect,review P,0,6 ondemand architecture layering-review skills/layering-review.md Checking whether an EXISTING app's module boundaries still hold: the layer map (bin/app-layer-map.sh) computes the order the modules would stack in and draws only the edges that point back up, so one tangle of N mutually reachable modules becomes a named list of edges with a weight and a ref kind; Stage 3 in existing-app-change mode, and the honest blast radius for a slice architect,ba 0,3 ondemand architecture @@ -129,6 +131,7 @@ module-dependency-review skills/module-dependency-review.md Judging module tangl microflow-loop-antipatterns skills/microflow-loop-antipatterns.md Reading what loop bodies do (LOOP_TQ, deferred commit, nested loop, REST in loop, transaction control per item, scheduled-event reachability) from described MDL; the catalog holds top-level activities only and cannot see inside a loop architect,review,mdl 0,5,6 ondemand build/mdl assess-migration skills/assess-migration.md Assessing or planning a migration up front, before any pipeline is chosen ba 0 ondemand source migration-pipeline skills/migration-pipeline.md Running the extraction pipeline ba 0,1 ondemand source +rehost-app skills/rehost-app.md Rehosting or regenerating an existing Mendix .mpr as MDL into a fresh app — "port this app to a clean Mendix 11 project", re-platform a whole app, rebuild it from its own model: Rehost vs MPK move, pipelines/mendix-mpr/rehost-extract.sh and its check/quarantine/exec/resume loop, reading the fidelity ledger, done = build-green with named hand-offs lead,ba,mdl 0,1,5,6 ondemand source migrate-general skills/migrate-general.md Migrating from a stack that has no dedicated pipeline ba 0 ondemand source migrate-outsystems skills/migrate-outsystems.md Migrating an OutSystems app ba 0 ondemand source source-os11 skills/source-os11.md Understanding OutSystems 11 source ba 0,1 ondemand source diff --git a/bug-logs/mxcli-upstream-report-2026-10-12.md b/bug-logs/mxcli-upstream-report-2026-10-12.md new file mode 100644 index 00000000..99ce646e --- /dev/null +++ b/bug-logs/mxcli-upstream-report-2026-10-12.md @@ -0,0 +1,927 @@ +# mxcli upstream bug report — draft, 2026-10-12 + +**Status: DRAFT for the mxcli maintainers (mendixlabs/mxcli). Not filed. No issue has been opened and nothing has been posted anywhere.** + +- **mxcli version:** 0.26.0 (2026-10-11), Linux cloud container, no Studio Pro +- **Apps used:** Siemens FSM template 9.24.2 (and 10.24 / 11.13.0 conversions of it); mendix/genai-showcase-app 11.12.0; ako/mxcli-banking 11.13.0 +- **Date:** 2026-10-12 +- **Source:** field notes of one day of round-trip runs; every verbatim message was copied from a capture. Paths are written `/`; repro files cited by repo-relative path. +- **Severity:** *blocks round-trip* = describe output rejected by check/exec, or data loss; *silent drift* = passes but wrong; *usability*. +- **Occurrences** are counts as the source notes state them (error lines, documents, modules or reports); "not counted" means the notes give none. +- **Ledger:** `BUG-n` is given only where `bin/bug-lookup.sh` found a related entry; none is an exact duplicate. + +## Summary + +| id | title | group | severity | occurrences | +|---|---|---|---|---| +| U1 | describe prints a nanoflow data source/action without its argument mapping | Describe output rejected by mxcli itself | blocks round-trip | 70 | +| U2 | describe emits `content { }` for native pluggable-widget child slots that are named `template` | Describe output rejected by mxcli itself | blocks round-trip | 35 | +| U3 | describe entity emits `index ( desc);` with no column | Describe output rejected by mxcli itself | blocks round-trip | 25 | +| U4 | legacy widgets (Forms$DataGrid) have no headless route to the React client; describe prints no data source or columns | Widgets and native | blocks round-trip | 22 x CE0582 | +| U5 | rules cannot be described or created | Describe omits or truncates | blocks round-trip | 14 rules, 8 flows deferred | +| U6 | describe loses or never emits the data view a widget sat in | Describe output rejected by mxcli itself | blocks round-trip | 12 | +| U7 | Java/JS action type parameters are not expressible: describe emits a form check refuses or exec writes entity `.Entity` | Describe output rejected by mxcli itself | blocks round-trip | about 9 | +| U8 | message definitions cannot be expressed; export mappings built on them fail | Describe omits or truncates | blocks round-trip | 4 x CE1613 | +| U9 | describe emits catalog order; exec and check refuse forward and cross-module references, so mutual recursion cannot be created | Check/exec semantics | blocks round-trip | 4 reports | +| U10 | ContentParams parameters are described as `$obj/Attr` paths, which check refuses | Describe output rejected by mxcli itself | blocks round-trip | 3 | +| U11 | three more describe outputs check cannot parse: import-mapping `root (Wrapper)`, dynamic-text `decimalPrecision: -1`, `DecimalPrecision` -1 | Describe output rejected by mxcli itself | blocks round-trip | 3 | +| U12 | exec is not atomic after the pre-check passes | Check/exec semantics | blocks round-trip | 3 | +| U13 | a grant naming an association member is refused when it precedes the association | Check/exec semantics | blocks round-trip | 3 | +| U14 | microflows and nanoflows cannot be created or altered in a 9.x project | Mendix 9.x project support | blocks round-trip | 3 reports | +| U15 | Layout placeholders inside a pluggable widget's child slot are not seen: `declares no placeholders at all` | Describe output rejected by mxcli itself | blocks round-trip | 2 | +| U16 | describe of Java action puts the parameter separator after the `--` comment | Describe output rejected by mxcli itself | blocks round-trip | 1 run | +| U17 | describe of a microflow emits a variable use check calls undeclared | Describe output rejected by mxcli itself | blocks round-trip | 1 | +| U18 | describe of a merge/decision cannot be re-created (#923): WARNING lines, layout drift | Describe output rejected by mxcli itself | blocks round-trip | 1 | +| U19 | describe emits a selection data source naming a widget it did not emit | Describe output rejected by mxcli itself | blocks round-trip | 1 | +| U20 | create refuses an enum split with more than 16 branches that describe produced | Describe output rejected by mxcli itself | blocks round-trip | 1 | +| U21 | lint MDL-REST04 flags a valid authentication microflow | Describe output rejected by mxcli itself | blocks round-trip | 1 | +| U22 | exec writes an import-mapping microflow with a null entity and the `.mpr` can no longer be loaded | Check/exec semantics | blocks round-trip | 1 | +| U23 | a microflow created before a Java action of the same name fails with a misleading error | Check/exec semantics | blocks round-trip | 1 | +| U24 | exec's reference pre-check does not see an image collection created earlier in the same script | Check/exec semantics | blocks round-trip | 1 | +| U25 | `marketplace install --file` of a 10.x package into 11.x does not warn that its pages will fail | Marketplace and project setup | silent drift | 22 x CE0582 | +| U26 | combo box in association mode bound to a param-bound snippet widget passes check and exec, then fails CE7007 | Widgets and native | silent drift | 9 x CE7007 | +| U27 | `describe module ... with all` omits whole document types | Describe omits or truncates | silent drift | 4 reports | +| U28 | check accepts `change $E (...)` / `$E = empty` where `$E` comes from `retrieve ... limit 1` (a list under mdl 1); mx check then fails, and a splice does not clear it | Check/exec semantics | silent drift | 4 x CE1613 | +| U29 | published OData `expose ("Description")` stores the quotes in the attribute name | OData/REST authoring | silent drift | 4 x CE1613 | +| U30 | mxcli-written Data grid 2 fails CE0463 on a 10.x target until `fix widgets` | Widgets and native | silent drift | 3 reports | +| U31 | `describe module ... with all` truncates at 10 000 lines without a warning | Describe omits or truncates | silent drift | 2 | +| U32 | published OData with no KEY, or a KEY on an attribute without a unique rule, passes check | OData/REST authoring | silent drift | 2 | +| U33 | `fix design-properties` may reorder or mis-map design properties (`'Make card': on` where the source began with `'Spacing bottom'`) | Widgets and native | silent drift | 2 documents | +| U34 | describe userrole drops `CheckSecurity: false` | Describe omits or truncates | silent drift | 1 | +| U35 | describe of a doc comment loses a markdown `* ` bullet | Describe omits or truncates | silent drift | 1 | +| U36 | describe javascript action keeps EXTRA CODE only as a comment; re-running `create or modify` overwrites a hand-restored `.js` | Describe omits or truncates | silent drift | 1 | +| U37 | sort on an inherited attribute is written against the specialization | Check/exec semantics | silent drift | 1 | +| U38 | set-mode `check` prints "Check passed!" after an earlier failure in the same invocation | Check/exec semantics | silent drift | 1 | +| U39 | `create external entities from M.Client into M` names entities after the entity set, not the entity type | OData/REST authoring | silent drift | 1 | +| U40 | `mxcli new --version 10.24.0` ships a layout that cannot build and still exits 0 | Marketplace and project setup | silent drift | 1 | +| U41 | check is not version-aware: it passes scripts that exec then refuses for a version reason | Mendix 9.x project support | usability | 2 | +| U42 | `mxcli describe module ` subcommand emits only the module and its roles | Describe omits or truncates | usability | 1 | +| U43 | `RETURNS Boolean AS $Closed` does not declare `$Closed`; both obvious forms are refused with confusing messages | Check/exec semantics | usability | 1 | +| U44 | `mxcli run --local` cannot run a Mendix 9 app | Mendix 9.x project support | usability | 1 | +| U45 | `alter settings runtime ( JavaVersion: '17' )` on a 9.x model prints `Unchanged model settings` | Mendix 9.x project support | usability | 1 | +| U46 | mxcli cannot create a native navigation profile | Widgets and native | usability | 1 | +| U47 | combo box bound to an association through the attribute key passes check, exec refuses | Widgets and native | usability | 1 | +| U48 | `marketplace update` leaves CE0463 widget errors, neither runs nor mentions `mx update-widgets`, and warns on a corrupt zip | Marketplace and project setup | usability | 1 | +| U49 | `marketplace update --to` refuses without `--force`, citing local edits nobody made | Marketplace and project setup | usability | 1 | +| U50 | no headless update for a marketplace module already in the project | Marketplace and project setup | usability | 1 | +| U51 | `marketplace download --output ` fails; only a file path works | Marketplace and project setup | usability | 1 | +| U52 | commands without `-p` silently pick a stray `.mpr` in the working directory | CLI usability | usability | 1 | +| U53 | System Java actions are unknown to mxcli | CLI usability | usability | 1 | + +Totals: 53 distinct defects — 24 blocks round-trip, 16 silent drift, 13 usability. + +## Describe output rejected by mxcli itself + +### U1 — describe prints a nanoflow data source/action without its argument mapping + +- **Command / document type:** `describe page` / `describe snippet` (native Repeater, nanoflow action) +- **Severity:** blocks round-trip +- **Verbatim error:** `widget 'repeater4': nanoflow data source NativeMobileForm.DS_GetInstanceNativePictureImage has no argument for parameter 'NativeInstanceItem' — Mendix rejects this with CE1571.` (also `nanoflow action ... (Action:) has no argument for parameter(s) ...`) +- **Seen:** FSM template: NativeMobile, NativeMobileForm, NativeMobilePart, NativeMobileRelated, NativeMobileAsset, NativeMobileTimeRegistration, NativeNote. — occurrences: 70 (bughunt; 7 native modules) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: describe emits the parameter mapping the source page holds. Actual: call printed with no arguments. Either the mapping is lost on describe (silent drift if check were lenient) or the page stored none (unverified which). + +**Repro** + +```mdl +-- pipelines/mendix-mpr/samples/fsm-template-9.24.2/native-bughunt/D4-nanoflow-datasource-no-args.mdl (EXCERPT, not isolated: needs a nanoflow, blocked by the 9.x entry) +-- describe prints: repeater repeater4 (DataSource: nanoflow NativeMobileForm.DS_GetInstanceNativePictureImage) +-- check then refuses its own output (CE1571, parameter NativeInstanceItem) +``` + +### U2 — describe emits `content { }` for native pluggable-widget child slots that are named `template` + +- **Command / document type:** `describe page`, `describe layout`, `describe snippet` (native Repeater, SafeAreaView) +- **Severity:** blocks round-trip +- **Verbatim error:** `` `content` is not a container of `repeater` — it declares: template [MDL-WIDGET26] `` and `` `content` is not a container of `safeareaview` ``; exec: `content needs a name` +- **Seen:** FSM template: NativeNote, NativeMobile (page Login), NativeMobileForm, NativeMobileRelated, NativeMobileAsset, NativeMobilePart, NativeMobileTimeRegistration. — occurrences: 35 (32 Repeater + 2 SafeAreaView in bughunt; 1 report in increment 3) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: describe names the slot as the widget declares it (`template`). Actual: `content`; after `widget init` the slot is `template content { }` and exec then refuses the layout with `declares no placeholder at all` (see the placeholder entry). + +**Repro** + +```mdl +-- pipelines/mendix-mpr/samples/fsm-template-9.24.2/native-bughunt/D2-page-repeater-content.mdl (needs com.mendix.widget.native.Repeater.mpk in widgets/ and `mxcli widget init`) +create module Repro; +create or modify persistent entity Repro.E ( Name: String(200) ); +create or modify page Repro.P1 (Title: 'P1', Layout: Atlas_Core.Atlas_Default) { + placeholder Main { + repeater repeater1 (DataSource: database from Repro.E) { + content { + container container2 ( ) { + } + } + } + } +} +``` + +### U3 — describe entity emits `index ( desc);` with no column + +- **Command / document type:** `describe entity` (also `describe module ... with all`) +- **Severity:** blocks round-trip +- **Verbatim error:** `statement N: index on unknown attribute 'desc'` (describe printed `index ( desc);`) +- **Seen:** FSM template: NativeTechnician, NativeReason, NativeMobileWarehouse, NativeMobile, NativeMobilePart, NativeMobileForm, NativeMobileTimeRegistration, NativeNote, NativeMobileAsset, NativeMobileRelated (every entity module except NativeModelCheck/NativeRestAPI). — occurrences: 25 (23 in bughunt, 1 each in increments 3 and 5) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: describe emits the indexed column(s), or omits the index. Actual: an index with an empty column list; check rejects it, and every flow, page and grant naming the entity cascades (NativeReason: 0 of 45 documents survive check). + +**Repro** + +```mdl +-- pipelines/mendix-mpr/samples/fsm-template-9.24.2/native-bughunt/D1-index-empty-column.mdl (isolated, fresh 9.24.2 app) +create module Repro; +create or modify persistent entity Repro.E ( + Name: String(200), + ChangedDate: AutoChangedDate +) +index ( desc); +``` + +### U6 — describe loses or never emits the data view a widget sat in + +- **Command / document type:** `describe page` +- **Severity:** blocks round-trip +- **Verbatim error:** `widget 'textBox1': Attribute 'Search' is bound but there is no enclosing data container providing entity context` (also datePicker1..4 `Start`/`End`); increment 2: `argument Task: $Task cannot be stored — widget actionButton17 is not inside a data view` [MDL-PAGEARG01] +- **Seen:** FSM template: NativeMobileTimeRegistration, NativeMobileAsset, NativeMobileRelated; genai-showcase-app (a ConversationalUI page). — occurrences: 12 (11 in bughunt, 1 in increment 2) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: the enclosing data container is emitted. Actual: widget emitted bare; check rejects. + +**Repro** (not reduced) + +Not reduced (describe any page whose input widgets sit in a data view the source holds; which side drops it is unverified). + +### U7 — Java/JS action type parameters are not expressible: describe emits a form check refuses or exec writes entity `.Entity` + +- **Command / document type:** `describe javascript action`, `describe java action`, then `check`/`exec` +- **Severity:** blocks round-trip +- **Verbatim error:** JS: `line 13427:67 mismatched input '<' expecting {FOLDER, '('}` (describe prints `Name(Entity: "Entity" not null, ...)`). Java: describe prints `MxObject: "Entity"` with no declaration; mxbuild: `Property 'Entity' of Java action parameter 'MxObject': The selected entity '.Entity' no longer exists.` Caller side: `Entity: entity ` is an entity-type parameter and mx check gives CE0115. +- **Seen:** mendix/genai-showcase-app (GenAICommons, 7 build errors); FSM template (NativeMobile, NativeNote). — occurrences: about 9 (D10 x1, increment 3 x1 JS, 7 build errors for Java in increment 2) +- **Toolkit workaround:** none +- **Existing ledger entry:** BUG-87 (describe drops the type-parameter name, related), BUG-61 (CALL JAVA ACTION with typed parameters, related) +- **Expected vs actual:** Expected: an object-of-type-parameter parameter has an MDL form that round-trips. Actual: none exists; `Entity: Entity` resolves to entity `.Entity`. + +**Repro** + +```sh +mxcli -p .mpr -c "describe java action GenAICommons.ChunkCollection_AddKnowledgeBaseChunk" # then exec the output, then mx check +# JS: describe a native JS action with a type parameter and `check` the output (not reduced) +``` + +### U10 — ContentParams parameters are described as `$obj/Attr` paths, which check refuses + +- **Command / document type:** `describe snippet`/`describe page` (dynamic text) +- **Severity:** blocks round-trip +- **Verbatim error:** `dynamictext "text42": ContentParams parameter {1} is written as "$currentObject/NotResolvedReason" — a template parameter is evaluated against the widget's own context object`; increment 2: `ContentParams parameter {1} is written as "$ChatContext/Topic"` [MDL-WIDGET24] +- **Seen:** FSM template: NativeMobileRelated; genai-showcase-app. — occurrences: 3 (bughunt 2, increment 2 1) +- **Toolkit workaround:** none +- **Existing ledger entry:** BUG-23 (the `$currentObject/` prefix resolving as a literal, related) +- **Expected vs actual:** Expected: describe emits a form check accepts. Actual: describe and check disagree. + +**Repro** (not reduced) + +Not reduced. + +### U11 — three more describe outputs check cannot parse: import-mapping `root (Wrapper)`, dynamic-text `decimalPrecision: -1`, `DecimalPrecision` -1 + +- **Command / document type:** `describe import mapping`, `describe snippet` +- **Severity:** blocks round-trip +- **Verbatim error:** `line 342:59 extraneous input '('`; `ContentParams: ({1} = ToolName format (decimalPrecision: -1))` -> `no viable alternative`; `DecimalPrecision must be a non-negative integer, got &{-1}` [MDL-WIDGET18] +- **Seen:** genai-showcase-app. — occurrences: 3 (increment 2, seen not isolated) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: describe output parses. Actual: rejected. + +**Repro** (not reduced) + +Not reduced; seen, not isolated. + +### U15 — Layout placeholders inside a pluggable widget's child slot are not seen: `declares no placeholders at all` + +- **Command / document type:** `describe layout`, then `exec`/`check` +- **Severity:** blocks round-trip +- **Verbatim error:** `layout 'ConversationalUI.Layout_MasterBase' declares no placeholders at all` [MDL081]; increment 3 after `widget init`: `declares no placeholder at all` +- **Seen:** genai-showcase-app (ConversationalUI); FSM template (NativeMobile layout with SafeAreaView). — occurrences: 2 (increment 2 and increment 3) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: placeholders nested in widget slots count. Actual: the layout is refused. + +**Repro** (not reduced) + +Not reduced (any layout whose placeholder sits inside a pluggable widget slot). + +### U16 — describe of Java action puts the parameter separator after the `--` comment + +- **Command / document type:** `describe java action` +- **Severity:** blocks round-trip +- **Verbatim error:** `Name: String not null -- This sets the Name of the tool...,` then check: `line 255:4 mismatched input 'Name' expecting ')'` +- **Seen:** genai-showcase-app (GenAICommons). — occurrences: 1 run (not counted) +- **Toolkit workaround:** `lib/fix-describe-output.py` +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: the comma precedes the comment. Actual: the comma lands inside the comment, so the next parameter is a syntax error. + +**Repro** + +```sh +mxcli -p .mpr -c "describe java action GenAICommons." > a.mdl && mxcli check a.mdl -p .mpr +``` + +### U17 — describe of a microflow emits a variable use check calls undeclared + +- **Command / document type:** `describe microflow`, then `exec` +- **Severity:** blocks round-trip +- **Verbatim error:** `microflow 'GenAICommons.Span_CalculateSumDurationUntilExecution' has validation errors: - variable 'SumDuration' is not declared` +- **Seen:** genai-showcase-app (GenAICommons). — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: output re-executes. Actual: refused. + +**Repro** (not reduced) + +Not reduced: `describe microflow GenAICommons.Span_CalculateSumDurationUntilExecution`, exec the output. + +### U18 — describe of a merge/decision cannot be re-created (#923): WARNING lines, layout drift + +- **Command / document type:** `describe microflow` +- **Severity:** blocks round-trip +- **Verbatim error:** `merge shared1: nothing joins it.` [MDL-FLOW02]; target describe drifts: `'@curve(from: (15, 0), to: (-30, 0))' -> "@caption 'empty'"` +- **Seen:** genai-showcase-app. — occurrences: 1 +- **Toolkit workaround:** `lib/compare-mdl.py` rates #923 warnings `not-expressible` +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: merges and decisions round-trip. Actual: warning plus layout drift. + +**Repro** (not reduced) + +Not reduced: describe any flow with a shared merge. + +### U19 — describe emits a selection data source naming a widget it did not emit + +- **Command / document type:** `describe page` +- **Severity:** blocks round-trip +- **Verbatim error:** `widget 'dataView1': selection DataSource references 'grid1' but no widget with that name exists on this page` +- **Seen:** FSM template: NativeModelCheck. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: the referenced widget is emitted. Actual: dangling reference. + +**Repro** (not reduced) + +Not reduced. + +### U20 — create refuses an enum split with more than 16 branches that describe produced + +- **Command / document type:** `describe microflow` then `check` +- **Severity:** blocks round-trip +- **Verbatim error:** `enum split has 28 branches; at most 16 branches are supported` (`NativeModelCheck_Process`) +- **Seen:** FSM template: NativeModelCheck. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: describe and create agree on the limit (or the limit is lifted). Actual: a Studio Pro-valid flow cannot be re-created. + +**Repro** (not reduced) + +Not reduced: a microflow with a 28-case enumeration split. + +### U21 — lint MDL-REST04 flags a valid authentication microflow + +- **Command / document type:** `check` (lint) +- **Severity:** blocks round-trip +- **Verbatim error:** `MDL-REST04: authentication microflow ... has parameter $Token, which Mendix does not supply to an authentication microflow ... (CE0336)` +- **Seen:** FSM template: NativeRestAPI. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: valid source passes. Actual: lint error on describe output of a Studio Pro-valid microflow. + +**Repro** (not reduced) + +Not reduced. + +## Describe omits or truncates + +### U5 — rules cannot be described or created + +- **Command / document type:** `describe rule`, `exec` of a decision that calls a rule +- **Severity:** blocks round-trip +- **Verbatim error:** `Unknown type: rule`; exec: `if condition calls 'PgVectorKnowledgeBase.Rule_DatabaseConfiguration_IsConfigured(...)', which is not a rule in this project` +- **Seen:** genai-showcase-app (PgVectorKnowledgeBase and others). — occurrences: 14 rules, 8 flows deferred (increment 2) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: rules are describable and creatable. Actual: refused; flows calling them cannot be created. + +**Repro** + +```sh +mxcli -p .mpr -c "show rules" # lists 14 +mxcli -p .mpr -c "describe rule PgVectorKnowledgeBase.Rule_DatabaseConfiguration_IsConfigured" +``` + +### U8 — message definitions cannot be expressed; export mappings built on them fail + +- **Command / document type:** `exec` of an export mapping +- **Severity:** blocks round-trip +- **Verbatim error:** `Error: export mapping GenAICommons.EM_Reference: message definition "GenAICommons.MD_Reference.Reference" not found — this project has no message definitions` (mxbuild: CE1613 `The selected export mapping '...' no longer exists`) +- **Seen:** genai-showcase-app (GenAICommons). — occurrences: 4 x CE1613 (increment 2) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: message definitions describable and creatable. Actual: no way to create one. + +**Repro** (not reduced) + +Not reduced: exec any export mapping built on a message definition. + +### U27 — `describe module ... with all` omits whole document types + +- **Command / document type:** `describe module with all` +- **Severity:** silent drift +- **Verbatim error:** No error. Output silently lacks nanoflows, snippets, JS actions, JSON structures, import/export mappings, task queues, image collections and some pages. +- **Seen:** genai-showcase-app (AgentCommons, 10 of 22 pages), FSM template (NativeNote 35 nanoflows, all 12 Native modules), ako/mxcli-banking (MyFirstModule.Images). — occurrences: 4 reports (increments 1, 2, 3 and bughunt; 10 of 22 pages missing in AgentCommons, 35 nanoflows in NativeNote) +- **Toolkit workaround:** `rehost-extract.sh` project-tree fill-in (recovered all 1142 FSM documents) +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: every describable document in the module. Actual: a subset; nothing says so. + +**Repro** + +```sh +mxcli -p .mpr -c "describe module AgentCommons with all" # compare against: mxcli project-tree --json, module AgentCommons +``` + +### U31 — `describe module ... with all` truncates at 10 000 lines without a warning + +- **Command / document type:** `mxcli -c "describe module M with all"` +- **Severity:** silent drift +- **Verbatim error:** None: output stops in the middle of a statement. +- **Seen:** genai-showcase-app (AgentCommons); FSM template (NativeMobile ends mid-statement). — occurrences: 2 (increment 2, bughunt) +- **Toolkit workaround:** `rehost-extract.sh` `describe_module()` drops the incomplete last statement and describes the rest one by one +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: whole module or an explicit truncation message. Actual: silent cut. + +**Repro** + +```sh +mxcli -p .mpr -c "describe module AgentCommons with all" | wc -l # 10000, last statement incomplete +``` + +### U34 — describe userrole drops `CheckSecurity: false` + +- **Command / document type:** `describe user role` +- **Severity:** silent drift +- **Verbatim error:** None: replaying the output re-creates the role with the default. +- **Seen:** ako/mxcli-banking (Banking). — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** BUG-144 (create user role writes no CheckSecurity, related) +- **Expected vs actual:** Expected: the property is emitted. Actual: silent security change on replay. + +**Repro** (not reduced) + +Not reduced: describe a role with CheckSecurity false, replay. + +### U35 — describe of a doc comment loses a markdown `* ` bullet + +- **Command / document type:** `describe` (documentation text) +- **Severity:** silent drift +- **Verbatim error:** None (the one drift row of 155 on the banking round trip). +- **Seen:** ako/mxcli-banking. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: text preserved. Actual: bullet lost. + +**Repro** (not reduced) + +Not reduced. + +### U36 — describe javascript action keeps EXTRA CODE only as a comment; re-running `create or modify` overwrites a hand-restored `.js` + +- **Command / document type:** `describe javascript action`, `create or modify javascript action` +- **Severity:** silent drift +- **Verbatim error:** None: the generated file lacks the source's EXTRA CODE import (describe prints `-- EXTRA CODE:` as a comment). +- **Seen:** FSM template (NativeNote). — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: EXTRA CODE round-trips and is not overwritten. Actual: lost on re-create. + +**Repro** (not reduced) + +Not reduced. + +### U42 — `mxcli describe module ` subcommand emits only the module and its roles + +- **Command / document type:** `mxcli describe module` +- **Severity:** usability +- **Verbatim error:** None. Help says "Describe a module (all contents)"; output is `mdl 1; create or modify module Banking; create or modify module role ...`, same for `--mdl 0` and `--format json`. +- **Seen:** ako/mxcli-banking (Banking). — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: all contents or help text that says otherwise. Actual: module and roles only. + +**Repro** + +```sh +mxcli describe module Banking -p .mpr # module + roles only +mxcli -p .mpr -c "describe module Banking with all" # the real emitter, documented only in `mxcli changelog` +``` + +## Check/exec semantics (atomicity, forward refs, check passes but exec refuses) + +### U9 — describe emits catalog order; exec and check refuse forward and cross-module references, so mutual recursion cannot be created + +- **Command / document type:** `describe module ... with all`, `check`, `exec` +- **Severity:** blocks round-trip +- **Verbatim error:** `MDL-ORDER01 ... references microflow Banking.SUB_SeedBillers before it is created (statement 145)` x14; `Refusing to execute: 14 unresolved reference(s). Nothing was written.`; `references microflow X before it is created (statement 240), and the project does not have it`; `child entity module not found: NativeMobilePart`; `microflow not found: NativeReason.` +- **Seen:** ako/mxcli-banking (Banking); genai-showcase-app (GenAICommons); FSM template (NativeMobileWarehouse, NativeModelCheck, NativeNote, NativeMobileTimeRegistration, NativeMobileForm, NativeMobilePart). — occurrences: 4 reports (increment 1: 14 x ORDER01; increment 2: statement 240; bughunt: 6 modules; increment 3) +- **Toolkit workaround:** `pipelines/mendix-mpr/lib/topo-order-mdl.py` (cannot order an entity whose after-commit microflow takes the entity) +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: emit in dependency order (mxcli knows `callers`/`callees`), and allow a create-stub-then-modify pass for cycles. Actual: output refused by its own check. + +**Repro** (not reduced) + +Not reduced: describe any module with a flow that calls a later flow; mutual recursion GenAICommons.Response_CallRecursively <-> ChatCompletions_WithHistory. + +### U12 — exec is not atomic after the pre-check passes + +- **Command / document type:** `exec` +- **Severity:** blocks round-trip +- **Verbatim error:** Increment 3: a widget-build error left 18 earlier statements written. Increment 4: the drop + create that `MDL-V1-REBUILD` recommends ran the `drop`, then failed on the `create`; mx check: 2 x CE1613. Bughunt: exec reached only the module level. +- **Seen:** FSM template (NativeNote, seam probe). — occurrences: 3 (increments 3 and 4, bughunt; 18 statements left written) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: all-or-nothing (`Refusing to execute ... Nothing was written.` is the right behaviour for the pre-check and should extend to write-time errors). Actual: partial writes; the dropped flow is gone. + +**Repro** + +```sh +# not reduced: any script where statement N fails at write time +create or modify microflow M.A () begin return; end; -- on a 9.x model this succeeds as a check, fails at exec +drop microflow M.A; +create or modify microflow M.A () begin return; end; +``` + +### U13 — a grant naming an association member is refused when it precedes the association + +- **Command / document type:** `describe module`, `exec` +- **Severity:** blocks round-trip +- **Verbatim error:** `Error: entity EchoConnector.Credentials has no member(s) Credentials_Configuration; grant only names members of the entity`; `entity NativeNote.NativeImage has no member(s) NativeImage_NativeNote` +- **Seen:** genai-showcase-app (EchoConnector); FSM template (NativeNote). — occurrences: 3 (increment 2, increment 3 note, bughunt) +- **Toolkit workaround:** topo-order-mdl.py makes grants their own units after the associations +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: describe emits grants after associations, or exec resolves later members. Actual: refused. + +**Repro** (not reduced) + +Not reduced: describe order of the EchoConnector module. + +### U22 — exec writes an import-mapping microflow with a null entity and the `.mpr` can no longer be loaded + +- **Command / document type:** `exec` (microflow with `import from mapping`) +- **Severity:** blocks round-trip +- **Verbatim error:** mx check: `System.ArgumentNullException: Value cannot be null. (Parameter 'value') at Mendix.Modeler.DataTypes.EntityType.set_EntityId` — "An error occurred when trying to set the 'Entity' property of a Object type in a Microflow with ID ...". A plain re-`create or modify` is refused with `MDL-V1-REBUILD`. +- **Seen:** genai-showcase-app (Embeddings_SemanticSearch). — occurrences: 1 +- **Toolkit workaround:** `drop microflow X;` plus the same `create or modify microflow X` in one script once the mapping exists +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: refuse the unresolved mapping. Actual: the project is written unloadable (data loss). + +**Repro** (not reduced) + +Not reduced: exec Embeddings_SemanticSearch.Book_CreateDatasetIfNeeded before IM_SemanticSearch_Books exists (cause is a hypothesis). + +### U23 — a microflow created before a Java action of the same name fails with a misleading error + +- **Command / document type:** `exec` +- **Severity:** blocks round-trip +- **Verbatim error:** `warning: could not look up Java action ... java action not found` then `Error: failed to create microflow: refusing to write unit ...: attribute reference not qualified` +- **Seen:** genai-showcase-app (GenAICommons). — occurrences: 1 +- **Toolkit workaround:** topo-order-mdl.py orders the Java/JS action first +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: a clear order error. Actual: an unrelated-looking attribute error. + +**Repro** (not reduced) + +Not reduced: exec GenAICommons.ConsumedKnowledgeBase_GetCollectionOptions (the flow) before the Java action of the same name. + +### U24 — exec's reference pre-check does not see an image collection created earlier in the same script + +- **Command / document type:** `exec` +- **Severity:** blocks round-trip +- **Verbatim error:** `unknown image collection "FSM_Specific.Material_Design_Icons"` (whole script refused) +- **Seen:** FSM template (NativeNote). — occurrences: 1 (plus 1 unconfirmed) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: earlier statements count. Actual: refused. A same-text `image collection not found` was seen against an empty scratch app (missing dependency, not re-tested). + +**Repro** (not reduced) + +Not reduced: `create image collection M.Icons ...` followed by a page that uses it, one script. + +### U28 — check accepts `change $E (...)` / `$E = empty` where `$E` comes from `retrieve ... limit 1` (a list under mdl 1); mx check then fails, and a splice does not clear it + +- **Command / document type:** `check`, `create or modify` +- **Severity:** silent drift +- **Verbatim error:** mx check: `CE1613 "The selected attribute 'List of M.Ent.Attr' no longer exists."` After changing `limit 1` to `first`, `create or modify` reports `spliced: 1 replaced` and describe shows `first`, but mx check keeps the 4 CE1613; only drop + create clears them. +- **Seen:** FSM template (seam probe, 10.24 copy). — occurrences: 4 x CE1613 (increment 4) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: check flags a list used as an object, and a spliced flow is rebuilt. Actual: both pass; the model keeps the stale reference. + +**Repro** + +```mdl +retrieve $E from M.Ent limit 1; +change $E ( Attr = 'x' ); +``` + +### U37 — sort on an inherited attribute is written against the specialization + +- **Command / document type:** `exec` (sort list operation) +- **Severity:** silent drift +- **Verbatim error:** mx check: `CE1613 "The selected attribute 'Embeddings_SemanticSearch.BookChunk.Similarity' no longer exists." at List operation activity 'Sort'` +- **Seen:** genai-showcase-app (Embeddings_SemanticSearch.DS_Query_GetBookChunks). — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: attribute qualified against the entity that owns it. Actual: wrong qualification; source and target describe are identical, so a describe-only comparison calls it identical. + +**Repro** + +```mdl +-- BookChunk extends GenAICommons.KnowledgeBaseChunk (Similarity is on the generalization) +$L = sort $BookChunkList by Similarity desc; +``` + +### U38 — set-mode `check` prints "Check passed!" after an earlier failure in the same invocation + +- **Command / document type:** `mxcli check` (set mode) +- **Severity:** silent drift +- **Verbatim error:** `Check passed!` printed after an earlier failure line. +- **Seen:** ako/mxcli-banking. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: a final verdict that reflects the failure. Actual: contradictory final line. + +**Repro** (not reduced) + +Not reduced. + +### U43 — `RETURNS Boolean AS $Closed` does not declare `$Closed`; both obvious forms are refused with confusing messages + +- **Command / document type:** `check` +- **Severity:** usability +- **Verbatim error:** `$Closed = true;` -> "a reassignment says 'set' under mdl 1"; `SET $Closed = true;` -> "variable 'Closed' is not declared" +- **Seen:** ako/mxcli-banking. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: the return variable is declared, or the message says `DECLARE $Closed Boolean = false;` is needed first. Actual: the two messages contradict. + +**Repro** + +```mdl +create or modify microflow M.F () returns Boolean as $Closed +begin + $Closed = true; + return $Closed; +end; +``` + +## Mendix 9.x project support + +### U14 — microflows and nanoflows cannot be created or altered in a 9.x project + +- **Command / document type:** `create microflow`, `create nanoflow`, `alter microflow` +- **Severity:** blocks round-trip +- **Verbatim error:** `Error: failed to create microflow: unit 91c57c83-...: this Mendix 9.24.2 project cannot store Microflows$Microflow.StableId (Mendix 10.2.0+); Microflows$SequenceFlow.CaseValues (Mendix 10.15.0+): the properties did not exist yet in this version, and Studio Pro cannot open a document that holds one.` Nanoflow: `CreateNanoflow: insert: ... cannot store Microflows$SequenceFlow.CaseValues (Mendix 10.15.0+)`. `alter microflow` adding an activity: `cannot store Microflows$SequenceFlow.CaseValues (Mendix 10.15.0+)`. +- **Seen:** FSM template 9.24.2 (every module with a flow; NativeRestAPI 4 of 4). — occurrences: 3 reports (increment 4 x2 including alter, bughunt: 100% of flows tried) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: the writer omits properties the version lacks (the refusal itself is correct and safe). Actual: no microflow or nanoflow can be written to 9.x, so no seam or round-trip work on the monolith can follow. + +**Repro** + +```mdl +-- pipelines/mendix-mpr/samples/fsm-template-9.24.2/native-bughunt/D3-nanoflow-9242-sequenceflow-casevalues.mdl (fresh `mxcli new Scratch --version 9.24.2 --theme none --skip-init`) +create module Repro; +create or modify microflow Repro.MF_Min () +begin + return; +end; +create or modify nanoflow Repro.NF_Min () +begin + return; +end; +``` + +### U41 — check is not version-aware: it passes scripts that exec then refuses for a version reason + +- **Command / document type:** `check` vs `exec` on a 9.x project +- **Severity:** usability +- **Verbatim error:** check: `Check passed!`; exec: the StableId/CaseValues refusal above. +- **Seen:** FSM template (seam probe). — occurrences: 2 (increment 4) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: check reports the version limit. Actual: late failure. + +**Repro** (not reduced) + +Same as the entry above: `mxcli -p <9.24.mpr> -c "create microflow M.X () begin return; end;"` + +### U44 — `mxcli run --local` cannot run a Mendix 9 app + +- **Command / document type:** `mxcli run --local` +- **Severity:** usability +- **Verbatim error:** mxbuild 9.24's Gradle 7.6: `Unsupported class file major version 65` (mxcli resolves JDK 21 even with JAVA_HOME set to 17) +- **Seen:** FSM template 9.24.2. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: a JDK the model version supports, or a clear message. Actual: a Gradle stack error; a 9.x model has no JavaVersion setting to steer it. + +**Repro** + +```sh +JAVA_HOME= mxcli run --local --db-type hsqldb -p <9.24 app> +``` + +### U45 — `alter settings runtime ( JavaVersion: '17' )` on a 9.x model prints `Unchanged model settings` + +- **Command / document type:** `alter settings` +- **Severity:** usability +- **Verbatim error:** `Unchanged model settings` (no hint the property does not exist in that version) +- **Seen:** FSM template 9.24.2. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: "property not available in Mendix 9.24.2". Actual: looks like a no-op success. + +**Repro** + +```mdl +alter settings runtime ( JavaVersion: '17' ); +``` + +## OData/REST authoring + +### U29 — published OData `expose ("Description")` stores the quotes in the attribute name + +- **Command / document type:** `create published odata service ... expose` +- **Severity:** silent drift +- **Verbatim error:** `CE1613 "The selected attribute 'AssetManagement.Asset."Description"' no longer exists."` (x4) +- **Seen:** AssetsApp slice (AssetManagement). — occurrences: 4 x CE1613 (increment 5; captures 24, 26) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: quotes are syntax, not part of the name. Actual: the stored attribute name includes quotes; check and exec pass. + +**Repro** (not reduced) + +Not reduced: quote a keyword-named attribute in an expose list. + +### U32 — published OData with no KEY, or a KEY on an attribute without a unique rule, passes check + +- **Command / document type:** `create published odata service` +- **Severity:** silent drift +- **Verbatim error:** mx check: `CE6585 "Published entity '...' must have a key defined."`; `CE6624 "Add a unique validation rule ..."` +- **Seen:** AssetsApp slice (AssetManagement). — occurrences: 2 (increment 5) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: check flags both. Actual: late mx check failure. + +**Repro** (not reduced) + +Not reduced: a published entity with no `KEY`; a `KEY` on an attribute with no unique validation rule. + +### U39 — `create external entities from M.Client into M` names entities after the entity set, not the entity type + +- **Command / document type:** `create external entities` +- **Severity:** silent drift +- **Verbatim error:** Later scripts fail with "entity does not exist" (`M.Parts` created, `M.Part` expected as Studio Pro does). +- **Seen:** AssetsApp slice (client of the Assets seam). — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: entity type name. Actual: entity set name. + +**Repro** (not reduced) + +Not reduced. + +## Widgets and native + +### U4 — legacy widgets (Forms$DataGrid) have no headless route to the React client; describe prints no data source or columns + +- **Command / document type:** `describe page` +- **Severity:** blocks round-trip +- **Verbatim error:** `-- Forms$DataGrid (grid1) -- NOT re-executable` (no datasource/columns/buttons) +- **Seen:** FSM template 10.24 copy (AssetsApp slice). — occurrences: 22 x CE0582 (increment 5) +- **Toolkit workaround:** `mxcli bson dump` + `slice-assets/tools/legacy-extract.py` + `gen-widgets.py` -> `alter page ... replace ... with { datagrid ... }` +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: enough detail to recreate as Data grid 2. Actual: placeholder comment only. + +**Repro** (not reduced) + +Not reduced: describe any page holding a legacy Data grid. + +### U26 — combo box in association mode bound to a param-bound snippet widget passes check and exec, then fails CE7007 + +- **Command / document type:** `create/alter snippet` (combobox) +- **Severity:** silent drift +- **Verbatim error:** `CE7007 "Selected value is not valid for entity 'Asset_Customer/Customer'. Please select the value again."` (x9) +- **Seen:** AssetsApp slice (Snippet in AssetManagement). — occurrences: 9 x CE7007 (increment 5) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: check demands a data container. Actual: wrapping in `dataview (DataSource: $Param)` clears it. + +**Repro** + +```mdl +-- replace a param-bound legacy reference selector in a snippet: +combobox x (Association: M.A, DataSource: database M.E, CaptionAttribute: Attr) -- no data container +``` + +### U30 — mxcli-written Data grid 2 fails CE0463 on a 10.x target until `fix widgets` + +- **Command / document type:** `create page` / `marketplace update`, then mxbuild +- **Severity:** silent drift +- **Verbatim error:** `CE0463 "The definition of this widget has changed. Update this widget..."` +- **Seen:** ako/mxcli-banking (10.24 target); FSM template 10.24 copies. — occurrences: 3 reports (increment 1: 15 units on 10.24; increment 4: 27 x; increment 5) +- **Toolkit workaround:** `mxcli fix widgets -p .mpr` after any page exec on a 10.x target +- **Existing ledger entry:** BUG-101 (Gallery CE0463, related) +- **Expected vs actual:** Expected: widget written against the definition the project carries (DataWidgets 2.31.0 / 2.5.6 on 10.24). Actual: newer definition. + +**Repro** (not reduced) + +Not reduced: `create page` with a datagrid on a `mxcli new --version 10.24.0` project, then `mx check`. + +### U33 — `fix design-properties` may reorder or mis-map design properties (`'Make card': on` where the source began with `'Spacing bottom'`) + +- **Command / document type:** `mxcli fix design-properties` +- **Severity:** silent drift +- **Verbatim error:** None. +- **Seen:** AssetsApp slice (AssetManagement.Snippet_Asset_Notes, WarehouseManagement.Warehouse_View). — occurrences: 2 documents (unverified) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: source order and values kept. Actual: possibly changed. + +**Repro** (not reduced) + +Not reduced; unverified. + +### U46 — mxcli cannot create a native navigation profile + +- **Command / document type:** `create or replace navigation NativePhone` +- **Severity:** usability +- **Verbatim error:** "a web profile's home is a page or a microflow" +- **Seen:** FSM template (NativeNote slice). — occurrences: 1 +- **Toolkit workaround:** `mx create-project` from the Blank Native starter +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: a native profile. Actual: refused; a native slice needs the Blank Native Mobile App starter (Marketplace 109511) via `mx create-project`. + +**Repro** + +```mdl +create or replace navigation NativePhone ( ... ); -- not reduced +``` + +### U47 — combo box bound to an association through the attribute key passes check, exec refuses + +- **Command / document type:** `alter snippet ... replace ... with { combobox }` +- **Severity:** usability +- **Verbatim error:** check: `Check passed!`; exec: `property attributeEnumeration binds Asset_Customer, which is an association ... not an attribute` +- **Seen:** AssetsApp slice. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: check refuses. Actual: only exec does (exec is correct). + +**Repro** + +```mdl +alter snippet M.S { replace with { + combobox x (Attribute: $Asset.Asset_Customer, CaptionAttribute: CustomerManagement.Customer.CustomerName) +} }; +``` + +## Marketplace and project setup + +### U25 — `marketplace install --file` of a 10.x package into 11.x does not warn that its pages will fail + +- **Command / document type:** `marketplace install --file` +- **Severity:** silent drift +- **Verbatim error:** `CE0582` on 22 legacy-widget pages after install +- **Seen:** AssetsApp slice (11.13.0). — occurrences: 22 x CE0582 (increment 5) +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: a warning at install. Actual: silent until mxbuild. + +**Repro** (not reduced) + +Not reduced: install a 10.x package holding legacy widgets into an 11.x app. + +### U40 — `mxcli new --version 10.24.0` ships a layout that cannot build and still exits 0 + +- **Command / document type:** `mxcli new --version 10.24.0` +- **Severity:** silent drift +- **Verbatim error:** mxbuild step 6/7 ends `BUILD FAILED`; `new` prints "Project created" and exits 0 (`App_Default` uses Atlas design properties `Flex container`, `Align items Y`, `Disable row wrap`, `Grow / shrink (self)` unknown to the bundled Atlas_Core 3.17.0) +- **Seen:** scratch app on 10.24.0 (decompose fixture). — occurrences: 1 +- **Toolkit workaround:** `ALTER PAGES SET LAYOUT = Atlas_Core.Atlas_Default WHERE LAYOUT = MyFirstModule.App_Default; DROP LAYOUT MyFirstModule.App_Default;` +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: non-zero exit or a matching layout. Actual: success message on a project that cannot build (fine on 11.13.0). + +**Repro** + +```sh +mxcli new Scratch --version 10.24.0 --theme none --skip-init +``` + +### U48 — `marketplace update` leaves CE0463 widget errors, neither runs nor mentions `mx update-widgets`, and warns on a corrupt zip + +- **Command / document type:** `marketplace update` +- **Severity:** usability +- **Verbatim error:** `skipping com.mendix.widget.web.PopupMenu.mpk: package.xml: zip: checksum error`; 27 x CE0463 afterwards +- **Seen:** FSM template (seam probe, 10.24 copy). — occurrences: 1 (27 x CE0463) +- **Toolkit workaround:** `mx update-widgets ` +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: update widgets or say to. Actual: silent. + +**Repro** (not reduced) + +Not reduced. + +### U49 — `marketplace update --to` refuses without `--force`, citing local edits nobody made + +- **Command / document type:** `marketplace update 109515 --to 5.2.1` +- **Severity:** usability +- **Verbatim error:** refusal reporting local edits in JavaScript actions `Wait`, `Vibrate` (likely conversion noise) +- **Seen:** FSM template 10.24 copy. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: no phantom diff. Actual: forced override needed. + +**Repro** (not reduced) + +Not reduced: run on a freshly `mx convert`ed 10.24 copy. + +### U50 — no headless update for a marketplace module already in the project + +- **Command / document type:** `marketplace install`, `mxcli new` +- **Severity:** usability +- **Verbatim error:** `marketplace install` of a present module is "NOT done automatically"; `mxcli new --version 11.12.0` ships Administration 4.3.2 while the source has 4.5.0 +- **Seen:** genai-showcase-app. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: an update path. Actual: hand-off to Studio Pro. + +**Repro** + +```sh +mxcli new X --version 11.12.0 && mxcli marketplace install --version 4.5.0 -p X/X.mpr +``` + +### U51 — `marketplace download --output ` fails; only a file path works + +- **Command / document type:** `marketplace download` +- **Severity:** usability +- **Verbatim error:** `rename ... file exists` +- **Seen:** ako/mxcli-banking. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: a directory is accepted, or a clear message. Actual: rename error. + +**Repro** + +```sh +mxcli marketplace download --output +``` + +## CLI usability + +### U52 — commands without `-p` silently pick a stray `.mpr` in the working directory + +- **Command / document type:** any command without `-p` +- **Severity:** usability +- **Verbatim error:** Only a `Using project: ...` notice. +- **Seen:** FSM template (increment 3). — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: refuse or ask. Actual: acts on whichever `.mpr` is present. + +**Repro** + +```sh +cd && mxcli describe module M +``` + +### U53 — System Java actions are unknown to mxcli + +- **Command / document type:** `exec` of a flow calling `System.VerifyPassword` +- **Severity:** usability +- **Verbatim error:** `warning: could not look up Java action System.VerifyPassword` +- **Seen:** genai-showcase-app. — occurrences: 1 +- **Toolkit workaround:** none +- **Existing ledger entry:** none found +- **Expected vs actual:** Expected: system actions known. Actual: warning on a valid call. + +**Repro** (not reduced) + +Not reduced. + +## Not mxcli (Mendix vendor) + +Reported here only so maintainers do not chase them; report to Mendix. + +- `mx module-import` refuses a package built with 9.x in 11.x: `The package could not be imported, because it was created with version 9.24.2 of Mendix Studio Pro. Please open it in any version in the range from 10.21.0 to 10.24.99 first.` (exit 117; surfaced through `mxcli marketplace install 220472 --version 3.4.1`; genai-showcase-app). +- `mx update-widgets fsm.mpr` with a relative path: `System.ArgumentNullException: Value cannot be null. (Parameter 'path')`; an absolute path works (FSM template, 10.24). +- `mx create-project` 11.13 with a relative `--output-dir`: `System.UriFormatException` (FSM template, NativeNote slice). diff --git a/contrib/inbox/2026-10-12-decompose-rehost-increment-1-mxcli-findings.md b/contrib/inbox/2026-10-12-decompose-rehost-increment-1-mxcli-findings.md new file mode 100644 index 00000000..db07db2e --- /dev/null +++ b/contrib/inbox/2026-10-12-decompose-rehost-increment-1-mxcli-findings.md @@ -0,0 +1,30 @@ +**From:** toolkit lab runs for `process/decompose-and-rehost-plan-2026-10-12.md`, increments 1a (Rehost, ako/mxcli-banking 11.13.0) and 1b (Decompose fixture, Mendix 10.24.0) +**Date:** 2026-10-12 +**Kind:** bug / learning candidates +**Field evidence:** verbatim captures in the lab trees (`lab/rehost/captures`, `lab/decompose/captures`, not shipped); each item below quotes the message seen +**Proposed target:** `bug-logs/mxcli-bugs.md` (items 1–8), `skills/learned-mdl-preflight.md` (items 9–11; item 9 already promoted in the same commit) + +mxcli v0.26.0 (2026-10-11), Linux cloud container, no Studio Pro. + +## mxcli defects / doc gaps (report upstream to mendixlabs/mxcli) + +1. **`mxcli describe module ` subcommand emits only the module and its roles.** Help text says "Describe a module (all contents)". Output is `mdl 1; create or modify module Banking; create or modify module role …` and nothing else, for `--mdl 0` and `--format json` too. The whole-module emitter is the MDL statement `describe module with all`, reachable only via `-c`, and documented only in `mxcli changelog`. +2. **`describe module … with all` emits flows in catalog order, and `mxcli check` refuses its own output**: 14× `MDL-ORDER01 … references microflow Banking.SUB_SeedBillers before it is created (statement 145)`; `exec` then `Refusing to execute: 14 unresolved reference(s). Nothing was written.` (correct refusal). mxcli knows the call graph (`callers`/`callees`) and could emit in call order. Workaround shipped: `pipelines/mendix-mpr/lib/topo-order-mdl.py`. +3. **`with all` omits image collections** — `MyFirstModule.Images` is `describable` in `project-tree --json` and `describe imagecollection` emits it, but the module-level emitter leaves it out. +4. **`describe userrole` drops `CheckSecurity: false`** — replaying the output re-creates the role with the default, a silent security change. +5. **`describe` of a doc comment loses a markdown `* ` bullet** (the one drift row of 155 on the banking round trip). +6. **Set-mode `check` prints "Check passed!" after an earlier failure in the same invocation.** +7. **`mxcli new --version 10.24.0` ships a layout that cannot build**: `App_Default` uses Atlas design properties (`Flex container`, `Align items Y`, `Disable row wrap`, `Grow / shrink (self)`) that the bundled Atlas_Core 3.17.0 does not know; mxbuild in step 6/7 ends `BUILD FAILED` but `new` still prints "Project created" and exits 0. Workaround: `ALTER PAGES SET LAYOUT = Atlas_Core.Atlas_Default WHERE LAYOUT = MyFirstModule.App_Default; DROP LAYOUT MyFirstModule.App_Default;`. Fine on 11.13.0. +8. **CE0463 on every mxcli-written DATAGRID on a 10.24 target** ("The definition of this widget has changed. Update this widget…"): mxcli writes Data grid 2 against a newer DataWidgets definition than the 2.31.0 the 10.24 scaffold carries. `mxcli fix widgets -p .mpr` resolves it (15 units changed, MPR v2 preserved). Rule for 10.x targets: after any page exec, `fix widgets` before mxbuild. `bin/bug-lookup.sh CE0463` currently returns nothing. + +## MDL learnings (preflight candidates) + +9. **CRLF inside string literals is load-bearing.** Studio Pro stores annotation text with `\r\n`; `describe` emits it verbatim; a tool that reads the file in text mode and writes `\n` changes the flow, and `check` says `this change cannot be spliced into the stored flow: the free annotations change … [MDL-V1-REBUILD]`. Byte-preserving I/O (`newline=''`) only. +10. **`RETURNS Boolean AS $Closed` does not declare `$Closed`.** `$Closed = true;` → "a reassignment says 'set' under mdl 1"; `SET $Closed = true;` alone → "variable 'Closed' is not declared". An aggregate or a retrieve declares; a plain return variable needs `DECLARE $Closed Boolean = false;` first. +11. **`LOG INFO NODE 'Util' $Message;` is not grammar.** The message is a template: `LOG INFO NODE 'Util' '{1}' WITH ({1} = $Message);`, and a non-String argument must be wrapped (`toString($Count)`). + +## Toolkit items (ours, not mxcli's) + +12. `project-bin/app-facts.sh` hard-requires the `sqlite3` CLI (`command -v sqlite3 || exit 2`) for seven one-column guard queries while the facts come from Python's `sqlite3` module. The CLI is not installable in the cloud container (`apt-get`: `Unable to locate package sqlite3`). Route the guards through `$PY`. +13. `project-bin/app-facts.sh` joins `refs` to `objects` by qualified name, so attribute-level refs (`member`, `xpath`) never reach `dependencies.json`; `bin/cut-score.sh`'s `member`/`xpath` lanes cannot fire until the producer carries them. +14. `mxcli marketplace download --output` takes a file path; a directory fails with `rename … file exists`. diff --git a/contrib/inbox/2026-10-12-increment-3-nativenote-findings.md b/contrib/inbox/2026-10-12-increment-3-nativenote-findings.md new file mode 100644 index 00000000..5d52c9bd --- /dev/null +++ b/contrib/inbox/2026-10-12-increment-3-nativenote-findings.md @@ -0,0 +1,28 @@ +# mxcli findings — increment 3, NativeNote slice (FSM 9.24.2 -> Mendix 11.13.0), 2026-10-12 + +Source: `pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-nativenote/` (captures, `mdl/edges-*.mdl`, +`cut-ledger.tsv`). mxcli v0.26.0, Linux cloud container, no Studio Pro. Unreviewed inbox drop. + +- `describe javascript action` with a type parameter emits `Name(Entity: "Entity" not null, …)`, + which `check` refuses (`mismatched input '<' expecting {FOLDER, '('}`). An *object of type parameter* + parameter is not expressible: `Entity: Entity` resolves to entity `.Entity`; `Entity: entity ` + is an entity-type parameter and the caller then fails mx check CE0115. +- `describe layout` of a native layout with SafeAreaView emits `content { … }`; exec: `content needs a + name`. After `mxcli widget init` the slot is `template content { … }`, and exec refuses the layout: + `declares no placeholder at all` — placeholders inside a pluggable widget's child slot are not seen. +- exec's reference pre-check does not see an image collection created earlier in the same script + (`unknown image collection "FSM_Specific.Material_Design_Icons"`), refusing the whole script. +- exec is not atomic after the pre-check: a widget-build error left 18 earlier statements written. +- Re-running `create or modify javascript action` overwrites a hand-restored `.js`; the generated file + lacks the source's EXTRA CODE import, which describe emits only as a `-- EXTRA CODE:` comment. +- mxcli cannot create a native navigation profile (`create or replace navigation NativePhone` → "a web + profile's home is a page or a microflow"); a native slice needs the Blank Native Mobile App starter + (Marketplace 109511) via `mx create-project` instead of `mxcli new`. +- Commands without `-p` silently pick a stray `.mpr` in the working directory (`Using project: …`). +- `describe module … with all` omits all nanoflows (35 in NativeNote) — as in increment 2. +- Observed, untested: `describe entity` emitted `index ( desc);` (empty column) for + `NativeTechnician.NativeTechnician`. +- Not mxcli: `mx create-project` 11.13 with a relative `--output-dir` → `System.UriFormatException`. +- Toolkit: `pipelines/mendix-mpr/lib/topo-order-mdl.py` cannot order an entity whose after-commit + microflow takes the entity, nor an entity grant naming an association created later — candidates + for the extractor once its current run is finished. diff --git a/contrib/inbox/2026-10-12-increment-4-seam-probe-findings.md b/contrib/inbox/2026-10-12-increment-4-seam-probe-findings.md new file mode 100644 index 00000000..7a38b650 --- /dev/null +++ b/contrib/inbox/2026-10-12-increment-4-seam-probe-findings.md @@ -0,0 +1,42 @@ +# mxcli and runtime findings — increment 4, seam probe (FSM monolith -> NativeNote slice, Mendix 11.13.0), 2026-10-12 + +Source: `pipelines/mendix-mpr/samples/fsm-template-9.24.2/seam-probe/` (captures, `mdl/`, FINDINGS.md). +mxcli v0.26.0, Linux cloud container, no Studio Pro. Unreviewed inbox drop. + +- `mxcli run --local` cannot run a Mendix 9 app. It resolves JDK 21 even with JAVA_HOME set to 17, + and mxbuild 9.24's Gradle 7.6 fails: `Unsupported class file major version 65`. A 9.x model has no + JavaVersion setting to steer it. Repro: `JAVA_HOME= mxcli run --local --db-type hsqldb -p <9.24 app>`. +- A microflow cannot be created in a 9.x project: + `this Mendix 9.24.2 project cannot store Microflows$Microflow.StableId (Mendix 10.2.0+); Microflows$SequenceFlow.CaseValues (Mendix 10.15.0+)`. + Repro: `mxcli -p <9.24.mpr> -c "create microflow M.X () begin return; end;"`. Blocks any monolith-side + seam authoring on 9.x. +- `alter microflow` that adds an activity in a 9.x project is refused: `cannot store Microflows$SequenceFlow.CaseValues (Mendix 10.15.0+)`. +- `mxcli check` passes scripts that exec then refuses for a version reason, and exec is not atomic. The + drop + create that `MDL-V1-REBUILD` recommends ran the `drop` and then failed on the `create`. The flow + was deleted, and `mx check` reported 2 × CE1613. +- `mxcli check` accepts `change $E (…)` and `$E = empty` when `$E` comes from `retrieve … limit 1`, which + is a list under `mdl 1`. `mx check` then reports + `CE1613 "The selected attribute 'List of M.Ent.Attr' no longer exists."` (4 ×). +- After `limit 1` is changed to `first`, `create or modify` reports `spliced: 1 replaced` and describe + shows `first`, but `mx check` keeps the 4 CE1613. Only drop + create clears them. +- `alter settings runtime ( JavaVersion: '17' )` on a 9.x model prints `Unchanged model settings`. It + does not say the property does not exist in that version. +- `marketplace update` leaves 27 × CE0463 (widget definitions changed) and does not run or mention + `mx update-widgets`. It also warns + `skipping com.mendix.widget.web.PopupMenu.mpk: package.xml: zip: checksum error`. +- Runtime: the Mendix 9.24.2 runtime boots headless on Linux. + - It needs Java 11; 17 is refused with `Incompatible java version: 17.0.20.1 … Java 11 is required.` + - The launcher needs `MX_INSTALL_PATH`, or it fails with `Error finding mx install path.` + - Every constant must be passed in `update_configuration`, or it fails on `MendixSSO.TokenEncryptionKey`. + - Start takes 10 s on an empty HSQLDB (2378 DDL commands). + - mxbuild 9.24 builds with `--java-home=`. +- Runtime: a REST call from 10.24 to 11.13 works with nothing tied to a Mendix version. + `create published rest service` with primitive microflow parameters, called by + `call rest service post … returns String`, needed no mappings. +- Converted 10.24 copy: getting it to boot needed one-time fixes nobody had measured: + - 2 marketplace updates; + - `mx update-widgets`; + - 1 XPath (CE0161 on a widget mxcli cannot author); + - JavaVersion 21; + - custom index.html removed; + - 3 Java edits: `final` generated parameters in DeepClone, and `rollbackTransAction` → `rollbackTransaction`. diff --git a/contrib/inbox/2026-10-12-increment-5-web-assets-findings.md b/contrib/inbox/2026-10-12-increment-5-web-assets-findings.md new file mode 100644 index 00000000..a6c3d251 --- /dev/null +++ b/contrib/inbox/2026-10-12-increment-5-web-assets-findings.md @@ -0,0 +1,34 @@ +# mxcli findings — increment 5, web slice Assets & Parts (FSM 9.24.2 -> AssetsApp 11.13.0, monolith on 10.24), 2026-10-12 + +Source: `pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-assets/` (captures, `mdl/`, `tools/`). mxcli v0.26.0, +Linux cloud container, no Studio Pro. Unreviewed inbox drop. + +- `mxcli check` passes a combo box bound to an association through the attribute key, exec refuses it: + `combobox x (Attribute: $Asset.Asset_Customer, CaptionAttribute: CustomerManagement.Customer.CustomerName)` -> + check `Check passed!`, exec `property attributeEnumeration binds Asset_Customer, which is an association … not an attribute`. + Repro: that line in `alter snippet M.S { replace with { … } };` (captures/13). +- A combo box in association mode (`Association: M.A, DataSource: database M.E, CaptionAttribute: Attr`) on a widget bound + straight to a snippet parameter (no data container) passes check and exec; `mx check` then gives + `CE7007 "Selected value is not valid for entity 'Asset_Customer/Customer'. Please select the value again."` (x9). + Wrapping it in `dataview (DataSource: $Param)` clears it. Repro: replace a param-bound legacy reference selector. +- Published OData `expose ("Description")` stores the quotes: `CE1613 "The selected attribute 'AssetManagement.Asset."Description"' no longer exists."` (x4). + Repro: any keyword-named attribute quoted in an expose list (captures/24, 26). +- Published OData with no `KEY` passes `mxcli check`; `mx check` then `CE6585 "Published entity '…' must have a key defined."`. + A `KEY` on an attribute without a unique validation rule also passes check, then `CE6624 "Add a unique validation rule …"`. +- `create external entities from M.Client into M` names the entities after the entity set (`M.Parts`), not the entity type + (`M.Part`) as Studio Pro does; scripts written against the type name fail with "entity does not exist". +- `describe entity` emits `index ( desc);` with no column (`NativeMobileAsset.NativeAsset`, FSM 10.24 copy) — confirms + increment 3's "possible" item. +- Legacy widgets have no headless conversion route to the React client (the only client on Mendix 11): `describe` prints + `-- Forms$DataGrid (grid1) -- NOT re-executable` with no datasource/columns/buttons, and `marketplace install --file` of a + 10.x package into 11.x does not warn that its pages will fail `CE0582` (x22 here). Workaround used: `mxcli bson dump` + + `slice-assets/tools/legacy-extract.py` + `gen-widgets.py` -> `alter page … replace … with { datagrid … }`. +- A Data grid 2 written by `create page` on a 10.24 project (DataWidgets 2.5.6) fails `CE0463` until `mxcli fix widgets`. +- `marketplace update 109515 --to 5.2.1` on a freshly `mx convert`ed 10.24 copy refuses without `--force`, reporting local edits + in JavaScript actions (`Wait`, `Vibrate`) nobody made (likely conversion noise). +- Vendor (Mendix `mx` 10.24, not mxcli): `mx update-widgets fsm.mpr` (relative) -> `System.ArgumentNullException: Value cannot be null. (Parameter 'path')`; + absolute path works. Same class as `mx create-project` relative `--output-dir`. +- Possible, unverified: `mxcli fix design-properties` left `'Make card': on` where the source began with `'Spacing bottom'` + (`AssetManagement.Snippet_Asset_Notes`, `WarehouseManagement.Warehouse_View`) — reorder or wrong mapping. +- Measured, not a defect: a local -> external association (`TicketManagement.Ticket` -> external `AssetsSeam.Assets`) passes + `mx check` 0 on 10.24 (captures/35) — so monolith-owned associations into a slice's entities are not a platform blocker. diff --git a/contrib/inbox/2026-10-12-increment-6-core-services-findings.md b/contrib/inbox/2026-10-12-increment-6-core-services-findings.md new file mode 100644 index 00000000..ed1e69fc --- /dev/null +++ b/contrib/inbox/2026-10-12-increment-6-core-services-findings.md @@ -0,0 +1,42 @@ +# Increment 6 (Decompose, core services) — new mxcli 0.26.0 defects + +Source: FSM template field run, `pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-assets-core/` (captures cited). +Unreviewed inbox drop. Triage into `bug-logs/mxcli-bugs.md`. + +1. **`check --references` ignores attributes added earlier in the same script.** `alter entity M.E add attribute P: + AutoNumber default 1;`, then a microflow `retrieve $X from M.E where [P = $v] first;`, then `mxcli check s.mdl -p app.mpr + --references` → `the constraint names "P", which is neither an attribute nor an association of it — mxbuild rejects + the constraint (CE0161 …)`. exec and `mx check` are fine (captures 01, 02). +2. **`drop entity` silently removes cross-module associations to the entity, and their member rights elsewhere.** Repro: + entity `B.T`, association `A.X_T from A.X to B.T` (declared in A), `drop entity B.T`. The drop prints only `Dropped + entity: B.T`, and then `describe association A.X_T` gives `Error: association not found`. The member rights on `A.X`'s + access rules go too (capture 05; `slice-core-external.mdl` header). +3. **A dropped owner-Both association leaves a hidden MemberAccess on the other entity's access rules.** Repro: + association `C.S_A from C.S to A.A owner Both`, `drop association C.S_A` → `mx check` `CE1613 "The selected + association 'C.S_A' no longer exists." at Access rule of entity 'A.A'`. `describe entity A.A` does not show the + member, and `revoke all on entity A.A from R;` + `grant …` does not remove it. Workaround: create an association + of that name from a throw-away entity in module C, then drop it (`slice-core-fix2.mdl`, capture 12). +4. **Consumed OData: a headers microflow given as `ConfigurationMicroflow:` is CE6808 on 11.13, and the slot cannot be + cleared.** `create consumed odata service … ConfigurationMicroflow: microflow M.Headers` (returns `List of + System.HttpHeader`) passes on 10.24 and gives `CE6808 "The microflow that configures a consumed OData service + should return a System.ConsumedODataConfiguration"` on 11.13. After that: `alter … set (HeadersMicroflow: …)` adds + the headers slot and keeps the configuration slot; `create or modify` with only `HeadersMicroflow:` → + `Unchanged OData client`; a real configuration microflow → `CE1135` (URL in constant and microflow), and + `ServiceUrl` cannot be removed (CE6825 guard). Only `drop consumed odata service` + create works (capture 12). +5. **`set (ConfigurationMicroflow: none)` stores a microflow named `none`** (also `empty`). `describe` then shows + `ConfigurationMicroflow: microflow none`, and `mx check` 11.13 crashes with a stack trace ending `at + Mendix.MxToolset.MxToolset.ProcessArguments(String[] commandLineArgs) in Mendix.MxToolset\MxToolset.cs:line 113`. +6. **`check` passes, exec refuses the same statement (MDL-V1-REBUILD), mid-script.** `create or modify microflow + AssetManagement.Asset_DeleteRosters` (body `retrieve $RosterList …; delete $RosterList` replaced by a call) → `check`: + `Check passed!`; exec: `this change cannot be spliced into the stored flow: alter microflow …: replace $RosterList: + $RosterList is still used by: delete $RosterList refresh`. `check` did flag 5 other flows of the same script for + the same rule. Earlier statements stay applied (capture 07). +7. **`create external entities` creates navigation associations only within the target module, silently.** Creating + `Ticket` into module T and `MonoAsset` into module C gives no `Ticket_Asset` and no warning. Creating the same entity + set into a second module passes mxcli and fails `mx check` `CE6609 "External entity with the same remote source + already exists."` (captures 05, 08, 09). +8. **`describe consumed odata service` shows an absolute local path:** `MetadataUrl: 'core-metadata.xml'` describes as + `MetadataUrl: 'file:////web-assets/AssetsApp/core-metadata.xml'`. It is unclear whether the model stores it; if + it does, it leaks a machine path to Team Server. + +Confirmed again: increment 5 defect 1 (combo box `Attribute:` on an association passes `check`, exec refuses), capture 07. diff --git a/contrib/inbox/2026-10-12-increment-7-strangled-findings.md b/contrib/inbox/2026-10-12-increment-7-strangled-findings.md new file mode 100644 index 00000000..fb16b372 --- /dev/null +++ b/contrib/inbox/2026-10-12-increment-7-strangled-findings.md @@ -0,0 +1,39 @@ +# Increment 7 (Decompose, strangled) — new mxcli 0.26.0 findings + +Source: Siemens FSM template, increment 7 (`pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-assets-strangled/`). +Monolith FSM 10.24 copy (MPR v1), slice AssetsApp 11.13 (MPR v2). New defects only; increment 6 #2/#3 confirmed again. + +1. **`move entity` leaves qualified names inside XPath/expression text unchanged.** By-name references are rewritten + ("Updated references in 38 document(s)"), but 62 strings in 24 units still named `AssetManagement.Asset…` + (e.g. `[TicketManagement.Ticket_Asset/AssetManagement.Asset/AssetManagement.Asset_Technician_Preferred/…]`). + Repro: a flow with `retrieve … where [X.R/M.E/Attr = $v]`; `move entity M.E to N;`; `describe` the flow → still `M.E`; + `mx check` → CE1613/CE0161 once `M` is dropped. Workaround: BSON string rewrite (`tools/bson-rename.py`). +2. **Published OData: an inherited attribute of a specialization is stored under the specialization.** `publish entity + Part.UsedPart … expose (PublicID (KEY), SystemID …)` (both on `Part.SelectedPart`) → `[CE1613] "The selected attribute + 'Part.UsedPart.PublicID' no longer exists." at Published attribute PublicID from entity Part.UsedPart`. The qualified + form `Part.SelectedPart.PublicID as 'PublicID'` is a parse error (`mismatched input '.' expecting {',', ')'}`). + Workaround: an own AutoNumber key on each specialization. +3. **`alter snippet/page … { drop w; }` cannot find a widget nested inside a legacy widget**, although `describe` lists it: + `Error: failed to resolve snippet target: widget "referenceSetSelector2" not found` + (`Technician.Snippet_Technician_DataTabs`). The whole script stops there. Workaround: BSON array-element removal. +4. **`grant read (…) on entity E to R where [xpath]` adds a second access rule when R already has one**; the old rule + (and the CE1613 it carried) stays. `describe entity` afterwards shows two `grant … to CustomerManagement.ViewMy` lines. + Expected: replace, or refuse with a hint to `revoke` first. +5. **A consumed OData service's cached `$metadata` cannot be refreshed.** After the metadata file changed, + `create or modify external entities from S into M entities (…)` → `4 updated`, but no new attribute appears. + No refresh statement exists. Workaround: `drop consumed odata service` + `create` (external entities survive). +6. **describe → drop + create of an unchanged microflow broke a decision.** `Scheduler.PlannerContext_GetWorkOrders` + (316 lines): afterwards `[CE0117] "Error(s) in expression." at Decision 'User is SDE?'`; the described decision line is + byte-identical before and after. Restored the original unit. +7. **describe output not re-executable:** `Synchronization.OfflineTicketAction_Validate` → + `microflow returns Boolean but not all code paths have a return statement [MDL003]` (case + merges). +8. **`mxcli oql --json` returns numbers as strings** (`[{"Amount":"10"}]`), so `=== 10` fails in journeys. + +Platform behaviour worth a learned-* line (not mxcli): +- a local entity's XPath cannot compare its local → external association with an object: `[TicketManagement.Ticket_Asset = $Asset]` + → CE0161; nor can an external entity be retrieved by its reverse association (`[M.Inventory_Part = $Inventory]` → CE0161); +- an AutoNumber key on a page-created entity fails at commit without read access: `Failed to commit: Value of auto-number + attribute 'PublicID' not available, could be caused by insufficient access rights to serialize the mendix object to the client`; +- trial licence: a consumer's OData calls got `The OData endpoint returned 402 Payment Required` once machine sessions accumulated; +- an external entity republished in a published OData service passes `mx check` and answers 404 at runtime; +- DataGrid 2 authored by mxcli on 10.24 is CE0463 until `mxcli fix widgets`. diff --git a/contrib/inbox/2026-10-12-native-mxcli-bughunt.md b/contrib/inbox/2026-10-12-native-mxcli-bughunt.md new file mode 100644 index 00000000..8fab8b87 --- /dev/null +++ b/contrib/inbox/2026-10-12-native-mxcli-bughunt.md @@ -0,0 +1,59 @@ +# mxcli v0.26.0 defects on native-mobile documents — FSM template round trip, 2026-10-12 + +Unreviewed inbox drop. Source: Siemens FSM template (Mendix 9.24.2), the 12 own `Native*` modules +(NativeMobile, NativeMobileAsset, NativeMobileForm, NativeMobilePart, NativeMobileRelated, +NativeMobileTimeRegistration, NativeMobileWarehouse, NativeModelCheck, NativeNote, NativeReason, +NativeRestAPI, NativeTechnician). Method: `describe module with all` plus the tree fill-in of +`rehost-extract.sh` (1142 documents, all described) -> `mxcli check -p ` with a +quarantine loop (so references resolve; every error below is mxcli rejecting mxcli's own output) -> +`exec` into a copy of the source with the 12 modules dropped. Repros: `pipelines/mendix-mpr/samples/fsm-template-9.24.2/native-bughunt/`. +Counts are error lines across all quarantine rounds (one statement can be reported once per round). + +## Defects + +| id | document type | verbatim error | repro | modules | count | severity | +|---|---|---|---|---|---|---| +| D3 confirmed (also stated in the Decompose plan entry of CHANGELOG, no repro filed) | microflow, nanoflow (any) | `Error: failed to create microflow: unit 91c57c83-…: this Mendix 9.24.2 project cannot store Microflows$Microflow.StableId (Mendix 10.2.0+); Microflows$SequenceFlow.CaseValues (Mendix 10.15.0+): the properties did not exist yet in this version, and Studio Pro cannot open a document that holds one.` (nanoflow: `CreateNanoflow: insert: … cannot store Microflows$SequenceFlow.CaseValues (Mendix 10.15.0+)`) | `D3-…mdl`: an empty `create or modify microflow Repro.MF_Min () begin return; end;` already fails on a fresh 9.24.2 app | every module with a flow (NativeRestAPI isolated: all 4 flows fail one by one) | 100% of flows tried | blocks round-trip: no microflow or nanoflow can be written to a 9.24.2 project, so nothing that calls one can follow | +| D1 NEW | entity | `statement N: index on unknown attribute 'desc'` — describe printed `index ( desc);` (no column) | `D1-…mdl` | NativeTechnician, NativeReason, NativeMobileWarehouse, NativeMobile, NativeMobilePart, NativeMobileForm, NativeMobileTimeRegistration, NativeNote, NativeMobileAsset, NativeMobileRelated (every entity module but NativeModelCheck/RestAPI) | 23 | blocks round-trip: the entity is rejected, and every flow, page and grant naming it cascades (NativeReason: 0 of 45 documents survive check) | +| D2 NEW | page, snippet (native Repeater widget) | `` `content` is not a container of `repeater` — it declares: template [MDL-WIDGET26] `` | `D2-…mdl` (needs `com.mendix.widget.native.Repeater.mpk` + `mxcli widget init`) | NativeNote, NativeMobile, NativeMobileForm, NativeMobileRelated, NativeMobileAsset, NativeMobilePart, NativeMobileTimeRegistration | 32 | blocks round-trip (same family as the known SafeAreaView `content`; the Repeater slot is also `template`) | +| D4 NEW | page, snippet | `widget 'repeater4': nanoflow data source NativeMobileForm.DS_GetInstanceNativePictureImage has no argument for parameter 'NativeInstanceItem' — Mendix rejects this with CE1571.` (also `nanoflow action … (Action:) has no argument for parameter(s) …`) | `D4-…mdl` excerpt, not isolated (blocked by D3) | NativeMobile, NativeMobileForm, NativeMobilePart, NativeMobileRelated, NativeMobileAsset, NativeMobileTimeRegistration, NativeNote | 70 | blocks round-trip, or silent drift if the source mapped the parameter from the context object: describe prints the call without the mapping | +| D5 NEW | page | `widget 'textBox1': Attribute 'Search' is bound but there is no enclosing data container providing entity context` (also datePicker1..4 `Start`/`End`) | not reduced | NativeMobileTimeRegistration, NativeMobileAsset, NativeMobileRelated | 11 | blocks round-trip (describe loses or never emits the data view the widget sat in; unverified which) | +| D6 NEW | snippet | `dynamictext "text42": ContentParams parameter {1} is written as "$currentObject/NotResolvedReason" — a template parameter is evaluated against the widget's own context object` | not reduced | NativeMobileRelated | 2 | blocks round-trip | +| D7 NEW | page | `widget 'dataView1': selection DataSource references 'grid1' but no widget with that name exists on this page` | not reduced | NativeModelCheck | 1 | blocks round-trip (describe names a widget it did not emit) | +| D8 NEW | microflow | `enum split has 28 branches; at most 16 branches are supported` (`NativeModelCheck_Process`) | not reduced | NativeModelCheck | 1 | blocks round-trip (a limit in create that describe does not respect) | +| D9 NEW | microflow | `MDL-REST04: authentication microflow … has parameter $Token, which Mendix does not supply to an authentication microflow … (CE0336)` | not reduced | NativeRestAPI | 1 | lint on valid source; blocks round-trip through check | +| D10 confirmed | javascript action | `line 13427:67 mismatched input '<' expecting {FOLDER, '('}` (type parameter `Name(…)`) | see nativenote finding | NativeMobile | 1 | blocks round-trip | +| D11 confirmed | page (native layout) | `` `content` is not a container of `safeareaview` — it declares: template [MDL-WIDGET26] `` | see nativenote finding | NativeMobile (page Login) | 2 | blocks round-trip | +| D12 confirmed | module | `with all` omits nanoflows, snippets, JS actions; 10 000-line `-c` truncation (NativeMobile ends mid-statement) | n/a | NativeMobile and others | all | silent drift (tree fill-in recovered all 1142 documents) | +| D13 confirmed | exec | forward / cross-module references refused (`child entity module not found: NativeMobilePart`; `microflow not found: NativeReason.…`); grant before association (`entity NativeNote.NativeImage has no member(s) NativeImage_NativeNote`); not atomic | n/a | NativeMobileWarehouse, NativeModelCheck, NativeNote, NativeMobileTimeRegistration, NativeMobileForm, NativeMobilePart | 6 modules | blocks round-trip | +| D14 confirmed | image collection | `image collection not found: FSM_Specific.Material_Design_Icons` seen only against an empty scratch app (missing dependency) — not re-tested against the source | n/a | — | — | — | + +Cosmetic, not counted: `commit` default-events notice (167 on the empty-scratch run), `Wrap dataview … in layoutgrid` lint hints. + +Not tested because D3 and D13 stop exec first: target `describe` versus source `describe` (nothing but modules, roles and 3 NativeRestAPI constants reached the target). + +## Per module + +`create` statements = `create or modify` lines in the raw describe (entities, flows, pages, snippets, actions, module and roles). +Rejected by check = statements quarantined by `check` against the source app (root cause or cascade from D1/D2/D4). Executed = +documents written to the 9.24.2 target. Missing dependency: 0 (checked against the full source app). + +| module | create statements | clean in check | rejected by check | executed | exec blocker | +|---|---|---|---|---|---| +| NativeTechnician | 15 | 3 | 11 | 0 (module+roles) | D1 cascade | +| NativeReason | 45 | 3 | 37 | 0 (module+roles) | D1 cascade | +| NativeMobileWarehouse | 39 | 6 | 28 | 0 | cross-module ref (D13) | +| NativeModelCheck | 51 | 47 | 3 | 0 | cross-module ref (D13), D3 | +| NativeRestAPI | 12 | 11 | 1 | 3 constants + role | D3 on all 4 flows | +| NativeNote | 66 | 24 | 39 | 0 | grant before association (D13) | +| NativeMobileTimeRegistration | 80 | 14 | 61 | 0 | cross-module ref | +| NativeMobileAsset | 75 | 41 | 32 | 0 | cross-module ref | +| NativeMobileRelated | 97 | 76 | 21 | 0 | check lint D6 | +| NativeMobileForm | 104 | 35 | 62 | 0 | cross-module ref | +| NativeMobilePart | 156 | 45 | 99 | 0 | cross-module ref | +| NativeMobile | 461 | 136 | 308 | 0 | pre-check refusal, D2/D11, D3 | + +All 12 modules covered; execute stage reached only the module level, so classes "defect / missing dependency / other" +read: check-rejected = defect or cascade of defect (D1, D2, D4–D9), missing dependency = 0, exec-blocked = D3 and D13. +Re-run once D3 is fixed (or against an 11.x target, where the FSM native modules need the Blank Native starter) to expose +the next layer. diff --git a/contrib/inbox/2026-10-12-rehost-increment-2-genai-mxcli-findings.md b/contrib/inbox/2026-10-12-rehost-increment-2-genai-mxcli-findings.md new file mode 100644 index 00000000..99106ab1 --- /dev/null +++ b/contrib/inbox/2026-10-12-rehost-increment-2-genai-mxcli-findings.md @@ -0,0 +1,107 @@ +# mxcli v0.26.0 defects found regenerating the GenAI Showcase App (Rehost increment 2) + +**From:** mendix/genai-showcase-app (Rehost increment 2) — MendixMau +**Date:** 2026-10-12 +**Kind:** bug +**Field evidence:** `pipelines/mendix-mpr/samples/genai-showcase-11.12.0/` (run.log, ledger, mx check / mxbuild captures, FINDINGS.md); Mendix 11.12.0 source, 36 modules regenerated into `mxcli new --version 11.12.0`; Linux container, no Studio Pro +**Proposed target:** `bug-logs/mxcli-bugs.md` (one BUG-n each), upstream mxcli issues; extractor items to `pipelines/mendix-mpr/` + +--- + +Format: defect — verbatim message — one-line repro — what works around it (or "nothing"). + +## mxcli defects +- **`-c describe module … with all` truncates at 10 000 lines without a warning.** There is no message: + the output simply stops in the middle of a statement. Repro: `mxcli -p .mpr -c "describe module AgentCommons with all" | wc -l` stops at the 10 000-line cap. + Workaround: `rehost-extract.sh` `describe_module()` drops the incomplete last statement and describes + the documents it is missing one by one. +- **`with all` leaves document types out.** Nanoflows, snippets, JS actions, JSON structures, + import/export mappings, task queues, image collections and some pages are missing (10 of 22 pages in + AgentCommons). Repro: compare `describe module M with all` against `project-tree --json` for M. + Workaround: the `project-tree` fill-in in `rehost-extract.sh`. +- **`describe java action` puts the parameter separator after the `--` comment.** Verbatim: + `Name: String not null -- This sets the Name of the tool...,` then check: + `line 255:4 mismatched input 'Name' expecting ')'`. Repro: `describe java action GenAICommons.` + and `check` the output. Workaround: `lib/fix-describe-output.py`. +- **A microflow and a Java action with the same name.** If the flow is created first: + `warning: could not look up Java action … java action not found` → `Error: failed to create microflow: + refusing to write unit …: attribute reference not qualified`. Repro: exec + GenAICommons.ConsumedKnowledgeBase_GetCollectionOptions (the flow) before the Java action with the same + name. Workaround: `lib/topo-order-mdl.py` orders the Java/JS action first. +- **A grant that names an association member before the association exists.** Verbatim: + `Error: entity EchoConnector.Credentials has no member(s) Credentials_Configuration; grant only names + members of the entity`. Repro: describe order of the EchoConnector module. Workaround: topo makes + grants their own units, which depend on the associations they name. +- **Message definitions cannot be expressed.** Verbatim: `Error: export mapping GenAICommons.EM_Reference: + message definition "GenAICommons.MD_Reference.Reference" not found — this project has no message + definitions`. Repro: exec any export mapping built on a message definition. Workaround: nothing. These + are ledger failed-exec rows, and they cause 4 × CE1613 at mxbuild (`The selected export mapping '…' no longer exists`). +- **A Mendix 9-built MPK cannot be installed into 11.** Verbatim: `build a reference project from the + package: mx module-import failed: exit status 117` / `The package could not be imported, because it was + created with version 9.24.2 of Mendix Studio Pro. Please open it in any version in the range from + 10.21.0 to 10.24.99 first.` Repro: `mxcli marketplace install 220472 --version 3.4.1 -p <11.12 app>`. + Workaround: regenerate the module from source with the Rehost extractor. +- **The marketplace version stamped in the source is absent.** Verbatim: `version "1.5.1" not found; run + 'mxcli marketplace versions '`. Repro: `mxcli marketplace install 239450 --version 1.5.1`; the + oldest version listed is 3.0.0. Workaround: regenerate from source. +- **Template module versions do not match the source, and there is no headless update.** + `marketplace install` of a module that is already present is "NOT done automatically". Repro: + `mxcli new --version 11.12.0` ships Administration 4.3.2 while the source has 4.5.0. Workaround: + nothing; these are ledger hand-off rows. +- **#923: merges and decisions cannot be represented.** describe prints WARNING lines, and the target + describe drifts on layout (`'@curve(from: (15, 0), to: (-30, 0))' -> "@caption 'empty'"`). Repro: + describe any flow with a shared merge. Workaround: `lib/compare-mdl.py` rates #923 warnings as + `not-expressible`. +- **NEW — exec writes a microflow whose `import from mapping` result has a null entity, and the `.mpr` + can no longer be loaded.** Verbatim (mx check): + `System.ArgumentNullException: Value cannot be null. (Parameter 'value') at + Mendix.Modeler.DataTypes.EntityType.set_EntityId` — "An error occurred when trying to set the 'Entity' + property of a Object type in a Microflow with ID …". Repro: exec + Embeddings_SemanticSearch.Book_CreateDatasetIfNeeded in the order run 4 used (the topo order let the + flow run ahead of IM_SemanticSearch_Books; that mxcli then writes a null entity is a hypothesis, and so + is the cause). Workaround: `drop microflow X;` plus the same `create or modify microflow X` in one + script once the mapping exists. A plain re-`create or modify` is refused with `MDL-V1-REBUILD`. +- **NEW — a Java action type parameter is described as `MxObject: "Entity"`, with no type-parameter + declaration, and exec writes entity `.Entity`.** Verbatim (mxbuild): `Property 'Entity' of Java action + parameter 'MxObject': The selected entity '.Entity' no longer exists.` Repro: + `describe java action GenAICommons.ChunkCollection_AddKnowledgeBaseChunk`, exec it, then `mx check`. + Workaround: nothing (7 build errors). The same defect affects JS actions (increment-3 inbox file). +- **NEW — a sort on an inherited attribute is written against the specialization.** Verbatim: `CE1613 + "The selected attribute 'Embeddings_SemanticSearch.BookChunk.Similarity' no longer exists." at List + operation activity 'Sort'`. `sort $BookChunkList by Similarity desc` on a BookChunk that + `extends GenAICommons.KnowledgeBaseChunk`. The target describe is identical to the source, so a + describe-only check calls it identical. Repro: exec Embeddings_SemanticSearch.DS_Query_GetBookChunks, + then `mx check`. Workaround: nothing. +- **NEW — rules cannot be described or created.** Verbatim: `describe rule …` → `Unknown type: rule`. + exec of a flow whose decision calls a rule: `if condition calls + 'PgVectorKnowledgeBase.Rule_DatabaseConfiguration_IsConfigured(...)', which is not a rule in this + project`. Repro: `mxcli -c "show rules"` lists 14 rules; `describe rule` refuses all of them. + Workaround: nothing (8 flows deferred, 12 rules uncounted). +- **NEW — describe emits MDL that its own `check` rejects.** 30 documents. Examples: + `ContentParams parameter {1} is written as "$ChatContext/Topic"` [MDL-WIDGET24]; + `merge shared1: nothing joins it.` [MDL-FLOW02]; + `layout 'ConversationalUI.Layout_MasterBase' declares no placeholders at all` [MDL081]; + `DecimalPrecision must be a non-negative integer, got &{-1}` [MDL-WIDGET18]; + `argument Task: $Task cannot be stored — widget actionButton17 is not inside a data view` [MDL-PAGEARG01]. + Repro: describe the named document, then `check` the output against any target. Workaround: the + extractor quarantines these statements (but see the toolkit items below). +- **NEW — exec refuses forward references inside one script, so mutual recursion cannot be created.** + Verbatim: `references microflow X before it is created (statement 240), and the project does not have + it`. Repro: GenAICommons.Response_CallRecursively ↔ ChatCompletions_WithHistory. Workaround: nothing. + Hypothesis: a create-stub-then-modify pass would get round it. +- **NEW — a describe output fails validation.** Verbatim: `microflow + 'GenAICommons.Span_CalculateSumDurationUntilExecution' has validation errors: - variable 'SumDuration' + is not declared`. Repro: describe it and exec it. Workaround: nothing. +- **Seen, not isolated:** `warning: could not look up Java action System.VerifyPassword` (System Java + actions are unknown to mxcli); import mapping `with json structure … root (Wrapper)` → `line 342:59 + extraneous input '('`; dynamic text `ContentParams: ({1} = ToolName format (decimalPrecision: -1))` → + `no viable alternative`. + +## Toolkit (extractor) items. These are not mxcli; they are for the next extractor commit. +- `lib/quarantine-mdl.py` does not map lint-form errors (`✗ … [CODE]` followed by `at Q.N`). It also + trusts mxcli's `statement N:` numbering, which differs from `lib/mdlsplit.py` (AgentCommons 48 vs 49, + AmazonBedrockConnector 357 vs 360, MxGenAIConnector 135 vs 144). +- `rehost-extract.sh` `check_exec`: when nothing can be quarantined, the remainder is neither executed + nor deferred. In run 4 this lost 7 modules, 1040 statements and 1520 ledger rows. +- The deferred comment nests on every retry pass, and the `never executed` count is 7× too high. +- The ledger's denominator does not count rules, because they are not in the catalog types. diff --git a/contrib/inbox/2026-10-13-genai-nightly-findings.md b/contrib/inbox/2026-10-13-genai-nightly-findings.md new file mode 100644 index 00000000..b320b82a --- /dev/null +++ b/contrib/inbox/2026-10-13-genai-nightly-findings.md @@ -0,0 +1,69 @@ +# mxcli nightly defects found regenerating the GenAI Showcase App (Rehost, nightly re-run) + +**From:** mendix/genai-showcase-app (Rehost increment 2, nightly re-run) — MendixMau +**Date:** 2026-10-13 +**Kind:** bug +**Field evidence:** `pipelines/mendix-mpr/samples/genai-showcase-11.12.0/nightly/` (run-log.txt, fidelity-ledger.tsv, mx-check-target.txt, mxbuild-target.txt, FINDINGS.md). mxcli `nightly-80-gbd45c47e`, Mendix 11.12.0, 36 modules regenerated into `mxcli new --version 11.12.0` with 5 marketplace modules installed. +**Proposed target:** `bug-logs/mxcli-bugs.md` (one BUG-n each), upstream mxcli issues. None of these is in the 2026-10-13 nightly retest table (U1–U53), except where a U-id is named as related. + +--- + +Format: defect — verbatim message — one-line repro — what works around it (or "nothing"). + +- **N-1. `project-tree --json` omits rules and message definition collections.** Both are describable + on the nightly (`describe rule` works, U5 FIXED), but neither appears in the tree, so a + tree-driven extractor never emits them and they are missing from its denominator. + Repro: `mxcli -p "GenAI Showcase App.mpr" -c "show rules"` lists 9 `Rule_` documents; + `mxcli -p … project-tree --json | grep -c Rule_` prints 0. No `message` type in the tree either. + Effect on GenAI: 5 × CE1613 for export mappings built on message definitions + (`Error: export mapping GenAICommons.EM_Reference: message definition "GenAICommons.MD_Reference.Reference" not found — this project has no message definitions`), + and flows that call rules fail exec validation (`if condition calls 'AgentCommons.Rule_PromptToUse_ContainsVariable(...)', which is not a rule in this project`). + Workaround: none in the toolkit yet; an extractor could add `show rules ` and the message + definition listing to its fill-in. +- **N-2. exec writes a ContentParams association path into `AttributeRef.Attribute` as a raw + string, and the model then cannot load.** check passes; exec reports `Created snippet + AgentCommons.Snippet_Output_List`. Then `mx check` and mxbuild stop with + `System.ArgumentNullException: Value cannot be null. (Parameter 'value')` — "An error occurred when + trying to set the 'Attribute' property of a Attribute in a Snippet with ID 7215e844-…". The + stored value is `AgentCommons.Value.Value_Variable/AgentCommons.Variable/Key` (BSON scan of + `mprcontents`). The describe line is + `ContentParams: ({1} = $currentObject/AgentCommons.Value_Variable/AgentCommons.Variable/Key)`; + removing `$currentObject/` stores `AgentCommons.Value_Variable/AgentCommons.Variable/Key`, which is + just as unloadable. The source stores this parameter with `AttributeRef` null. Related to U10 + (STILL_BROKEN, the `$currentObject/` form, which check refuses), but this is the + association-path form, which check lets through. Workaround: replace the parameter by hand (hand-off). +- **N-3. check and exec accept an import or export mapping whose entity does not exist.** + `Created import mapping ExampleMicroflows.IM_Ticket` / `Created export mapping + ExampleMicroflows.EM_Ticket` while `ExampleMicroflows.Ticket` was absent (`describe entity` → + `Error: entity not found`). `mx check` then reports 16 errors of the form `[CE1613] "The selected attribute + 'ExampleMicroflows.Ticket.Identifier' no longer exists." at Value mapping element 'Identifier'`. + Expected: check refuses the mapping like it refuses a microflow parameter of a missing entity + (`entity ExampleMicroflows.Ticket does not exist`). +- **N-4. (likely) A pluggable widget attribute bound to an outer data context loses that context + in describe.** `markdown markdownViewer3 (stringAttribute: SystemPrompt, …)` sits inside + `dataview dataView7 (DataSource: microflow ConversationalUI.DS_CopyHelper_Create)`. exec binds it to + the innermost entity, and `mx check` reports `[CE1613] "The selected attribute + 'ConversationalUI.CopyHelper.SystemPrompt' no longer exists." at Markdown viewer 'markdownViewer3'` + (3 errors). `CopyHelper` has only `IsCopied`. The source binding has not been read from BSON yet, + so confirm that first. +- **N-5. describe emits a snippet call without the `Params:` its snippet requires.** exec: + `Error: failed to build page: failed to build widget: snippet AgentCommons.Snippet_Agent_SelectVersion + requires parameter $Agent — add Params: (Agent = $) to the SNIPPETCALL, or place the call + inside a data context`. So `AgentCommons.Version_Overview` never lands, which gives 4 × CE1613 at its callers. The same + happens to `MCPClient_ConnectToServer.Snippet_Prompt_Arguments` (`requires parameter $Prompt`). Maybe + the same family as U6. +- **N-6. exec errors that do not name the failing statement.** `Error: failed to build snippet: + failed to build widget: template for AgentCommons.MCP in list view listView1: AgentCommons.MCP is + not a specialization of AgentCommons.Tool, so the template can never match an object the list view + shows`. That names two entities but not the snippet being built, and exec prints no statement + number. A resume loop cannot place it, so the toolkit falls back to one statement per process + (`lib/exec-each.py`; 320 statements of ConversationalUI ran that way). Expected: prefix the failure with + `statement N:` as `--continue-on-error` already does. +- **N-7. Building blocks describe as comments only.** `describe buildingblock + ConversationalUI.ConversationStarter` prints `-- Building Block: …` lines with no `create` + statement. These are the 4 `not-emitted` ledger rows. Hand-off. + +Still open from the retest (re-seen on this run, no new information): U7 (Java action type parameter → `.Entity`, now 14 build +errors because more modules land), U18 (#923 decisions, 3 `not-expressible` rows), U23 (`refusing to write +unit …: attribute reference not qualified`), U31 (stdout of `-c describe module … with all` still ends at +10 001 lines; the toolkit's truncation defence still fires on AgentCommons and ConversationalUI). diff --git a/contrib/inbox/2026-10-13-increment-8-dispatch-findings.md b/contrib/inbox/2026-10-13-increment-8-dispatch-findings.md new file mode 100644 index 00000000..3d4eaa8d --- /dev/null +++ b/contrib/inbox/2026-10-13-increment-8-dispatch-findings.md @@ -0,0 +1,44 @@ +# Increment 8 (Decompose, Dispatch & Workforce) — new mxcli 0.26.0 findings + +Source: Siemens FSM template, increment 8 (`pipelines/mendix-mpr/samples/fsm-template-9.24.2/slice-dispatch/`). +Monolith FSM 10.24 copy (MPR v1), slice DispatchApp 11.13 (MPR v2). New defects only; known ones are listed at the end. + +1. **describe page → create refuses its own output (ContentParams path).** `describe page Scheduler.PageContext_View` + prints `ContentParams: ({1} = $currentObject/FilterCaption)`; exec of that output refuses: + `dynamictext "text7": ContentParams parameter {1} is written as "$currentObject/FilterCaption" — a template parameter is + evaluated against the widget's own context object, so it takes the attribute name …`. Workaround: `{1} = FilterCaption`. +2. **describe page → create resolves a nested visibility rule against the wrong entity.** Same page, after its + `TicketManagement.WorkOrder` became an external entity: `Error: failed to build page: failed to build widget: dynamictext + text14: Visible: Status in (…): Scheduler.PlannerContext_WorkOrder_Selected has no attribute Status`. The statement before + it in the script (a microflow) was already written. Workaround: `alter page … { set (Visible: …) on text14; }` instead of + re-creating the page. +3. **exec pre-check does not see what an earlier statement of the same script created** — variant of mendixlabs/mxcli#1061 + (queues): an attribute added by `alter entity M.E add attribute if not exists PublicID: AutoNumber default 1;` → + `retrieve from TicketManagement.Appointment: the constraint names "PublicID", which is neither an attribute nor an + association of it`; entities from `create external entities …` → `page 'DispatchSeam.Appointments_FromCore' has + reference errors: - entity not found: DispatchSeam.Appointment`. Both scripts are valid; `--no-check` applies them. +4. **A rule-based decision describes as `if not(Rule(...)) then`.** drop + create then writes an expression split, and an + expression cannot call a rule: `[CE0117] "Error(s) in expression." at Decision 'User is SDE?'` + (`Scheduler.PlannerContext_GetWorkOrders`). This is the root cause of increment 7 inbox #6. Workaround: inline the + rule's test (`retrieve $R from System.UserRole where [id = '[%UserRole_ServiceDeskEmployee%]'][System.UserRoles = $currentUser] first;` + then `if $R = empty then`). +5. **`alter microflow X { insert before return true begin … end; };` is not idempotent.** Each re-run of the script inserts + the call again (3 runs → 3 calls in `General.ASU_General`). Expected: refuse, or report that the activity is present. + +Platform behaviour worth a learned-* line (not mxcli): +- external entities: an XPath on an attribute not published **Filterable** fails as a bare `CE0161 "Error(s) in XPath + constraint."` (30 probe microflows to find it); sorting needs **Sortable** (`CE7247 "Cannot sort on attribute + 'PlannedWorkStart' because it does not support sorting."`); a navigation is published on its owner only, so a reverse + traversal is CE0161 and a gallery over one is `CE8812 "Entity 'TicketManagement.WorkOrder' has no association + 'TicketManagement.Appointment_WorkOrder'."`; +- enumerations as Strings (P8) also break enum visibility rules: `CE0749 "The selected value 'Status' is not of type Boolean or Enumeration."`; +- `on error without rollback` inside a loop: `CE0644 "Error handling type must be 'Rollback' inside a looped activity."`; +- `mx check` text output has no document names; `mx check -j out.json` has them (`locations[].document-name`); +- Mendix 11 Java API vs 10.24 user code: `context()` → `getContext()`, `Core.isSubClassOf(String,String)` gone + (`Core.getMetaObject(type).isSubClassOf(…)`), proxy parameters need `getMendixObject()`; +- a legacy `Forms$Table` hides its children from describe and `replace` (`widget "referenceSetSelector1" not found`): + replace the table. + +Known, seen again: BUG-87 (`entity <>`), bare `TypeParameter` Java action parameter (CE1613 `'.TypeParameter'`), +`format (decimalPrecision: -1)` not re-parsable, increment 7 #3 (nested widget not found) and #5 (consumed-service +metadata not refreshable: `0 created, 0 updated, 9 skipped`), non-atomic exec. diff --git a/pipelines/mendix-mpr/README.md b/pipelines/mendix-mpr/README.md new file mode 100644 index 00000000..6da1e141 --- /dev/null +++ b/pipelines/mendix-mpr/README.md @@ -0,0 +1,99 @@ +# Mendix MPR pipeline — the Rehost extractor + +**Source platform:** an existing Mendix app (`.mpr`, Mendix 9 and above, MPR v1 or v2). +**Plan:** `process/decompose-and-rehost-plan-2026-10-12.md` §4 (Rehost) — read a model with +`mxcli describe`, regenerate it as MDL into a fresh target app, and write a **fidelity ledger** +(one row per source document; the denominator in the header) instead of claiming 100 %. + +This is the only pipeline whose extractor reads a Mendix model. Downstream of the ledger nothing +here is Mendix-source-specific: the slice loop of the Decompose track reuses it to move a +module into a new app, and Stages 2–6 of the runbook apply unchanged. + +## Run + +``` +MXCLI=/path/to/mxcli pipelines/mendix-mpr/rehost-extract.sh [Module ...] +``` + +- The target must exist: `mxcli new --version --theme none`. The script + writes model content only; run the mxbuild gate (`project-bin/exec.sh`, or + `~/.mxcli/mxbuild//modeler/mx check`) afterwards. +- Modules default to every own module (empty Source column in `list modules`, System excluded). + Marketplace modules are recorded by name and version on the ledger, never regenerated. +- Output: `captures/` (verbatim mxcli output — the golden fixtures), `mdl/` (describe output, + the topologically ordered file that was executed, the target's describe for the diff), + `fidelity-ledger.tsv`. + +`lib/topo-order-mdl.py` reorders the flow statements of a `describe module … with all` script so +every callee is created before its caller (a flow and its `grant execute` lines move as one unit). +`lib/compare-mdl.py` normalises two describe outputs for the per-document identical/drift verdict. +Both read and write with `newline=''`: Studio Pro stores annotation text with CRLF inside MDL string +literals, and a tool that normalises it changes the flow and `exec` refuses the splice. + +## Field runs + +| Date | Source | Result | +|---|---|---| +| 2026-10-12 | ako/mxcli-banking `RRNetBanking.mpr` (Mendix 11.13.0, Banking 151 docs + MyFirstModule 4) → `mxcli new --version 11.13.0` | 155/155 documents emitted, 154 identical on target describe, 1 doc-comment drift; `mx check` 0 errors, mxbuild BUILD SUCCEEDED, `diff --format struct` empty. 35 s end to end | +| 2026-10-12 | FSM template v1.1.1 (Mendix 9.24.2, MPR v1, 64 MB) — **read probe only** | `list modules` 1.6 s, `graph-report` 18 s. Captures in `samples/fsm-template-9.24.2/` | +| 2026-10-12 | mendix/genai-showcase-app `GenAI Showcase App.mpr` (Mendix 11.12.0, 48 modules: 32 own + 15 marketplace; 456 entities, 1622 microflows, 180 pages, 204 Java actions) → `mxcli new --version 11.12.0` | **Measured, not build-green** — run 4, 36 modules regenerated, 36 min wall time (describes reused; 30 min more when fresh). Ledger over 2776 catalog documents: 579 identical, 271 drift, 1993 failed-exec, 43 not-emitted, 4 not-expressible, 2 target-extra. 1520 of the failures are 7 whole modules the extractor dropped: `lib/quarantine-mdl.py` cannot map lint-form check errors or mxcli's statement numbers, and `check_exec` then discards the rest of the module. Gates: the target first failed to load (`mx check` ArgumentNullException; mxcli wrote 3 microflows with a null `import from mapping` result entity, repaired by drop + create). After that, `mx check` gave 14 × CE1613 (Java action type parameters `.Entity`, missing message-definition export mappings, inherited-attribute sort) and mxbuild `BUILD FAILED`. Regression parity is not reachable headlessly (no message definitions, no rules, mutual recursion refused). Captures, ledger and verdict: `samples/genai-showcase-11.12.0/FINDINGS.md` | +| 2026-10-13 | mendix/genai-showcase-app (same source and target snapshot as run 4) on **mxcli nightly-80-gbd45c47e**, extractor with the E1–E3 fix | **36/36 modules landed, not build-green.** Ledger over 2776 documents: 1537 identical (was 579), 858 drift (271), 482 failed-exec (1993), 4 not-emitted (43), 9 not-expressible. Most of the gain is the toolkit fix (the 7 formerly dropped modules: 1363 rows landed); from the nightly: U37 (−2 build errors), U22 (null-entity model-load blocker gone), U16. Gates: the target does not load (`ArgumentNullException` on one snippet whose ContentParams association path mxcli stored as an attribute name); with that parameter replaced, `mx check` gives 42 × CE1613 in 5 named classes and mxbuild `BUILD FAILED`. 2 h 36 min end to end with reused describes (+28 min fresh). Journeys NOT RUN (live LLM credentials). Captures: `samples/genai-showcase-11.12.0/nightly/` | +| 2026-10-12 | FSM template v1.1.1 — **MPK route** (`export-module-mpk.py`): FeatureToggle, Utils, Visualizations → 10.24 and 11.13 | All installed; every residual `mx check` error a crossing edge; Utils with edges satisfied: `mx check` 0, mxbuild BUILD SUCCEEDED on 11.13 after 3 Java edits. Captures in `samples/mpk-export-probe/` | +| 2026-10-12 | FSM template v1.1.1 — **first slice (increment 3)**: `NativeNote` + 26 Synchronization_Note documents → Blank Native Mobile App 11.13.0 (9.24 copy → `mx convert` 10.24 → export → `install --file`; edges in MDL) | NativeNote 66/66 documents describe-identical on target; `mx check` 0, mxbuild BUILD SUCCEEDED; cut ledger (`bin/cut-ledger.sh`) 308 rows, 0 unresolved — 26 stubs (incl. `Synchronization.Synchronize`, synced work-order/technician entities), 98 hand-offs. A native slice cannot target `mxcli new` (no native profile). Captures in `samples/fsm-template-9.24.2/slice-nativenote/` | +| 2026-10-12 | FSM template v1.1.1 — **seam probe (increment 4)**: monolith → NativeNote slice (11.13) over published REST | One record crossed: 10.24 monolith `ALTER`ed push → `POST rest/seam/v1/notes` → upsert in the slice, proven by OQL on both runtimes. The 9.24.2 runtime boots headless (Java 11, M2EE), but mxcli cannot create or alter microflows in 9.x; `mxcli run` cannot run 9.x (forces JDK 21). Captures in `samples/fsm-template-9.24.2/seam-probe/` | +| 2026-10-12 | FSM template v1.1.1 — **web slice (increment 5)**: `AssetManagement`, `Part`, `WarehouseManagement`, `Tool` → `mxcli new` 11.13.0 web app, integrated with the 10.24 monolith copy | **Build-green; seam-live not reached.** `mx check` 0 (via 287 CE1613 → 22 × CE0582 React-client legacy widgets, converted from `mxcli bson dump`), mxbuild BUILD SUCCEEDED; 207+10 docs identical, 5 drift, 19 widget-converted; ledger 703 rows, 0 unresolved, 120 stub rows (unchanged at the seam). Seam: published OData (basic auth, Production security) consumed as external entities, 1 in-edge rewired, REST upsert as dual write; one Playwright journey across both apps PASS. Local→external association passes `mx check` on 10.24. Captures in `samples/fsm-template-9.24.2/slice-assets/` | +| 2026-10-12 | FSM template v1.1.1 — **core services (increment 6)**: the 10.24 monolith publishes its core (OData `CoreSeam.CoreAPI`, 12 entity sets; REST `CoreSeam.CoreActions` for rosters), the increment 5 Assets & Parts slice (11.13) consumes it | **Seam-live for the non-Mobile core.** Both apps `mx check` 0 and BUILD SUCCEEDED; ledger 709 rows, 0 unresolved; stub rows 120 → 32 (non-Mobile 89 → 1, plus 20 hand-offs that external entities cannot carry: access-rule XPath ×16, local→external grid columns ×4; Mobile 31 unchanged). Stub entities swapped for same-name external entities (navigations only exist within one consumer module), keys are added `PublicID` AutoNumbers, the Marketplace stub association is replaced by a local owner entity, machine role only (401/404 measured). New Playwright journey slice → monolith → slice PASS 9/9; increment 5 journey still PASS. Monolith outage makes the slice's asset page unusable (measured). 8 new mxcli defects. Captures in `samples/fsm-template-9.24.2/slice-assets-core/` | +| 2026-10-12 | FSM template v1.1.1 — **strangled (increment 7)**: `AssetManagement`, `Part`, `WarehouseManagement`, `Tool` deleted from the 10.24 monolith; it reads the 11.13 slice's OData `AssetsAPIv2` (external entities, module `AssetsClient`) and writes over REST `rest/assets/v2` | **Level 3, with 157 reasoned in-edge hand-offs.** In-edge rows 536: handoff 499 → 157, retire 36 → 204, resolved 1 → 175 (rebind 49, rewired 54, pull 54, rest 10, association 7); both apps `mx check` 0 and BUILD SUCCEEDED; monolith 61 → 56 modules, 3250 → 2972 documents; ledger 713 rows exit 0. Route: `move entity` into one consumer module, swap for same-name external entities, drop the modules (XPath text needed a BSON rebind — `move` leaves it); native copies fed by a pull from the slice; Ticket/Roster carry the slice asset key (mirror removed). Playwright journey across both apps PASS 16/16 twice (idempotent); increment 6 journey PASS 9/9. 8 new mxcli defects. Captures in `samples/fsm-template-9.24.2/slice-assets-strangled/` | +| 2026-10-13 | FSM template v1.1.1 — **Dispatch & Workforce (increment 8)**, run from the skills: `Scheduler`, `RosterManagement`, `Technician`, `Skill`, `TimeRegistration` → `mxcli new` 11.13.0 web app, fresh 10.24 monolith copy publishing OData `dispatchcore/v1` (13 entity sets) and REST `rest/dispatchcore/v1` (3 actions) | **Level 1 (build-green); seam live on every lane, level 2 not claimed (3 behaviour stubs).** Stub rows 271 → 3 web (+1 Mobile); ledger 884 rows, 0 unresolved; both apps `mx check` 0, Production security, 200/401/404 on both services; journey read + write + auth passes twice. ≈ 1 h 55 build work vs 4 h 50 for increments 5–7. Captures in `samples/fsm-template-9.24.2/slice-dispatch/` | + +Not yet exercised: workflows, REST/OData services, and the MDL regenerate step on a Mendix 9/10 source +(the read works; increment 3 moved NativeNote into Mendix 11 by MPK, not by regeneration). + +## The MPK route — `export-module-mpk.py` (moving a module without regenerating it) + +``` +python3 -I pipelines/mendix-mpr/export-module-mpk.py [--no-widgets] [--json] +mxcli marketplace install --file -p # new modules only; no PAT +``` + +The exporter writes a module package in Mendix's own format: a zip with `package.xml` and a +`project.mpr`. That `project.mpr` holds the source schema and `_MetaData` verbatim, the module's +unit subtree byte for byte, the project root, the conversion unit and four project documents. Two +of those documents are **scrubbed of secrets** (demo users, passwords, constants, credentials, +certificates), and so is navigation. The zip also carries the module's `javasource/`, +`javascriptsource/`, `themesource/` and `resources/` folders, the userlib jars that have a +`..RequiredLib` marker, and the widget `.mpk` files its pages reference. The script header +lists what it cannot know. It refuses MPR v2 sources and schemas that differ from the reference. + +**Version window (Mendix's, not ours).** `mxcli marketplace install` imports through the target's +`mx module-import`: + +- 9.24's `mx` has no `module-import`. +- 11.13 refuses packages older than 10.21. +- So a 9.x → 11.x move has two hops: `mx convert` a copy of the source to 10.24, export, then install. + +On Mendix 10 and 11, `mx create-module-package ` is the vendor's headless export. +It refuses modules that have errors, and it **regenerates every unit and entity ID**. This +exporter keeps them, and mxcli's install keeps them too: measured from 9.24 through 10.24 to 11.13. + +**Field run 2026-10-12, FSM template 9.24.2:** + +- FeatureToggle, Utils and Visualizations each installed into 10.24 and into 11.13. +- Every `mx check` error (2 / 7 / 134) was a crossing edge to a module the target lacks. There + were 0 version-conversion errors and 0 package-format errors. +- Utils with its edges satisfied (3 Marketplace modules, 1 constant in MDL) reached `mx check` + 0 errors. After three Java hand-off edits, mxbuild BUILD SUCCEEDED on 11.13.0. The edits: + `this.context()` is a Mendix 9 API; two orphan action files used `CustomJavaAction`; one jar's + RequiredLib marker names a former module. +- The Java bodies arrive byte-identical. + +Verbatim captures: `samples/mpk-export-probe/captures/`. The `R*` files are the final run with +the committed script. For Mendix 9 `mx` on Linux, the container needs `libgdiplus` and JDK 17. + +## mxcli findings from the first run (filed in `contrib/inbox/2026-10-12-decompose-rehost-increment-1-mxcli-findings.md`) + +The subcommand `mxcli describe module ` emits only the module and its roles; the whole-module +emitter is the MDL statement `describe module with all` via `-c`. Describe output is in catalog +order and the tool's own `check` refuses it (MDL-ORDER01). Image collections are omitted by +`with all`. `describe userrole` drops `CheckSecurity: false`. Demo-user passwords describe as +`'***'` and project security level describes as a report, not MDL — both are ledger hand-off rows. diff --git a/pipelines/mendix-mpr/export-module-mpk.py b/pipelines/mendix-mpr/export-module-mpk.py new file mode 100644 index 00000000..214182f3 --- /dev/null +++ b/pipelines/mendix-mpr/export-module-mpk.py @@ -0,0 +1,494 @@ +#!/usr/bin/env python3 +"""export-module-mpk.py — headless "export module to .mpk" for an MPR v1 (single SQLite) app. + +usage: export-module-mpk.py [--no-widgets] [--json] + +WHAT IT DOES + Writes a Mendix *module package* in the shape Studio Pro / Marketplace module packages have + (studied on Marketplace "Encryption" 9.0.3, built with 9.18.0): a zip holding `package.xml` + (…) and a + `project.mpr` that is a new SQLite database with + - the source's own CREATE TABLE statements, verbatim (same schema, no translation), + - the source's `_MetaData` row, verbatim (ProductVersion stays the source's version), + - the `Projects$Project` root unit, the `Projects$ProjectConversion` unit and the four + project documents (mx module-import refuses a package without them) — SystemTexts + verbatim; ProjectSecurity, ProjectSettings and Navigation *scrubbed*: demo users and their + passwords, user roles, admin password, guest role, configuration constant values / custom + settings / database credentials, certificates, the after-startup / before-shutdown / + health-check microflow names and the navigation profiles are emptied, and those units' + ContentsHash recomputed (sha256 of Contents, base64 — verified on all 4783 FSM units), + - the module unit (`ContainmentName='Modules'`, BSON Name == ) and every unit below + it by ContainerID, every column copied byte for byte (UnitID, ContentsHash, Contents …), + plus the module's files, matched on the lower-cased module name the way Mendix lays them out: + `javasource//` (minus `proxies/`, which the build regenerates), `javascriptsource//`, + `themesource//`, `resources//`, every `userlib/` that has a + `..RequiredLib` marker (with the marker — the same rule as `mx + create-module-package --filter-required-libs`), and (unless --no-widgets) every + `widgets/*.mpk` whose widget id is referenced by a unit of the module. + +WHY THE SCRUB + Reference module packages carry *blank* project documents. The monolith's own carry demo-user + passwords, constant values and database settings, which must not travel inside a module + package. A first version omitted the four documents; `mx module-import` (10.24) then failed + with "Sequence contains no elements" (exit 117) on every package, and succeeded with all four + present (each one alone, or any three, still failed). The scrub keeps the documents and + empties the secrets. It refuses to rewrite a document whose BSON does not round-trip + byte-identically. + +WHAT IT CANNOT KNOW + - Whether the target has the modules this module references (the cut's crossing edges). It + counts them and prints them; the target's `mx check` is what proves them. + - Whether the target's Mendix version accepts the package. It does not convert BSON: the + package is exactly as old as the source. (Field result, 2026-10-12: `mx module-import` + 11.13 refuses packages older than 10.21; 9.24's mx has no module-import at all.) + - Managed (Gradle) Java dependencies, jars without a RequiredLib marker, and JavaScript + action npm dependencies outside `javascriptsource//`. + +WHEN TO USE SOMETHING ELSE + On Mendix 10.x/11.x `mx create-module-package ` is the vendor's own headless + export (MxBuild toolset). Prefer it. This script exists for MPR v1 sources whose mx has no such + verb (Mendix 9.x). It refuses — rather than guessing — an MPR v2 source (mprcontents/), a + schema that is not the reference schema, a module name that is missing or ambiguous. + +Python stdlib only. Read-only on the source (opened with mode=ro). +Field run: FSM template v1.1.1 (Mendix 9.24.2, MPR v1) — see samples/mpk-export-probe/REPORT.md. +""" +import base64 +import hashlib +import json +import os +import re +import sqlite3 +import struct +import sys +import zipfile +import xml.etree.ElementTree as ET +from xml.sax.saxutils import quoteattr + +REF_UNIT_COLS = ["UnitID", "ContainerID", "ContainmentName", "TreeConflict", + "ContentsHash", "ContentsConflicts", "Contents"] +REF_META_COLS = {"_ProductVersion", "_BuildVersion", "_SchemaHash", "_DisableAutoMprV2Upgrade"} +SHELL_TYPES = {"Projects$Project", "Projects$ProjectConversion"} +# mx module-import refuses a package without all four ("Sequence contains no elements", exit 117, +# Mendix 10.24 — field run 2026-10-12). SystemTexts is copied verbatim; the three that carry +# secrets or monolith-wide references are scrubbed (see scrub_*). +PROJECT_DOC_TYPES = {"Security$ProjectSecurity", "Settings$ProjectSettings", + "Navigation$NavigationDocument", "Texts$SystemTextCollection"} + + +class Refuse(Exception): + pass + + +# --- minimal BSON reading: top-level fields only, enough for $Type / Name / WidgetId --------- +def _bson_top(b): + """Return the top-level string fields of a BSON document (others skipped).""" + out = {} + size = struct.unpack_from(" 1: + raise Refuse(f"module name {module!r} is ambiguous ({len(match)} units)") + if len([u for u in shell if _bson_top(by_id[u][2]).get("$Type") == "Projects$Project"]) != 1: + raise Refuse("expected exactly one Projects$Project root unit") + _, mod_uid, mod_top = match[0] + subtree, stack = [], [mod_uid] + while stack: + u = stack.pop() + subtree.append(u) + stack.extend(children.get(u, [])) + return shell, subtree, mod_top, {m[0] for m in modules}, by_id + + +def crossing_refs(by_id, subtree, module, all_modules): + """Qualified names Module.X in the subtree's string fields pointing at another module. + + The match must start at a name boundary: a plain substring test counted `NativeNote.X` as a + reference to module `Note` (FSM field run 2026-10-12, increment 3: "Note x54", 0 real). In BSON + the byte before a string's first character is the high byte of its int32 length (0x00), and + inside an XPath or expression it is `[`, `/`, `(`, a space or a quote — never [A-Za-z0-9_].""" + refs = {} + others = sorted((m for m in all_modules if m and m != module), key=len, reverse=True) + for u in subtree: + c = by_id[u][2] + for m in others: + if re.search(rb"(? widgets/.mpk, read from each widget package's own XML.""" + idx, wdir = {}, os.path.join(app_dir, "widgets") + if not os.path.isdir(wdir): + return idx + for f in sorted(os.listdir(wdir)): + if not f.endswith(".mpk"): + continue + try: + with zipfile.ZipFile(os.path.join(wdir, f)) as z: + for n in z.namelist(): + if n.endswith(".xml") and n != "package.xml": + try: + root = ET.fromstring(z.read(n)) + except ET.ParseError: + continue + if root.tag.endswith("widget") and root.get("id"): + idx.setdefault(root.get("id"), f"widgets/{f}") + except zipfile.BadZipFile: + continue + return idx + + +def package_xml(module, files): + lines = ['', + '', + ' ', + f' ', + ' ', + ' '] + lines += [f' ' for f in files] + lines += [' ', ' ', ''] + return "\n".join(lines) + "\n" + + +# --- main --------------------------------------------------------------------------------- +def export(src, module, out, with_widgets=True): + tmp_mpr = out + ".project.mpr.tmp" + try: + return _export(src, module, out, with_widgets, tmp_mpr) + finally: + if os.path.exists(tmp_mpr): + os.remove(tmp_mpr) + + +def _export(src, module, out, with_widgets, tmp_mpr): + db, tables = open_source(src) + shell, subtree, mod_top, all_modules, by_id = collect(db, module) + app_dir = os.path.dirname(os.path.abspath(src)) + if os.path.exists(tmp_mpr): + os.remove(tmp_mpr) + dst = sqlite3.connect(tmp_mpr) + for name in ("_MetaData", "Unit"): + dst.execute(tables[name]) + meta_cols = [r[1] for r in db.execute("pragma table_info('_MetaData')")] + meta = db.execute(f"select {', '.join(meta_cols)} from _MetaData").fetchone() + dst.execute(f"insert into _MetaData ({', '.join(meta_cols)}) values ({', '.join('?' * len(meta_cols))})", meta) + keep = shell + subtree + q = f"select {', '.join(REF_UNIT_COLS)} from Unit where UnitID = ?" + scrubbed = [] + for uid in keep: + row, changed = scrubbed_row(db.execute(q, (uid,)).fetchone()) + if changed: + scrubbed.append(_bson_top(row[6]).get("$Type")) + dst.execute(f"insert into Unit values ({', '.join('?' * 7)})", row) + dst.commit(); dst.close() + + files = module_files(app_dir, module) + widgets_used, widgets_shipped, widgets_missing = set(), [], [] + for u in subtree: + widgets_used |= _bson_strings(by_id[u][2], "WidgetId") + if with_widgets and widgets_used: + idx = widget_index(app_dir) + for w in sorted(widgets_used): + if w in idx: + if idx[w] not in widgets_shipped: + widgets_shipped.append(idx[w]) + else: + widgets_missing.append(w) + files += widgets_shipped + + if os.path.exists(out): + os.remove(out) + with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("package.xml", package_xml(module, files)) + z.write(tmp_mpr, "project.mpr") + for f in files: + z.write(os.path.join(app_dir, f), f) + + types = {} + for u in subtree: + t = _bson_top(by_id[u][2]).get("$Type", "?") + types[t] = types.get(t, 0) + 1 + return { + "module": module, + "product_version": meta[0], + "units_module_subtree": len(subtree), + "units_shell": len(shell), + "project_docs_scrubbed": scrubbed, + "unit_types": dict(sorted(types.items())), + "from_app_store": mod_top.get("AppStoreVersion") or None, + "files": len(files), + "java_files": sum(1 for f in files if f.startswith("javasource/") and f.endswith(".java")), + "js_files": sum(1 for f in files if f.startswith("javascriptsource/")), + "widgets_referenced": sorted(widgets_used), + "widgets_shipped": widgets_shipped, + "widgets_unresolved": widgets_missing, + "crossing_refs_by_module": dict(sorted(crossing_refs(by_id, subtree, module, all_modules).items())), + "out": out, + } + + +def main(argv): + args = [a for a in argv if not a.startswith("--")] + if len(args) != 3: + print(__doc__.split("\n\n")[0] + "\n" + __doc__.split("\n")[2], file=sys.stderr) + return 2 + try: + r = export(args[0], args[1], args[2], with_widgets="--no-widgets" not in argv) + except Refuse as e: + print(f"REFUSED: {e}", file=sys.stderr) + return 3 + if "--json" in argv: + print(json.dumps(r, indent=2)) + else: + print(f"wrote {r['out']}: module {r['module']} ({r['product_version']}), " + f"{r['units_module_subtree']} units + {r['units_shell']} shell, {r['files']} files " + f"({r['java_files']} .java, {r['js_files']} js), widgets shipped {len(r['widgets_shipped'])}" + f"/{len(r['widgets_referenced'])} referenced") + if r["widgets_unresolved"]: + print(" widget ids with no .mpk in widgets/ (core or platform widgets): " + + ", ".join(r["widgets_unresolved"])) + if r["crossing_refs_by_module"]: + print(" units naming other modules (crossing edges the target must satisfy): " + + ", ".join(f"{m}×{n}" for m, n in r["crossing_refs_by_module"].items())) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/pipelines/mendix-mpr/lib/compare-mdl.py b/pipelines/mendix-mpr/lib/compare-mdl.py new file mode 100755 index 00000000..c9ba539f --- /dev/null +++ b/pipelines/mendix-mpr/lib/compare-mdl.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Per-document comparison of two `describe module … with all` scripts (source, target). +Statements are grouped per document (a create plus the grant/index statements that +name the same document), keyed by (type, qualified name); order is ignored. +Prints TSV rows: module, document, type, verdict, evidence.""" +import os, re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements +def units(path): + stmts, _ = split_statements(open(path, encoding='utf-8', newline='').read()) + # any document type (`java action`, `json structure`, `task queue`, …); names may carry digits (AmazonS3Connector.Step2) + head = re.compile(r'^(?:@[^\n]*\n)*create or modify (?:([a-z][a-z -]*?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)|(module) ([A-Za-z0-9_]+));?', re.M) + grant = re.compile(r'^grant .*? on (?:[a-z ]+?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M) + docs, last = {}, None + for s in stmts: + body = re.sub(r'/\*\*.*?\*/\s*', '', s, flags=re.S) # doc comment compared separately + doc = re.search(r'/\*\*(.*?)\*/', s, re.S) + m = head.search(body) + if m: + t, name = (m.group(1), m.group(2)) if m.group(1) else (m.group(3), m.group(4)) + t = t.replace('non-persistent entity','entity').replace('persistent entity','entity').replace('view entity','entity') + last = (t, name); docs[last] = {'body': body.strip(), 'doc': (doc.group(1).strip() if doc else ''), 'grants': []} + continue + g = grant.search(body) + if g and last and g.group(1) == last[1]: docs[last]['grants'].append(body.strip()); continue + if g: + key = next((k for k in docs if k[1] == g.group(1)), None) + if key: docs[key]['grants'].append(body.strip()); continue + sys.stderr.write('unattached statement in %s: %s\n' % (path, body.strip()[:80])) + return docs +src, tgt = units(sys.argv[1]), units(sys.argv[2]) +module = sys.argv[3] +def firstdiff(a, b): + al, bl = a.split('\n'), b.split('\n') + for i, (x, y) in enumerate(zip(al, bl)): + if x != y: return 'line %d: %r -> %r' % (i+1, x.strip()[:70], y.strip()[:70]) + return 'length %d -> %d lines' % (len(al), len(bl)) +for k in sorted(src, key=lambda k: (k[0], k[1])): + t, name = k + if k not in tgt: + print('\t'.join([module, name, t, 'failed-exec', 'absent from target describe'])); continue + s, g = src[k], tgt[k] + ev = [] + if s['body'] != g['body']: ev.append('body: ' + firstdiff(s['body'], g['body'])) + if sorted(s['grants']) != sorted(g['grants']): ev.append('grants: %d -> %d' % (len(s['grants']), len(g['grants']))) + if s['doc'] != g['doc']: ev.append('documentation: ' + firstdiff(s['doc'], g['doc'])) + # describe marks a widget it cannot author with `-- … NOT re-executable`; the document lands without it + nre = re.findall(r'-- (\S+ \(\S+\)) -- NOT re-executable', s['body']) + if nre: print('\t'.join([module, name, t, 'not-expressible', 'describe marks %d widget(s) NOT re-executable: %s%s' % (len(nre), ', '.join(nre[:3]), '; ' + '; '.join(ev) if ev else '')])); continue + # a flow whose branches cross: describe says the text is NOT equivalent and must not be re-executed (mxcli #923) + neq = re.search(r'-- WARNING: (the decision at \([^)]*\)[^\n]*?)\. This description is NOT equivalent', s['body']) + if neq: print('\t'.join([module, name, t, 'not-expressible', 'describe: %s — NOT equivalent, must not be re-executed (mxcli #923)%s' % (neq.group(1), '; ' + '; '.join(ev) if ev else '')])); continue + print('\t'.join([module, name, t, 'emitted-drift' if ev else 'emitted-identical', '; '.join(ev) if ev else 'describe on target identical (body, grants, documentation)'])) +for k in sorted(set(tgt) - set(src)): + print('\t'.join([module, k[1], k[0], 'target-extra', 'present in target only (not in source describe)'])) diff --git a/pipelines/mendix-mpr/lib/exec-each.py b/pipelines/mendix-mpr/lib/exec-each.py new file mode 100755 index 00000000..10a0a0ab --- /dev/null +++ b/pipelines/mendix-mpr/lib/exec-each.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +"""exec-each.py + +Last resort of rehost-extract.sh's check_exec, for a script that `mxcli check` rejects in a way +lib/quarantine-mdl.py cannot map onto a statement, or whose exec error lib/exec-resume.py cannot place. +Executes the script ONE STATEMENT AT A TIME (each through `mxcli exec`, which runs check first), in +order: a statement that lands is in the target; one that fails is appended to with +its verbatim error and retried by the caller's deferred passes. is emptied (to +`mdl 1;`) — every statement has either landed or reached the deferred file, so nothing is dropped. + +GenAI Showcase 2026-10-12 (FINDINGS E2): when quarantine mapped nothing, check_exec returned 1 and +left the remainder in .ordered.mdl, never retried — 7 modules, 1040 statements, 1520 ledger rows +silently lost. Slow by design (one mxcli process per statement, ~1-2 s each); it runs only when the +mapping fails. Prints " ". +""" +import os, re, subprocess, sys, tempfile +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements, strip_deferred + +mxcli, tgt, src, deferred, cap = sys.argv[1:6] +stmts, tail = split_statements(open(src, encoding='utf-8', newline='').read()) +if tail.strip(): stmts.append(tail) +body = [strip_deferred(s) for s in stmts if s.strip() != 'mdl 1;'] +body = [s for s in body if re.search(r'^(create|grant|revoke|alter|drop|delete)\b', s, re.M)] +landed = failed = 0 +fd, one = tempfile.mkstemp(suffix='.mdl'); os.close(fd) +with open(cap, 'w', encoding='utf-8') as log, open(deferred, 'a', encoding='utf-8', newline='') as d: + if os.path.getsize(deferred) == 0: d.write('mdl 1;\n') + for s in body: + open(one, 'w', encoding='utf-8', newline='').write('mdl 1;\n' + s + '\n') + r = subprocess.run([mxcli, 'exec', '-p', tgt, one], capture_output=True, text=True, errors='replace') + out = '\n'.join(l for l in (r.stdout + r.stderr).split('\n') if 'vibe-coded PoC' not in l) + head = s.strip().split('\n')[0][:120] + if r.returncode == 0: + landed += 1; log.write('OK %s\n' % head); continue + failed += 1 + plain = re.sub(r'\x1b\[[0-9;]*m', '', out) + err = [l.strip() for l in plain.split('\n') if re.match(r'\s*(Error:|statement \d+:|- |✗|.*error)', l)] + reason = ' '.join(err)[:400] or plain.strip().split('\n')[-1][:400] + log.write('FAIL %s\n%s\n' % (head, plain)) + d.write('-- deferred: exec (one statement at a time) refused this statement: %s\n%s\n' % (reason, s)) +os.unlink(one) +open(src, 'w', encoding='utf-8', newline='').write('mdl 1;\n') +print(landed, failed) diff --git a/pipelines/mendix-mpr/lib/exec-resume.py b/pipelines/mendix-mpr/lib/exec-resume.py new file mode 100644 index 00000000..c612597b --- /dev/null +++ b/pipelines/mendix-mpr/lib/exec-resume.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""exec-resume.py + +`mxcli exec` stops at the first statement it cannot write; everything before it is in the target, +everything after it is not. Find the statement the `Error:` line names (its qualified name, or the +bare document name mxcli prints as `/.ObjectCollection` for a flow), move it to +with the verbatim error, and rewrite to the statements AFTER it, so the caller can carry +on instead of losing the rest of the module. Prints 1 when it could do that, 0 when the error names +no statement it can find (the caller then reports the module as failed). + +GenAI Showcase 2026-10-12: `Error: export mapping GenAICommons.EM_Reference: message definition +"GenAICommons.MD_Reference.Reference" not found — this project has no message definitions` (0.26.0 has +no message-definition document at all) would otherwise have cost GenAICommons its 170 remaining statements. +""" +import os, re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements, strip_deferred + +src, out, deferred = sys.argv[1], sys.argv[2], sys.argv[3] +text = open(src, encoding='utf-8', newline='').read() +stmts, tail = split_statements(text) +if tail.strip(): stmts.append(tail) +report = open(out, encoding='utf-8', errors='replace').read() +err = next((l for l in report.split('\n') if l.startswith('Error:')), '') +if not err: print(0); sys.exit(0) +head_re = re.compile(r'^(?:@[^\n]*\n|/\*\*.*?\*/\s*)*create or modify ([a-z][a-z -]*?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M | re.S) +grant_re = re.compile(r'^grant .*? on (?:[a-z ]+?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M) +qns = set(re.findall(r'\b([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', err)) +bare = set(re.findall(r'/([A-Za-z0-9_]+)\.ObjectCollection', err)) +hit = None +for i, s in enumerate(stmts): + m = head_re.search(s); g = grant_re.search(s.strip()) + name = m.group(2) if m and m.group(1) != 'module' else (g.group(1) if g else None) + if not name: continue + if name in qns or name.split('.')[1] in bare: + if g and not m and not err.startswith('Error: failed to grant') and 'grant' not in err: continue + hit = i; break +if hit is None: print(0); sys.exit(0) +# The statement that NAMES the document is not always the one that failed. `Error: microflow not found: +# X` + `hint: X is defined later in this script` comes from an EARLIER statement that references X +# (GenAI 2026-10-13: entity ExampleMicroflows.Ticket with `on before commit call …BCo_Ticket`); deferring +# X's definition and resuming after it silently dropped every statement between the two (E5: 5 +# documents). So: on a forward-reference error, defer the first statement that mentions X without +# defining it; and resume from the earliest statement that mentions any name in the error, not after +# the deferred one — `create or modify` re-runs of statements that already landed are idempotent. +fwd = re.search(r'\b([A-Za-z0-9_]+\.[A-Za-z0-9_]+) is defined later in this script|references [a-z ]+? ([A-Za-z0-9_]+\.[A-Za-z0-9_]+) before it is created', report) +names = qns | {q for q in re.findall(r'\b([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', report) if fwd and q == (fwd.group(1) or fwd.group(2))} +def mentions(s, q): return re.search(r'\b' + re.escape(q) + r'\b', s) +lo = min([i for i, s in enumerate(stmts) if any(mentions(s, q) for q in names) or + any(re.search(r'/' + re.escape(b) + r'\b', s) for b in bare)] + [hit]) +if fwd: + x = fwd.group(1) or fwd.group(2) + for i in range(lo, len(stmts)): + m = head_re.search(stmts[i]) + if mentions(stmts[i], x) and not (m and m.group(2) == x): hit = i; break +with open(deferred, 'a', encoding='utf-8', newline='') as f: + if os.path.getsize(deferred) == 0: f.write('mdl 1;\n') + f.write('-- deferred: exec refused this statement: %s\n%s\n' % (err[:400].replace('\n', ' '), strip_deferred(stmts[hit]))) +rest = [s for i, s in enumerate(stmts) if i >= lo and i != hit] +open(src, 'w', encoding='utf-8', newline='').write('mdl 1;\n' + '\n'.join(s for s in rest if s.strip() != 'mdl 1;') + '\n') +print(1) diff --git a/pipelines/mendix-mpr/lib/fix-describe-output.py b/pipelines/mendix-mpr/lib/fix-describe-output.py new file mode 100644 index 00000000..95d1c257 --- /dev/null +++ b/pipelines/mendix-mpr/lib/fix-describe-output.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""fix-describe-output.py — repair describe output that mxcli's own parser rejects. + +Only shapes that are unambiguously an emitter defect (mxcli 0.26.0, GenAI Showcase 2026-10-12): + 1. java action / javascript action parameter lines put the separating `,` AFTER the `-- comment`, + so the comment swallows it: ` Name: String not null -- The name.,` → `mismatched input + 'Name' expecting ')'`. Move the comma in front of the comment. 10 of 36 modules failed check on + this alone, and 25 more failed as a cascade. +Everything else stays byte-identical (CRLF inside literals is load-bearing). +""" +import re, sys +t = open(sys.argv[1], encoding='utf-8', newline='').read() +n = [0] +def comma(m): + n[0] += 1 + return m.group(1) + ',' + m.group(2).rstrip() + m.group(3) +t = re.sub(r'^( [A-Za-z0-9_"]+: [^\n]*?)( -- [^\n]*?),[ \t]*(\r?\n)', comma, t, flags=re.M) +open(sys.argv[2], 'w', encoding='utf-8', newline='').write(t) +sys.stderr.write('fix-describe-output: %d parameter commas moved out of comments\n' % n[0]) diff --git a/pipelines/mendix-mpr/lib/mdlsplit.py b/pipelines/mendix-mpr/lib/mdlsplit.py new file mode 100644 index 00000000..7d4d53f5 --- /dev/null +++ b/pipelines/mendix-mpr/lib/mdlsplit.py @@ -0,0 +1,56 @@ +"""Split a describe script into top-level statements, shared by topo-order-mdl.py and +compare-mdl.py. A statement ends at a column-0 line that ends in ';' — but only when that line +starts and ends OUTSIDE a string literal ('…', doubled quote escapes), a $$…$$ block (Java/JS +bodies, JSON samples) or a comment (-- to end of line, /* … */). The GenAI Showcase (increment 2) +has microflows returning multi-line JSON string literals whose inner lines sit at column 0 and +end in ';', and the plain column-0 rule cut those flows into fragments that were then reordered. +Text is passed through byte-for-byte: no newline translation (CRLF inside literals is load-bearing).""" +import re +OPENER = re.compile(r'^(create|grant|revoke|alter|drop|delete|mdl|@|/\*|--)') +# after an INDENTED terminator only a keyword, an annotation or a doc comment at column 0 closes the +# statement (` );` then `@Position(380, 130)` + `create … entity`: ChatCompletions_FunctionCalling) +STRONG = re.compile(r'^(create |grant |revoke |alter |drop |delete |@[A-Za-z]|/\*\*)') +def split_statements(text): + stmts, cur, state, pending = [], [], 'code', None # state: code | str | dollar | block + for ln in text.split('\n'): + start = state + # a candidate boundary closes only when the next non-blank line opens a statement: Studio Pro + # emits multi-line expressions with continuation lines at column 0 ("else $X, Y = $Z);") + if pending and ln.strip(): + if start == 'code' and (OPENER if pending == 'col0' else STRONG).match(ln): + stmts.append('\n'.join(cur)); cur = [] + pending = None + i, n, comment = 0, len(ln), False + while i < n: + c = ln[i] + if state == 'code': + if c == "'": state = 'str' + elif ln.startswith('$$', i): state = 'dollar'; i += 1 + elif ln.startswith('--', i): comment = True; break # comment to end of line + elif ln.startswith('/*', i): state = 'block'; i += 1 + elif state == 'str': + if c == "'": state = 'code' # '' escape toggles twice, net unchanged + elif state == 'dollar': + if ln.startswith('$$', i): state = 'code'; i += 1 + elif state == 'block': + if ln.startswith('*/', i): state = 'code'; i += 1 + i += 1 + cur.append(ln) + # the line may START inside a $$ body: java/javascript actions end with a column-0 `$$;` + # (GenAI Showcase: 15 java actions of GenAICommons merged into one "statement", and check's + # statement numbers no longer matched ours). The opener lookahead above still guards the + # multi-line-literal case. + # a json structure with a CUSTOM NAME map ends on an indented ` );` (GenAI Showcase, 2026-10-13: + # 4 modules split 1-9 statements short of mxcli's count, so check's `statement N:` mapped onto + # the wrong statement or past the end — FINDINGS E1b). An indented terminator closes only when + # the next non-blank line opens with a column-0 keyword. + if state == 'code' and not comment and ln.rstrip().endswith(';'): + pending = 'col0' if not ln[0].isspace() else 'indent' + if pending and cur: stmts.append('\n'.join(cur)); cur = [] + return stmts, '\n'.join(cur) + +def strip_deferred(s): + """Drop the leading `-- deferred: …` reason lines a statement carries from an earlier pass.""" + ls = s.split('\n') + while ls and (ls[0].startswith('-- deferred:') or not ls[0].strip()): ls.pop(0) + return '\n'.join(ls) diff --git a/pipelines/mendix-mpr/lib/quarantine-mdl.py b/pipelines/mendix-mpr/lib/quarantine-mdl.py new file mode 100644 index 00000000..3cd37d27 --- /dev/null +++ b/pipelines/mendix-mpr/lib/quarantine-mdl.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python3 +"""quarantine-mdl.py + +Move the statements that `mxcli check` rejected out of (rewritten in place) and append +them to , so the rest of the module can be executed instead of nothing at all. check is +all-or-nothing per file: one `-- line N:` syntax error or one `statement N:` reference error blocks +every other statement of the module (GenAI Showcase, 2026-10-12: 1 of 36 modules executed; 25 of +the 35 failures were cascades of 10 modules whose java actions mxcli emits with a trailing `,` inside +a `--` comment). Deferred statements are retried by the caller once the modules they reference exist; +what still fails is a ledger row with the verbatim check error, never a silent drop. + +Mapping (GenAI Showcase 2026-10-13, E1): `statement N:` reference errors map by number, verified against +the qualified name on the line and re-mapped by name when the numbered statement does not mention it; +`✗ … [CODE]` lint errors map by the document on the following `at Q.N` line, or by the `Q.N` in the +message when that line is empty. check counts statements without the leading `mdl 1;` (verified +against ChatCompletions_ComplexityReduction "statement 6" and GenAICommons "statement 233"). Each +deferred statement carries one `-- deferred:` line with the verbatim error. Prints the number of +statements moved, 0 when the check output names none it can map (the caller then falls back to +lib/exec-each.py, never drops the module). +""" +import os, re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements, strip_deferred + +src, chk, deferred = sys.argv[1], sys.argv[2], sys.argv[3] +text = open(src, encoding='utf-8', newline='').read() +stmts, tail = split_statements(text) +if tail.strip(): stmts.append(tail) +report = re.sub(r'\x1b\[[0-9;]*m', '', open(chk, encoding='utf-8', errors='replace').read()) # check colours its output +lines = report.split('\n') + +head_re = re.compile(r'^(?:@[^\n]*\n|/\*\*.*?\*/\s*)*create or modify ([a-z][a-z -]*?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M | re.S) +grant_re = re.compile(r'^(?:grant|revoke) .*? on (?:[a-z ]+?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M) +qn_re = re.compile(r'\b([A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*)\b') +defined, granted = {}, {} +for i, s in enumerate(stmts): + m = head_re.search(s) + if m and m.group(1) != 'module': defined.setdefault(m.group(2), []).append(i) + g = grant_re.search(s) + if g and not m: granted.setdefault(g.group(1), []).append(i) + +# statement index (0-based into stmts) of each 1-based check statement number: check does not count +# the leading `mdl 1;` (GenAICommons: "statement 233" = the 234th split statement, `mdl 1;` first), +# nor a comment-only tail. +numbered = [i for i, s in enumerate(stmts) if s.strip() != 'mdl 1;' and not all( + l.strip().startswith('--') or not l.strip() for l in s.split('\n'))] +bad, why = set(), {} +def mark(ks, reason): + for k in ks: + bad.add(k); why.setdefault(k, reason) +def detail(at): # the error line plus its indented `- ...` continuation lines + out = [lines[at].strip()] + for l in lines[at + 1:at + 8]: + if l.strip().startswith('- '): out.append(l.strip()) + else: break + return ' '.join(out) + +for at, l in enumerate(lines): + # (1) reference errors: `statement N: 'Q.N' …`. Map by NUMBER, verified against the first + # qualified name on the line; when the numbered statement does not mention it, map by NAME (the + # create, or the grant for a grant line) — a wrong number must never quarantine the wrong statement. + m = re.match(r'^\s*statement (\d+):(.*)$', l) + if m: + n, rest = int(m.group(1)) - 1, m.group(2) + idx = numbered[n] if 0 <= n < len(numbered) else None + q = next((x for x in qn_re.findall(rest) if x in defined or x in granted), None) + if idx is not None and (q is None or re.search(r'\b' + re.escape(q) + r'\b', stmts[idx])): + mark([idx], detail(at)) + elif q: + is_grant = re.match(r'\s*(grant|revoke)\b', rest) + mark((granted.get(q) if is_grant else defined.get(q)) or defined.get(q) or granted.get(q, []), detail(at)) + continue + # (2) lint errors: `✗ [CODE]`, the document on the NEXT line (`at Q.N`) — or, when that + # line is empty (`at ` with nothing after it), named in the message itself (`✗ snippet Q.N: …`, + # `✗ layout 'Q.N' …`). GenAI 2026-10-12: 7 modules dropped because neither form mapped (E1a). + if l.lstrip().startswith('\u2717') and not re.search(r'\d+ (reference |syntax )?error', l): + nxt = lines[at + 1].strip() if at + 1 < len(lines) else '' + cands = qn_re.findall(nxt[2:]) if nxt.startswith('at') else [] + q = next((x for x in cands + qn_re.findall(l) if x in defined), None) + if q: mark(defined[q], l.strip() + ('' if q in l else ' at ' + q)) + +# syntax errors come as `line N:M` — map the line onto the statement that holds it +if re.search(r'^Syntax errors found', report, re.M): + starts, line = [], 1 + for s in stmts: + starts.append(line); line += s.count('\n') + 1 + for at, l in enumerate(lines): + m = re.match(r'^\s*- line (\d+)', l) + if not m: continue + n = int(m.group(1)) + for i in range(len(stmts) - 1, -1, -1): + if starts[i] <= n: mark([i], 'syntax error: ' + l.strip()); break +# close over the module's own references: a statement that names a document defined by a deferred +# statement would fail the next round anyway (check skips references to objects created within the +# script, so deferring the definition exposes every caller) — MxGenAIConnector took 15 rounds of one +# layer each before this. The callers go to the deferred file too and come back in the same retry. +grown = True +while grown: + grown = False + names = {n for n, ks in defined.items() if any(k in bad for k in ks)} + for n in names: + pat = re.compile(r'\b' + re.escape(n) + r'\b') + for i, s in enumerate(stmts): + if i in bad or s.strip() == 'mdl 1;': continue + m = head_re.search(s); g = grant_re.search(s.strip()) + if (m and m.group(1) == 'module'): continue + if pat.search(s) and not (m and m.group(2) == n and m.group(1) in ('entity', 'non-persistent entity')): + bad.add(i); grown = True; why.setdefault(i, 'same-module closure: names %s, deferred with it' % n) +if not bad: + print(0); sys.exit(0) +keep = [s for i, s in enumerate(stmts) if i not in bad] +open(src, 'w', encoding='utf-8', newline='').write('\n'.join(keep) + '\n') +with open(deferred, 'a', encoding='utf-8', newline='') as f: + if os.path.getsize(deferred) == 0: f.write('mdl 1;\n') + for i in sorted(bad): + # one `-- deferred:` line per statement, never nested: a retried statement arrives with its + # previous reason attached, which used to be wrapped again on every pass (E3, 7x overcount) + s = strip_deferred(stmts[i]) + f.write('-- deferred: check rejected this statement: %s\n%s\n' % (why.get(i, '?')[:400].replace('\n', ' '), s)) +print(len(bad)) diff --git a/pipelines/mendix-mpr/lib/topo-order-mdl.py b/pipelines/mendix-mpr/lib/topo-order-mdl.py new file mode 100755 index 00000000..dff5b35b --- /dev/null +++ b/pipelines/mendix-mpr/lib/topo-order-mdl.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Reorder the statements of a `describe module … with all` script so every document is +created before any statement that references it. A unit is one `create or modify ` +statement plus the `grant … on … ` statements that follow it; a unit depends on every +other unit whose qualified name its text mentions. Kahn's sort, stable on the original order; +a cycle (recursion between flows, an entity whose event handler takes the entity) is reported +and its members are left in original order. Usage: topo-order-mdl.py in.mdl out.mdl + +Increment 1 ordered microflows only; increment 2 (GenAI Showcase, 48 modules) added nanoflows, +snippets called by pages, JSON structures used by mappings, and names with digits +(`AmazonS3Connector.Step2`), which the first pattern `[A-Za-z_]+` silently skipped.""" +import os, re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements +# newline='': annotation text carries CRLF inside string literals; translating it changes the flow and exec refuses the splice +stmts, tail = split_statements(open(sys.argv[1], encoding='utf-8', newline='').read()) +QN = r'[A-Za-z0-9_]+\.[A-Za-z0-9_]+' +head_re = re.compile(r'^(?:@[^\n]*\n|/\*\*.*?\*/\s*)*create or modify ([a-z][a-z -]*?) (' + QN + r')\b', re.M | re.S) +grant_re = re.compile(r'^grant .*? on (?:[a-z ]+?) (' + QN + r')\b', re.M) +# a unit = a create statement; a `grant … on ` statement is a unit of its own that depends on +# that document AND on every association of the module its member list names (bare member names: +# `grant read (Name, Credentials_Configuration) on entity M.Credentials` — exec rejects the grant when +# the association does not exist yet: "entity M.Credentials has no member(s) Credentials_Configuration", +# GenAI Showcase EchoConnector 2026-10-12; attaching grants to the entity put them before the association). +units = [] +for st in stmts: + m = head_re.search(st) + if m and m.group(1) != 'module': units.append((st, m.group(2), None)) + else: + g = grant_re.search(st.strip()) + units.append((st, None, g.group(1) if g else None)) +stmts = [u[0] for u in units] +idx_of = {} # name -> [unit indexes] (a name may carry two documents) +for k, u in enumerate(units): + if u[1]: idx_of.setdefault(u[1], []).append(k) +# a name carrying two documents (GenAI Showcase: microflow GenAICommons.ConsumedKnowledgeBase_GetCollectionOptions +# wraps a java action of the same name) must create the java/javascript action first — exec resolves +# the call in statement order and a flow created first makes the action "not found", so the flow is refused +LATE = ('microflow', 'nanoflow', 'page', 'snippet', 'layout') +for n, ks in idx_of.items(): + if len(ks) > 1: ks.sort(key=lambda k: (head_re.search(units[k][0]).group(1) in LATE, k)) +grant_units = [k for k, u in enumerate(units) if u[2]] +for k in grant_units: idx_of['grant#%d' % k] = [k] +movable = sorted(k for ks in idx_of.values() for k in ks) +pat = {n: re.compile(r'\b' + re.escape(n) + r'\b') for n in idx_of if not n.startswith('grant#')} +deps = {} +for n, ks in idx_of.items(): + body = '\n'.join(stmts[k] for k in ks) + deps[n] = {o for o in pat if o != n and pat[o].search(body)} + if n.startswith('grant#'): + on = units[ks[0]][2]; mod = on.split('.')[0] + deps[n] |= {on} if on in idx_of else set() + deps[n] |= {mod + '.' + w for w in re.findall(r'[A-Za-z_][A-Za-z0-9_]*', body) if (mod + '.' + w) in idx_of and (mod + '.' + w) != on} +movable = sorted(k for ks in idx_of.values() for k in ks) +# Kahn, stable on original order. A cycle (mutual recursion; an entity whose access rule names an +# association that names the entity) is broken by forcing the earliest remaining unit in original +# order and carrying on — never by abandoning the order of everything after it (that regressed the +# increment-1 fixture, where one entity/association cycle swallowed all 83 microflows). +order, placed, forced = [], set(), [] +remaining = sorted(idx_of, key=lambda n: idx_of[n][0]) +while remaining: + ready = [n for n in remaining if deps[n] <= placed] + if not ready: + ready = [remaining[0]]; forced.append(remaining[0]) + for n in ready: + order.append(n); placed.add(n); remaining.remove(n) +if forced: sys.stderr.write('cycle: forced ahead of an unmet reference: %s\n' % ', '.join(forced)) +seq = [k for n in order for k in idx_of[n]] # unit indexes in the new order +moved = sum(1 for a, b in zip(seq, movable) if a != b) +out = list(stmts) +for slot, k in zip(movable, seq): + out[slot] = stmts[k] +open(sys.argv[2], 'w', encoding='utf-8', newline='').write('\n'.join(out) + ('\n' + tail if tail.strip() else '\n')) +sys.stderr.write('statements=%d units=%d repositioned=%d\n' % (len(stmts), len(movable), moved)) diff --git a/pipelines/mendix-mpr/rehost-extract.sh b/pipelines/mendix-mpr/rehost-extract.sh new file mode 100755 index 00000000..5f616510 --- /dev/null +++ b/pipelines/mendix-mpr/rehost-extract.sh @@ -0,0 +1,255 @@ +#!/usr/bin/env bash +# rehost-extract.sh — read an existing Mendix .mpr, emit each own module as MDL, replay it into a +# fresh target .mpr of the same Mendix version, and write the fidelity ledger. +# +# pipelines/mendix-mpr/rehost-extract.sh [Module ...] +# +# /captures/ verbatim mxcli output (golden fixtures: list modules, show , +# project-tree.json, check/exec logs, timings.tsv per module, phases.tsv per phase) +# /mdl/ .mdl (source describe), .ordered.mdl (what was executed), +# target-.mdl (describe of the target afterwards), _settings/_userroles +# /fidelity-ledger.tsv one row per source document (denominator in the header) +# +# Modules default to every module `list modules` shows with an empty Source column except System. +# The target must already exist (`mxcli new --version --theme none`); this script +# creates nothing but model content. It does NOT run the build gate — run project-bin/exec.sh's +# mxbuild gate or `~/.mxcli/mxbuild//modeler/mx check ` afterwards. +# +# Field run (2026-10-12, mxcli v0.26.0, Linux container): ako/mxcli-banking RRNetBanking.mpr +# (Mendix 11.13.0, modules Banking 151 documents + MyFirstModule 4) -> `mxcli new --version 11.13.0`. +# 155/155 catalog documents emitted, 154 identical on target describe, 1 documentation drift, +# `mx check` and `mxbuild --target=deploy` clean. Everything in here was exercised on that run: +# - the whole-module emitter is the MDL statement `describe module with all` run through -c; +# the `mxcli describe module ` SUBCOMMAND emits only the module and its roles (0.26.0). +# - describe emits microflows in catalog order, so `check` reported 14 MDL-ORDER01 (callee +# created after caller) and exec refused the file; lib/topo-order-mdl.py reorders flow units +# (create + its grant execute statements) and the reordered file checked clean. +# - `describe module … with all` omits image collections (MyFirstModule.Images) — described +# separately below so the ledger has the row. +# - demo users describe with Password: '***' and exec refuses the placeholder; project security +# level describes as a report, not MDL. Both are ledger hand-off rows, never executed here. +# Not exercised: nanoflows, Java/JavaScript actions, snippets, workflows, REST/OData, marketplace +# install (the blank template already carried the source's seven modules at the same versions). +set -euo pipefail + +SRC="${1:?usage: rehost-extract.sh [Module ...]}" +TGT="${2:?target.mpr}" +OUT="${3:?out-dir}" +shift 3 +HERE="$(cd "$(dirname "$0")" && pwd)" +MXCLI="${MXCLI:-$(command -v mxcli || true)}" +[ -x "$MXCLI" ] || { echo "mxcli not found: set MXCLI=/path/to/mxcli" >&2; exit 2; } +PY="${PYTHON:-python3}" +mkdir -p "$OUT/captures" "$OUT/mdl" +mx() { "$MXCLI" "$@" 2>&1 | grep -v 'vibe-coded PoC'; } # one warning line per call, drop it +log() { printf '%s\n' "$*" >&2; } + +T_START=$("$PY" -c 'import time; print("%.3f" % time.time())') +phase() { "$PY" -c 'import sys, time; t = time.time(); print("%s\t%.1f" % (sys.argv[1], t - float(open(sys.argv[2]).read() or sys.argv[3])))' "$1" "$OUT/captures/.phase-mark" "$T_START" >> "$OUT/captures/phases.tsv"; "$PY" -c 'import time; print("%.3f" % time.time())' > "$OUT/captures/.phase-mark"; } +printf 'phase\tseconds\n' > "$OUT/captures/phases.tsv"; : > "$OUT/captures/.phase-mark" +# ---- 1. inventory (golden captures) --------------------------------------------------------- +mx -p "$SRC" list modules > "$OUT/captures/list-modules.txt" +"$MXCLI" -p "$SRC" project-tree --json 2>/dev/null > "$OUT/captures/project-tree.json" +if [ $# -eq 0 ]; then + # own module = empty Source column, not System + mapfile -t MODULES < <(awk -F'|' 'NR>3 && NF>3 { gsub(/ /,"",$2); gsub(/ /,"",$3); if ($3=="" && $2!="System" && $2!="") print $2 }' "$OUT/captures/list-modules.txt") +else + MODULES=("$@") +fi +log "own modules: ${MODULES[*]}" +for M in "${MODULES[@]}"; do + for t in entities associations enumerations microflows nanoflows pages snippets javaactions; do + mx -p "$SRC" show "$t" "$M" | grep -v '^Connected to' > "$OUT/captures/show-$t-$M.txt" + done +done + +phase inventory +# ---- 2. extract ------------------------------------------------------------------------------- +# describe_module : `describe module … with all` plus every describable +# document of the module that the whole-module emitter omits, described one by one from the +# project tree. On 0.26.0 `with all` omits image collections (increment 1) and, on the GenAI +# Showcase (increment 2), nanoflows, snippets, JavaScript actions, JSON structures, import/export +# mappings, task queues, scheduled events and some pages. Used for the source and, in step 4, for +# the target, so both sides carry the same document set. +describe_module() { + local mpr="$1" M="$2" out="$3" + # `|| true`: on the target side the module may not exist at all (its check failed) — the ledger says so + "$MXCLI" -p "$mpr" -c "describe module $M with all" 2>/dev/null | grep -v 'vibe-coded PoC' > "$out" || true + # 0.26.0 cuts `-c` output at 10 000 lines without saying so (AgentCommons and ConversationalUI on the + # GenAI Showcase both ended mid-page at line 10 001). Keep only complete statements; the tree fill-in + # below then describes every document the cut swallowed, one by one. + "$PY" -I - "$out" "$HERE/lib" <<'PYEOF' >&2 +import sys; sys.path.insert(0, sys.argv[2]) +from mdlsplit import split_statements +text = open(sys.argv[1], encoding='utf-8', newline='').read() +stmts, tail = split_statements(text) +if tail.strip(): + open(sys.argv[1], 'w', encoding='utf-8', newline='').write('\n'.join(stmts) + '\n') + print('describe module truncated at %d lines (incomplete last statement dropped: %s)' % (text.count('\n'), tail.strip().split('\n')[0][:80])) +PYEOF + "$MXCLI" -p "$mpr" project-tree --json 2>/dev/null > "$out.tree.json" + "$PY" -I - "$M" "$out" "$out.tree.json" <<'PYEOF' | while read -r t q; do +import json,re,sys +M, out, tree = sys.argv[1], sys.argv[2], sys.argv[3] +have = set(re.findall(r'^create or modify [a-z -]+? (' + re.escape(M) + r'\.[A-Za-z0-9_]+)', open(out, encoding='utf-8', newline='').read(), re.M)) +def walk(n): + for x in n: + q = x.get('qualifiedName', '') + if x.get('describable') and q.startswith(M + '.') and q not in have and x.get('type') not in ('modulerole',): + print(x['type'], q) + walk(x.get('children', [])) +walk(json.load(open(tree))) +PYEOF + "$MXCLI" -p "$mpr" describe "$t" "$q" 2>/dev/null | grep -v 'vibe-coded PoC' | sed '1{/^mdl 1;$/d}' >> "$out" + done + rm -f "$out.tree.json" +} +# timing in Python: `date +%s.%N` and `bc` do not exist on stock macOS +now() { "$PY" -c 'import time; print("%.3f" % time.time())'; } +since() { "$PY" -c 'import sys, time; print("%.1f" % (time.time() - float(sys.argv[1])))' "$1"; } +diff_s() { "$PY" -c 'import sys; print("%.1f" % (float(sys.argv[2]) - float(sys.argv[1])))' "$1" "$2"; } +TIMES="$OUT/captures/timings.tsv" +printf 'module\tdescribe_s\ttopo_s\tcheck_s\texec_s\tdescribe_target_s\n' > "$TIMES" +declare -A T_DESC T_TOPO T_CHECK T_EXEC T_TDESC +# REHOST_REUSE_MDL=1 skips the source describes when /mdl/.mdl already exists (a rerun after +# fixing the check/exec side; the describes are the slow part — 36 modules took 30 min on the GenAI Showcase). +for M in "${MODULES[@]}"; do + t0=$(now) + if [ "${REHOST_REUSE_MDL:-0}" = 1 ] && [ -s "$OUT/mdl/$M.mdl" ]; then log "reusing mdl/$M.mdl"; else describe_module "$SRC" "$M" "$OUT/mdl/$M.mdl"; fi + t1=$(now) + "$PY" -I "$HERE/lib/topo-order-mdl.py" "$OUT/mdl/$M.mdl" "$OUT/mdl/$M.ordered.mdl" 2>>"$OUT/captures/describe-$M.stderr" + "$PY" -I "$HERE/lib/fix-describe-output.py" "$OUT/mdl/$M.ordered.mdl" "$OUT/mdl/$M.ordered.mdl" 2>>"$OUT/captures/describe-$M.stderr" + T_DESC[$M]=$(diff_s "$t0" "$t1"); T_TOPO[$M]=$(since "$t1") +done +# module order: replay the modules so that each is created after every listed module it references +# (qualified-name mention in its describe output); REHOST_KEEP_ORDER=1 keeps the order given. A cycle +# between modules is reported and left in the given order — exec then fails on the forward reference +# and the ledger says so. +if [ "${REHOST_KEEP_ORDER:-0}" = 0 ]; then + mapfile -t MODULES < <("$PY" -I - "$OUT/mdl" "${MODULES[@]}" <<'PYEOF' +import re,sys +d, mods = sys.argv[1], sys.argv[2:] +body = {m: open('%s/%s.mdl' % (d, m), encoding='utf-8', newline='').read() for m in mods} +deps = {m: {o for o in mods if o != m and re.search(r'\b' + re.escape(o) + r'\.[A-Za-z0-9_]', body[m])} for m in mods} +order, placed, rest = [], set(), list(mods) +while rest: + ready = [m for m in rest if deps[m] <= placed] + if not ready: + sys.stderr.write('module cycle among: %s (left in catalog order)\n' % ', '.join(rest)); order += rest; break + for m in ready: order.append(m); placed.add(m); rest.remove(m) +print('\n'.join(order)) +PYEOF + ) + log "replay order: ${MODULES[*]}" +fi +"$MXCLI" -p "$SRC" describe settings Settings 2>/dev/null | grep -v 'vibe-coded PoC' > "$OUT/mdl/_settings.mdl" +mx -p "$SRC" -c "show user roles" > "$OUT/captures/show-userroles.txt" +: > "$OUT/mdl/_userroles.mdl" +awk -F'|' 'NR>3 && NF>3 { gsub(/ /,"",$2); if ($2!="") print $2 }' "$OUT/captures/show-userroles.txt" | while read -r r; do + "$MXCLI" -p "$SRC" describe userrole "$r" 2>/dev/null | grep -v 'vibe-coded PoC' >> "$OUT/mdl/_userroles.mdl" +done + +phase describe_topo_userroles +# ---- 3. check, then exec into the target (modules in replay order) --------------------------- +# check is all-or-nothing per file, so a statement it rejects is moved to mdl/.deferred.mdl +# (lib/quarantine-mdl.py) and the rest of the module is executed; the deferred statements are retried +# below until no retry makes progress, because most rejections are forward references into a module +# that is created later (cross-module cycles) or into a statement deferred for a syntax error. +status=0 +check_exec() { # check_exec