diff --git a/CHANGELOG.md b/CHANGELOG.md index e19d759e..624c5897 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- new(pipelines): **`pipelines/mendix-mpr/` — the Rehost extractor for an existing Mendix app (split from #293, 3/6).** `rehost-extract.sh` describes each own module with mxcli, reorders the statements so callees come first, replays them into a fresh target `.mpr` of the same Mendix version (deferring and retrying what `check`/`exec` refuse, never dropping a module), and writes a fidelity ledger with one row per source document and the denominator in its header; `export-module-mpk.py` writes a module package from an MPR v1 app for Mendix 9 sources whose `mx` cannot export; by default it empties demo users, user roles, admin password, per-configuration constant values, database credentials, certificates and navigation profiles in the project documents, while the module's own units (constant default values included) are copied byte for byte; any unknown `--` option is refused (exit 2). Code only, no samples; the script now runs without bash-4 features. After review: module constants' default values still travel in the package by default — the path the field runs installed — and a new `--scrub-constant-defaults` empties them and prints how many (unverified in Studio Pro: no package built with it has been installed and checked yet); the README states exactly what is and is not scrubbed; both scripts refuse source == output/target (exit 2) before writing; the own-module parse moved to `lib/own-modules.awk` and no longer assumes a fixed number of lead lines, with a fixture over verbatim `list modules` captures (`test-rehost-list-modules.sh`, 5 checks; the old script fails 2). Added to CLAUDE.md's Pipelines table. Field runs on public apps: a banking sample (Mendix 11.13, 155 documents) rehosted build-green, 154/155 identical; a GenAI showcase app (Mendix 11.12, 2776 documents) landed 36/36 modules but is not build-green (42 × CE1613); module packages from a field-service template (Mendix 9.24), exported on the default path (constant defaults copied), installed into 10.24 and 11.13. macOS and the two-tree layout not verified. — MendixMau - new(cut-ledger): **`bin/cut-ledger.sh` writes the cut ledger of one decompose slice — one row per crossing edge with its resolution (resolved / stub / handoff / retire / platform) and evidence — and exits 1 while any edge has none** (split from [#293](https://github.com/MendixMau/mxcli-project-toolkit/pull/293), 2/6). Reads the source app's catalog `refs` table (or a `--refs` TSV), a slice file (modules and single documents), first-match rules with mandatory evidence, `mx check` CE1613 names the catalog never recorded (check-only rows), the module exporter's JSON (modules it counts with no ledger row are printed), and verbatim `--extra` rows. #293 shipped it with no fixture; new `tests/wave2/test-cut-ledger.sh` (44 checks) runs on a captured line subset of #293's field run (43 of 346 crossing refs, one key per rule) and reproduces that run's own 57 ledger rows column for column, the pre-fix exporter note, UNRESOLVED → exit 1, refusals → exit 2 (now also an unreadable `--rules`/`--catalog`/`--exporter-json`, which escaped as exceptions with the UNRESOLVED code 1, and a `--refs` line with the wrong field count, which was silently dropped), and `--catalog` at single-tree and `app/` paths. Header corrected to the run's real counts (346 refs → 308 rows; it said 199). — MendixMau (#293 field run on a public template app) - new(cut-score), fix(app-facts): **`bin/cut-score.sh` scores a proposed split of an existing app into several apps; `app-facts.sh` no longer needs the sqlite3 CLI** (split from [#293](https://github.com/MendixMau/mxcli-project-toolkit/pull/293), 1/6). Given `--cut "A,B|C|D"` (or one app per level of `app-layer-map.json`) it lists every module edge that crosses an app boundary from `analysis/app-facts/dependencies.json`, names the lane each ref kind would need (OData external entity, published action or REST, or none), and sums a clean / costly / blocked verdict per partition with the denominator printed ("7 of 7 own edges"); an aggregate over a foreign entity is read from `.mxcli/catalog.db` (single-tree or `app/`), and reads NOT DETECTED with a reason when it cannot be. Adapted to master: the catalog's `build_mode` label is recorded, not tested (#259 — v0.24+ labels a full catalog `source`), and an empty `activities_data` reads NOT DETECTED instead of "no aggregates". `app-facts.sh` runs its guard queries through Python's sqlite3 when the CLI is missing (absent on some cloud containers and always on Git Bash), stripping the `\r` native Windows Python prints. Rows whose aggregate cannot be detected name the actual reason (no catalog / empty table / unreadable). `tests/wave2/test-cut-score.sh` 70/70 on a captured four-module fixture (14 new: label, empty table, row reasons, own modules derived from edges, both layouts); `test-app-report.sh` 109/109; `test-catalog-edge-guard.sh` 22/22, new case forcing the Python path with CRLF output (fails without the `tr`). — MendixMau (#293 field runs on public template apps) - new(sync-project): **`bin/sync-project.sh --only-bin [--only-bin …]` installs only the named `project-bin/` scripts into the project's `bin/` and touches nothing else; `--upgrade-bin` is now documented as what it is — a full sync that may also overwrite a modified copy.** A field run used `--upgrade-bin exec.sh --upgrade-bin _common.sh --upgrade-bin test-scratch.sh` as "install just these scripts" and got a full sync: 244 tracked files changed, one deleted, ~30 added, git hooks reinstalled. `--only-bin` decides everything before writing anything (a name outside the install manifest, NOINSTALL scripts included, a path-like or empty name, a locally modified copy without `--upgrade-bin `/`--force`, or an older copy the full sync's `_common.sh` refresh hold would keep, exits 1 with the project untouched; names are never glob-expanded), backs up an overwritten copy, and prints every file it wrote. The remedy lines in `app-restart.sh`, `test-scratch.sh`, `test-stack-up.sh`, `doctor.sh` and `harvest-learnings.sh` now name `--only-bin`. Fixture `tests/wave2/test-sync-only-bin.sh` (40 checks, single- and two-tree): one `git status` change, hooks and skills untouched, while a full `--upgrade-bin` sync of the same project installs other scripts and the hooks. — field run, issue #390 diff --git a/CLAUDE.md b/CLAUDE.md index 4dfe01de..5e1188ed 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,6 +67,7 @@ Migration assessment (`assess-migration`) is **bundled with mxcli** (`.ai-contex | OutSystems | `pipelines/outsystems/` | `cd pipelines/outsystems/pipeline && npm install` — see its `README.md` / `pipeline-guide.html` | | Java + Angular / Spring Boot | `pipelines/java-angular/` | `cd pipelines/java-angular/pipeline && npm install` — see its `README.md` | | Node/Express + React | `pipelines/node-express-react/` — **regex-based, proven on one source shape only; read its README's "Known gap" sections first** | `cd pipelines/node-express-react/pipeline && npm install` | +| Mendix (an existing `.mpr`, rehost or move a module) | `pipelines/mendix-mpr/` — **build-green on a small app only; read its README's "Field runs" first** | `MXCLI=/path/to/mxcli pipelines/mendix-mpr/rehost-extract.sh ` — Python 3 stdlib, no `npm install`; see its `README.md` | `node_modules/` is gitignored — `npm install` locally per pipeline. Set local paths in `pipelines//pipeline/config.json`; **never commit real local paths**. Curated sample outputs live under each pipeline. diff --git a/pipelines/mendix-mpr/README.md b/pipelines/mendix-mpr/README.md new file mode 100644 index 00000000..2ae74719 --- /dev/null +++ b/pipelines/mendix-mpr/README.md @@ -0,0 +1,124 @@ +# Mendix MPR pipeline — the Rehost extractor + +**Source platform:** an existing Mendix app (`.mpr`, Mendix 9 and above, MPR v1 or v2). +**What it does:** read a model with `mxcli describe`, regenerate it as MDL into a fresh target +app, and write a **fidelity ledger** (one row per source document; the denominator in the header) +instead of claiming 100 %. + +This is the only pipeline whose extractor reads a Mendix model. Downstream of the ledger nothing +here is Mendix-source-specific: the slice loop of the Decompose track reuses it to move a +module into a new app, and Stages 2–6 of the runbook apply unchanged. + +## Run + +``` +MXCLI=/path/to/mxcli pipelines/mendix-mpr/rehost-extract.sh [Module ...] +``` + +- The target must exist: `mxcli new --version --theme none`. The script + writes model content only; run the mxbuild gate (`project-bin/exec.sh`, or + `~/.mxcli/mxbuild//modeler/mx check`) afterwards. +- Modules default to every own module (empty Source column in `list modules`, System excluded; + parsed by `lib/own-modules.awk`, fixture `tests/wave2/test-rehost-list-modules.sh` over verbatim + captures). No own module and no module argument: exit 2. A target that resolves to the source + file: exit 2, before anything is written. + Marketplace modules are recorded by name and version on the ledger, never regenerated. +- Output: `captures/` (verbatim mxcli output — the golden fixtures), `mdl/` (describe output, + the topologically ordered file that was executed, the target's describe for the diff), + `fidelity-ledger.tsv`. + +`lib/topo-order-mdl.py` reorders the flow statements of a `describe module … with all` script so +every callee is created before its caller (a flow and its `grant execute` lines move as one unit). +`lib/compare-mdl.py` normalises two describe outputs for the per-document identical/drift verdict. +Both read and write with `newline=''`: Studio Pro stores annotation text with CRLF inside MDL string +literals, and a tool that normalises it changes the flow and `exec` refuses the splice. + +## Field runs + +All on public apps, Linux container, mxcli v0.26.0 unless noted. **Not verified:** macOS (the +script avoids bash-4 features and GNU-only `date`, but no run happened there) and the two-tree +layout (`.mpr` under `app/`) — the script takes explicit `.mpr` paths, so layout should not +matter, but no run proved it. + +| Date | Source (by app type) | Result | +|---|---|---| +| 2026-10-12 | Banking sample app (Mendix 11.13.0; one own module of 151 documents + MyFirstModule 4) → `mxcli new --version 11.13.0` | 155/155 documents emitted, 154 identical on target describe, 1 doc-comment drift; `mx check` 0 errors, mxbuild BUILD SUCCEEDED, `diff --format struct` empty. 35 s end to end | +| 2026-10-12 | Field-service template (Mendix 9.24.2, MPR v1, 64 MB) — **read probe only** | `list modules` 1.6 s, `graph-report` 18 s | +| 2026-10-12 | GenAI showcase app (Mendix 11.12.0; 48 modules: 32 own + 15 marketplace; 456 entities, 1622 microflows, 180 pages, 204 Java actions) → `mxcli new --version 11.12.0` | **Measured, not build-green.** 36 modules regenerated, 36 min (describes reused). Ledger over 2776 documents: 579 identical, 271 drift, 1993 failed-exec, 43 not-emitted, 4 not-expressible, 2 target-extra. 1520 failures were 7 whole modules dropped because check errors could not be mapped to a statement (fixed since: `lib/exec-each.py`). Target then failed `mx check` (14 × CE1613) and mxbuild | +| 2026-10-13 | Same GenAI showcase app on an mxcli nightly build, with the statement-mapping fixes | **36/36 modules landed, not build-green.** 1537 identical, 858 drift, 482 failed-exec, 4 not-emitted, 9 not-expressible. The target does not load (`ArgumentNullException` on one snippet parameter mxcli stored wrongly); with it replaced, `mx check` 42 × CE1613 in 5 classes. 2 h 36 min with reused describes. Journeys not run (needs live LLM credentials) | +| 2026-10-12 | Field-service template — **MPK route** (`export-module-mpk.py`): three utility/visualisation modules → 10.24 and 11.13 | All installed; every residual `mx check` error a crossing edge to a module the target lacks; one module with its edges satisfied: `mx check` 0, mxbuild BUILD SUCCEEDED on 11.13 after 3 Java edits | +| 2026-10-12 → 13 | Field-service template — **slices moved out of the monolith** (a native notes module; an asset/parts group of 4 modules; a dispatch/workforce group of 5) | Each slice `mx check` 0 and BUILD SUCCEEDED on 11.13; seams over published OData + REST, proven by Playwright journeys across both apps. These runs exercise the whole decompose loop, not this extractor alone | + +Not yet exercised: workflows, REST/OData services, and the MDL regenerate step on a Mendix 9/10 +source (the read works; the native slice moved into Mendix 11 by MPK, not by regeneration). + +## The MPK route — `export-module-mpk.py` (moving a module without regenerating it) + +``` +python3 -I pipelines/mendix-mpr/export-module-mpk.py [--no-widgets] [--scrub-constant-defaults] [--json] +mxcli marketplace install --file -p # new modules only; no PAT +``` + +The exporter writes a module package in Mendix's own format: a zip with `package.xml` and a +`project.mpr`. That `project.mpr` holds the source schema and `_MetaData` verbatim, the module's +unit subtree byte for byte, the project root, the conversion unit and +four project documents. + +**Module constant defaults travel in the MPK** unless you pass `--scrub-constant-defaults`. A key +or password kept as a constant's default value is in the package as plain text by default. + +**What is scrubbed, exactly** (each scrubbed unit gets a recomputed ContentsHash): + +| Unit | Emptied | Kept | +|---|---|---| +| Project security | demo users, user roles, admin password, admin and guest role; demo users and guest access switched off; security level set to `CheckNothing` (off) | everything else in the document | +| Project settings | per-configuration constant values, custom settings, database URL / user / password; certificates; after-startup, before-shutdown and health-check microflow names | other settings | +| Navigation | every navigation profile | the document shell | +| The module's own constants | **nothing by default** (the field-proven path). With `--scrub-constant-defaults`: `DefaultValue` of every constant; the run prints how many (`constant defaults emptied: N`). **Unverified in Studio Pro: not yet installed and checked** | everything by default; with the flag, every field but `DefaultValue` | + +**Not scrubbed:** module constant defaults (without the flag), string literals inside microflows, pages, Java/JavaScript code, `resources/` +files and REST/OData locations: a secret written there travels in the package. Read the module +before handing the `.mpk` on. With `--scrub-constant-defaults`, an emptied non-String +constant (Integer, Boolean …) will need a value in the target; how `mx check` reports an empty one +is not yet measured. + +The script refuses (exit 2), before writing anything, when the output path is the source `.mpr` +and when it is given an option it does not know (a typo such as `--scrub-constants` would +otherwise export with every constant default in it). +The zip also carries the module's `javasource/`, +`javascriptsource/`, `themesource/` and `resources/` folders, the userlib jars that have a +`..RequiredLib` marker, and the widget `.mpk` files its pages reference. The script header +lists what it cannot know. It refuses MPR v2 sources and schemas that differ from the reference. + +**Version window (Mendix's, not ours).** `mxcli marketplace install` imports through the target's +`mx module-import`: + +- 9.24's `mx` has no `module-import`. +- 11.13 refuses packages older than 10.21. +- So a 9.x → 11.x move has two hops: `mx convert` a copy of the source to 10.24, export, then install. + +On Mendix 10 and 11, `mx create-module-package ` is Mendix's own headless export. +It refuses modules that have errors, and it **regenerates every unit and entity ID**. This +exporter keeps them, and mxcli's install keeps them too: measured from 9.24 through 10.24 to 11.13. + +**Field run 2026-10-12, field-service template (Mendix 9.24.2):** + +- Three modules (a feature-toggle module, a utilities module, a charts module) each installed + into 10.24 and into 11.13. +- Every `mx check` error (2 / 7 / 134) was a crossing edge to a module the target lacks. There + were 0 version-conversion errors and 0 package-format errors. +- The utilities module with its edges satisfied (3 Marketplace modules, 1 constant in MDL) reached `mx check` + 0 errors. After three Java hand-off edits, mxbuild BUILD SUCCEEDED on 11.13.0. The edits: + `this.context()` is a Mendix 9 API; two orphan action files used `CustomJavaAction`; one jar's + RequiredLib marker names a former module. +- The Java bodies arrive byte-identical. + +Captures of these runs are not committed here (the pipeline ships code only). For Mendix 9 `mx` on Linux, the container needs `libgdiplus` and JDK 17. + +## mxcli findings from the first run (mxcli v0.26.0) + +The subcommand `mxcli describe module ` emits only the module and its roles; the whole-module +emitter is the MDL statement `describe module with all` via `-c`. Describe output is in catalog +order and the tool's own `check` refuses it (MDL-ORDER01). Image collections are omitted by +`with all`. `describe userrole` drops `CheckSecurity: false`. Demo-user passwords describe as +`'***'` and project security level describes as a report, not MDL — both are ledger hand-off rows. diff --git a/pipelines/mendix-mpr/export-module-mpk.py b/pipelines/mendix-mpr/export-module-mpk.py new file mode 100644 index 00000000..9c303c89 --- /dev/null +++ b/pipelines/mendix-mpr/export-module-mpk.py @@ -0,0 +1,550 @@ +#!/usr/bin/env python3 +"""export-module-mpk.py — headless "export module to .mpk" for an MPR v1 (single SQLite) app. + +usage: export-module-mpk.py [--no-widgets] [--scrub-constant-defaults] [--json] + +WHAT IT DOES + Writes a Mendix *module package* in the shape Studio Pro / Marketplace module packages have + (studied on Marketplace "Encryption" 9.0.3, built with 9.18.0): a zip holding `package.xml` + (…) and a + `project.mpr` that is a new SQLite database with + - the source's own CREATE TABLE statements, verbatim (same schema, no translation), + - the source's `_MetaData` row, verbatim (ProductVersion stays the source's version), + - the `Projects$Project` root unit, the `Projects$ProjectConversion` unit and the four + project documents (mx module-import refuses a package without them) — SystemTexts + verbatim; ProjectSecurity, ProjectSettings and Navigation *scrubbed*: demo users and their + passwords, user roles, admin password, guest role, configuration constant values / custom + settings / database credentials, certificates, the after-startup / before-shutdown / + health-check microflow names and the navigation profiles are emptied, and those units' + ContentsHash recomputed (sha256 of Contents, base64 — verified on all 4783 units of the field-service template), + - the module unit (`ContainmentName='Modules'`, BSON Name == ) and every unit below + it by ContainerID, every column copied byte for byte (UnitID, ContentsHash, Contents …). + That includes the module's own constants (`Constants$Constant`) WITH their DefaultValue: a + key or password kept as a constant default travels in the package. `--scrub-constant-defaults` + empties every DefaultValue, recomputes the ContentsHash and prints the count — UNVERIFIED in + Studio Pro: no package built with it has yet been installed and checked (`mx check`). An + emptied non-String default (Integer, Boolean …) needs a value set in the target, + plus the module's files, matched on the lower-cased module name the way Mendix lays them out: + `javasource//` (minus `proxies/`, which the build regenerates), `javascriptsource//`, + `themesource//`, `resources//`, every `userlib/` that has a + `..RequiredLib` marker (with the marker — the same rule as `mx + create-module-package --filter-required-libs`), and (unless --no-widgets) every + `widgets/*.mpk` whose widget id is referenced by a unit of the module. + +WHY THE SCRUB + Reference module packages carry *blank* project documents. The monolith's own carry demo-user + passwords, constant values and database settings, which must not travel inside a module + package. A first version omitted the four documents; `mx module-import` (10.24) then failed + with "Sequence contains no elements" (exit 117) on every package, and succeeded with all four + present (each one alone, or any three, still failed). The scrub keeps the documents and + empties the secrets. It refuses to rewrite a document whose BSON does not round-trip + byte-identically. + +WHAT IT CANNOT KNOW + - Whether the target has the modules this module references (the cut's crossing edges). It + counts them and prints them; the target's `mx check` is what proves them. + - Whether the target's Mendix version accepts the package. It does not convert BSON: the + package is exactly as old as the source. (Field result, 2026-10-12: `mx module-import` + 11.13 refuses packages older than 10.21; 9.24's mx has no module-import at all.) + - Managed (Gradle) Java dependencies, jars without a RequiredLib marker, and JavaScript + action npm dependencies outside `javascriptsource//`. + +WHEN TO USE SOMETHING ELSE + On Mendix 10.x/11.x `mx create-module-package ` is Mendix's own headless + export (MxBuild toolset). Prefer it. This script exists for MPR v1 sources whose mx has no such + verb (Mendix 9.x). It refuses — rather than guessing — an MPR v2 source (mprcontents/), a + schema that is not the reference schema, a module name that is missing or ambiguous. + +Python stdlib only. Read-only on the source (opened with mode=ro). +Field run 2026-10-12: a public field-service template app (Mendix 9.24.2, MPR v1, 64 MB); three +modules exported and installed into Mendix 10.24 and 11.13 (results in this pipeline's README.md). +""" +import base64 +import hashlib +import json +import os +import re +import sqlite3 +import struct +import sys +import zipfile +import xml.etree.ElementTree as ET +from xml.sax.saxutils import quoteattr + +REF_UNIT_COLS = ["UnitID", "ContainerID", "ContainmentName", "TreeConflict", + "ContentsHash", "ContentsConflicts", "Contents"] +REF_META_COLS = {"_ProductVersion", "_BuildVersion", "_SchemaHash", "_DisableAutoMprV2Upgrade"} +SHELL_TYPES = {"Projects$Project", "Projects$ProjectConversion"} +# mx module-import refuses a package without all four ("Sequence contains no elements", exit 117, +# Mendix 10.24 — field run 2026-10-12). SystemTexts is copied verbatim; the three that carry +# secrets or monolith-wide references are scrubbed (see scrub_*). +PROJECT_DOC_TYPES = {"Security$ProjectSecurity", "Settings$ProjectSettings", + "Navigation$NavigationDocument", "Texts$SystemTextCollection"} + + +class Refuse(Exception): + pass + + +class SamePath(Refuse): + """Source and output are the same file: a usage error (exit 2), never a write.""" + + +# --- minimal BSON reading: top-level fields only, enough for $Type / Name / WidgetId --------- +def _bson_top(b): + """Return the top-level string fields of a BSON document (others skipped).""" + out = {} + size = struct.unpack_from(" 1: + raise Refuse(f"module name {module!r} is ambiguous ({len(match)} units)") + if len([u for u in shell if _bson_top(by_id[u][2]).get("$Type") == "Projects$Project"]) != 1: + raise Refuse("expected exactly one Projects$Project root unit") + _, mod_uid, mod_top = match[0] + subtree, stack = [], [mod_uid] + while stack: + u = stack.pop() + subtree.append(u) + stack.extend(children.get(u, [])) + return shell, subtree, mod_top, {m[0] for m in modules}, by_id + + +def crossing_refs(by_id, subtree, module, all_modules): + """Qualified names Module.X in the subtree's string fields pointing at another module. + + The match must start at a name boundary: a plain substring test counted `NativeNote.X` as a + reference to module `Note` (field-service template run 2026-10-12, increment 3: "Note x54", 0 real). In BSON + the byte before a string's first character is the high byte of its int32 length (0x00), and + inside an XPath or expression it is `[`, `/`, `(`, a space or a quote — never [A-Za-z0-9_].""" + refs = {} + others = sorted((m for m in all_modules if m and m != module), key=len, reverse=True) + for u in subtree: + c = by_id[u][2] + for m in others: + if re.search(rb"(? widgets/.mpk, read from each widget package's own XML.""" + idx, wdir = {}, os.path.join(app_dir, "widgets") + if not os.path.isdir(wdir): + return idx + for f in sorted(os.listdir(wdir)): + if not f.endswith(".mpk"): + continue + try: + with zipfile.ZipFile(os.path.join(wdir, f)) as z: + for n in z.namelist(): + if n.endswith(".xml") and n != "package.xml": + try: + root = ET.fromstring(z.read(n)) + except ET.ParseError: + continue + if root.tag.endswith("widget") and root.get("id"): + idx.setdefault(root.get("id"), f"widgets/{f}") + except zipfile.BadZipFile: + continue + return idx + + +def package_xml(module, files): + lines = ['', + '', + ' ', + f' ', + ' ', + ' '] + lines += [f' ' for f in files] + lines += [' ', ' ', ''] + return "\n".join(lines) + "\n" + + +# --- main --------------------------------------------------------------------------------- +def export(src, module, out, with_widgets=True, scrub_constant_defaults=False): + # refuse before anything is removed or written: `out` == `src` would delete the source app + if os.path.exists(out) and os.path.realpath(out) == os.path.realpath(src): + raise SamePath(f"output {out} is the source .mpr itself — refusing before any write") + tmp_mpr = out + ".project.mpr.tmp" + if os.path.realpath(tmp_mpr) == os.path.realpath(src): + raise SamePath(f"temporary file {tmp_mpr} would be the source .mpr — refusing") + try: + return _export(src, module, out, with_widgets, tmp_mpr, scrub_constant_defaults) + finally: + if os.path.exists(tmp_mpr): + os.remove(tmp_mpr) + + +def _export(src, module, out, with_widgets, tmp_mpr, scrub_constant_defaults=False): + db, tables = open_source(src) + shell, subtree, mod_top, all_modules, by_id = collect(db, module) + app_dir = os.path.dirname(os.path.abspath(src)) + if os.path.exists(tmp_mpr): + os.remove(tmp_mpr) + dst = sqlite3.connect(tmp_mpr) + for name in ("_MetaData", "Unit"): + dst.execute(tables[name]) + meta_cols = [r[1] for r in db.execute("pragma table_info('_MetaData')")] + meta = db.execute(f"select {', '.join(meta_cols)} from _MetaData").fetchone() + dst.execute(f"insert into _MetaData ({', '.join(meta_cols)}) values ({', '.join('?' * len(meta_cols))})", meta) + keep = shell + subtree + q = f"select {', '.join(REF_UNIT_COLS)} from Unit where UnitID = ?" + scrubbed, constants_emptied = [], [] + for uid in keep: + row, changed = scrubbed_row(db.execute(q, (uid,)).fetchone(), scrub_constant_defaults) + if changed: + top = _bson_top(row[6]) + if top.get("$Type") == "Constants$Constant": + constants_emptied.append(f"{module}.{top.get('Name', '?')}") + else: + scrubbed.append(top.get("$Type")) + dst.execute(f"insert into Unit values ({', '.join('?' * 7)})", row) + dst.commit(); dst.close() + + files = module_files(app_dir, module) + widgets_used, widgets_shipped, widgets_missing = set(), [], [] + for u in subtree: + widgets_used |= _bson_strings(by_id[u][2], "WidgetId") + if with_widgets and widgets_used: + idx = widget_index(app_dir) + for w in sorted(widgets_used): + if w in idx: + if idx[w] not in widgets_shipped: + widgets_shipped.append(idx[w]) + else: + widgets_missing.append(w) + files += widgets_shipped + + if os.path.exists(out): + os.remove(out) + with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as z: + z.writestr("package.xml", package_xml(module, files)) + z.write(tmp_mpr, "project.mpr") + for f in files: + z.write(os.path.join(app_dir, f), f) + + types = {} + for u in subtree: + t = _bson_top(by_id[u][2]).get("$Type", "?") + types[t] = types.get(t, 0) + 1 + return { + "module": module, + "product_version": meta[0], + "units_module_subtree": len(subtree), + "units_shell": len(shell), + "project_docs_scrubbed": scrubbed, + "constant_defaults_emptied": constants_emptied, + "constant_defaults_scrubbed": scrub_constant_defaults, + "unit_types": dict(sorted(types.items())), + "from_app_store": mod_top.get("AppStoreVersion") or None, + "files": len(files), + "java_files": sum(1 for f in files if f.startswith("javasource/") and f.endswith(".java")), + "js_files": sum(1 for f in files if f.startswith("javascriptsource/")), + "widgets_referenced": sorted(widgets_used), + "widgets_shipped": widgets_shipped, + "widgets_unresolved": widgets_missing, + "crossing_refs_by_module": dict(sorted(crossing_refs(by_id, subtree, module, all_modules).items())), + "out": out, + } + + +KNOWN_FLAGS = {"--no-widgets", "--scrub-constant-defaults", "--json"} + + +def main(argv): + # an unknown flag is refused, never ignored: a typo such as `--scrub-constants` would otherwise + # export with every constant default in plain text while the caller believes it scrubbed them + unknown = [a for a in argv if a.startswith("--") and a not in KNOWN_FLAGS] + if unknown: + print(f"unknown option(s): {' '.join(unknown)} (known: {' '.join(sorted(KNOWN_FLAGS))})", + file=sys.stderr) + print(__doc__.split("\n\n")[0] + "\n" + __doc__.split("\n")[2], file=sys.stderr) + return 2 + args = [a for a in argv if not a.startswith("--")] + if len(args) != 3: + print(__doc__.split("\n\n")[0] + "\n" + __doc__.split("\n")[2], file=sys.stderr) + return 2 + try: + r = export(args[0], args[1], args[2], with_widgets="--no-widgets" not in argv, + scrub_constant_defaults="--scrub-constant-defaults" in argv) + except SamePath as e: + print(f"REFUSED: {e}", file=sys.stderr) + return 2 + except Refuse as e: + print(f"REFUSED: {e}", file=sys.stderr) + return 3 + if "--json" in argv: + print(json.dumps(r, indent=2)) + print("constant defaults " + (f"emptied: {len(r['constant_defaults_emptied'])} (--scrub-constant-defaults)" + if r["constant_defaults_scrubbed"] else "copied unchanged"), file=sys.stderr) + else: + print(f"wrote {r['out']}: module {r['module']} ({r['product_version']}), " + f"{r['units_module_subtree']} units + {r['units_shell']} shell, {r['files']} files " + f"({r['java_files']} .java, {r['js_files']} js), widgets shipped {len(r['widgets_shipped'])}" + f"/{len(r['widgets_referenced'])} referenced") + if r["constant_defaults_scrubbed"]: + print(f" constant defaults emptied (--scrub-constant-defaults): {len(r['constant_defaults_emptied'])}" + + (" (" + ", ".join(r["constant_defaults_emptied"]) + ")" if r["constant_defaults_emptied"] else "")) + else: + print(" constant defaults copied unchanged (module constants travel in the package; " + "--scrub-constant-defaults empties them)") + if r["widgets_unresolved"]: + print(" widget ids with no .mpk in widgets/ (core or platform widgets): " + + ", ".join(r["widgets_unresolved"])) + if r["crossing_refs_by_module"]: + print(" units naming other modules (crossing edges the target must satisfy): " + + ", ".join(f"{m}×{n}" for m, n in r["crossing_refs_by_module"].items())) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/pipelines/mendix-mpr/lib/compare-mdl.py b/pipelines/mendix-mpr/lib/compare-mdl.py new file mode 100755 index 00000000..c9ba539f --- /dev/null +++ b/pipelines/mendix-mpr/lib/compare-mdl.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Per-document comparison of two `describe module … with all` scripts (source, target). +Statements are grouped per document (a create plus the grant/index statements that +name the same document), keyed by (type, qualified name); order is ignored. +Prints TSV rows: module, document, type, verdict, evidence.""" +import os, re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements +def units(path): + stmts, _ = split_statements(open(path, encoding='utf-8', newline='').read()) + # any document type (`java action`, `json structure`, `task queue`, …); names may carry digits (AmazonS3Connector.Step2) + head = re.compile(r'^(?:@[^\n]*\n)*create or modify (?:([a-z][a-z -]*?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)|(module) ([A-Za-z0-9_]+));?', re.M) + grant = re.compile(r'^grant .*? on (?:[a-z ]+?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M) + docs, last = {}, None + for s in stmts: + body = re.sub(r'/\*\*.*?\*/\s*', '', s, flags=re.S) # doc comment compared separately + doc = re.search(r'/\*\*(.*?)\*/', s, re.S) + m = head.search(body) + if m: + t, name = (m.group(1), m.group(2)) if m.group(1) else (m.group(3), m.group(4)) + t = t.replace('non-persistent entity','entity').replace('persistent entity','entity').replace('view entity','entity') + last = (t, name); docs[last] = {'body': body.strip(), 'doc': (doc.group(1).strip() if doc else ''), 'grants': []} + continue + g = grant.search(body) + if g and last and g.group(1) == last[1]: docs[last]['grants'].append(body.strip()); continue + if g: + key = next((k for k in docs if k[1] == g.group(1)), None) + if key: docs[key]['grants'].append(body.strip()); continue + sys.stderr.write('unattached statement in %s: %s\n' % (path, body.strip()[:80])) + return docs +src, tgt = units(sys.argv[1]), units(sys.argv[2]) +module = sys.argv[3] +def firstdiff(a, b): + al, bl = a.split('\n'), b.split('\n') + for i, (x, y) in enumerate(zip(al, bl)): + if x != y: return 'line %d: %r -> %r' % (i+1, x.strip()[:70], y.strip()[:70]) + return 'length %d -> %d lines' % (len(al), len(bl)) +for k in sorted(src, key=lambda k: (k[0], k[1])): + t, name = k + if k not in tgt: + print('\t'.join([module, name, t, 'failed-exec', 'absent from target describe'])); continue + s, g = src[k], tgt[k] + ev = [] + if s['body'] != g['body']: ev.append('body: ' + firstdiff(s['body'], g['body'])) + if sorted(s['grants']) != sorted(g['grants']): ev.append('grants: %d -> %d' % (len(s['grants']), len(g['grants']))) + if s['doc'] != g['doc']: ev.append('documentation: ' + firstdiff(s['doc'], g['doc'])) + # describe marks a widget it cannot author with `-- … NOT re-executable`; the document lands without it + nre = re.findall(r'-- (\S+ \(\S+\)) -- NOT re-executable', s['body']) + if nre: print('\t'.join([module, name, t, 'not-expressible', 'describe marks %d widget(s) NOT re-executable: %s%s' % (len(nre), ', '.join(nre[:3]), '; ' + '; '.join(ev) if ev else '')])); continue + # a flow whose branches cross: describe says the text is NOT equivalent and must not be re-executed (mxcli #923) + neq = re.search(r'-- WARNING: (the decision at \([^)]*\)[^\n]*?)\. This description is NOT equivalent', s['body']) + if neq: print('\t'.join([module, name, t, 'not-expressible', 'describe: %s — NOT equivalent, must not be re-executed (mxcli #923)%s' % (neq.group(1), '; ' + '; '.join(ev) if ev else '')])); continue + print('\t'.join([module, name, t, 'emitted-drift' if ev else 'emitted-identical', '; '.join(ev) if ev else 'describe on target identical (body, grants, documentation)'])) +for k in sorted(set(tgt) - set(src)): + print('\t'.join([module, k[1], k[0], 'target-extra', 'present in target only (not in source describe)'])) diff --git a/pipelines/mendix-mpr/lib/exec-each.py b/pipelines/mendix-mpr/lib/exec-each.py new file mode 100755 index 00000000..7f708911 --- /dev/null +++ b/pipelines/mendix-mpr/lib/exec-each.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +"""exec-each.py + +Last resort of rehost-extract.sh's check_exec, for a script that `mxcli check` rejects in a way +lib/quarantine-mdl.py cannot map onto a statement, or whose exec error lib/exec-resume.py cannot place. +Executes the script ONE STATEMENT AT A TIME (each through `mxcli exec`, which runs check first), in +order: a statement that lands is in the target; one that fails is appended to with +its verbatim error and retried by the caller's deferred passes. is emptied (to +`mdl 1;`) — every statement has either landed or reached the deferred file, so nothing is dropped. + +GenAI showcase app 2026-10-12 (field run defect E2): when quarantine mapped nothing, check_exec returned 1 and +left the remainder in .ordered.mdl, never retried — 7 modules, 1040 statements, 1520 ledger rows +silently lost. Slow by design (one mxcli process per statement, ~1-2 s each); it runs only when the +mapping fails. Prints " ". +""" +import os, re, subprocess, sys, tempfile +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements, strip_deferred + +mxcli, tgt, src, deferred, cap = sys.argv[1:6] +stmts, tail = split_statements(open(src, encoding='utf-8', newline='').read()) +if tail.strip(): stmts.append(tail) +body = [strip_deferred(s) for s in stmts if s.strip() != 'mdl 1;'] +body = [s for s in body if re.search(r'^(create|grant|revoke|alter|drop|delete)\b', s, re.M)] +landed = failed = 0 +fd, one = tempfile.mkstemp(suffix='.mdl'); os.close(fd) +with open(cap, 'w', encoding='utf-8') as log, open(deferred, 'a', encoding='utf-8', newline='') as d: + if os.path.getsize(deferred) == 0: d.write('mdl 1;\n') + for s in body: + open(one, 'w', encoding='utf-8', newline='').write('mdl 1;\n' + s + '\n') + r = subprocess.run([mxcli, 'exec', '-p', tgt, one], capture_output=True, text=True, errors='replace') + out = '\n'.join(l for l in (r.stdout + r.stderr).split('\n') if 'vibe-coded PoC' not in l) + head = s.strip().split('\n')[0][:120] + if r.returncode == 0: + landed += 1; log.write('OK %s\n' % head); continue + failed += 1 + plain = re.sub(r'\x1b\[[0-9;]*m', '', out) + err = [l.strip() for l in plain.split('\n') if re.match(r'\s*(Error:|statement \d+:|- |✗|.*error)', l)] + reason = ' '.join(err)[:400] or plain.strip().split('\n')[-1][:400] + log.write('FAIL %s\n%s\n' % (head, plain)) + d.write('-- deferred: exec (one statement at a time) refused this statement: %s\n%s\n' % (reason, s)) +os.unlink(one) +open(src, 'w', encoding='utf-8', newline='').write('mdl 1;\n') +print(landed, failed) diff --git a/pipelines/mendix-mpr/lib/exec-resume.py b/pipelines/mendix-mpr/lib/exec-resume.py new file mode 100644 index 00000000..e8c9a65a --- /dev/null +++ b/pipelines/mendix-mpr/lib/exec-resume.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""exec-resume.py + +`mxcli exec` stops at the first statement it cannot write; everything before it is in the target, +everything after it is not. Find the statement the `Error:` line names (its qualified name, or the +bare document name mxcli prints as `/.ObjectCollection` for a flow), move it to +with the verbatim error, and rewrite to the statements AFTER it, so the caller can carry +on instead of losing the rest of the module. Prints 1 when it could do that, 0 when the error names +no statement it can find (the caller then reports the module as failed). + +GenAI showcase app 2026-10-12: `Error: export mapping GenAICommons.EM_Reference: message definition +"GenAICommons.MD_Reference.Reference" not found — this project has no message definitions` (0.26.0 has +no message-definition document at all) would otherwise have cost GenAICommons its 170 remaining statements. +""" +import os, re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements, strip_deferred + +src, out, deferred = sys.argv[1], sys.argv[2], sys.argv[3] +text = open(src, encoding='utf-8', newline='').read() +stmts, tail = split_statements(text) +if tail.strip(): stmts.append(tail) +report = open(out, encoding='utf-8', errors='replace').read() +err = next((l for l in report.split('\n') if l.startswith('Error:')), '') +if not err: print(0); sys.exit(0) +head_re = re.compile(r'^(?:@[^\n]*\n|/\*\*.*?\*/\s*)*create or modify ([a-z][a-z -]*?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M | re.S) +grant_re = re.compile(r'^grant .*? on (?:[a-z ]+?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M) +qns = set(re.findall(r'\b([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', err)) +bare = set(re.findall(r'/([A-Za-z0-9_]+)\.ObjectCollection', err)) +hit = None +for i, s in enumerate(stmts): + m = head_re.search(s); g = grant_re.search(s.strip()) + name = m.group(2) if m and m.group(1) != 'module' else (g.group(1) if g else None) + if not name: continue + if name in qns or name.split('.')[1] in bare: + if g and not m and not err.startswith('Error: failed to grant') and 'grant' not in err: continue + hit = i; break +if hit is None: print(0); sys.exit(0) +# The statement that NAMES the document is not always the one that failed. `Error: microflow not found: +# X` + `hint: X is defined later in this script` comes from an EARLIER statement that references X +# (GenAI 2026-10-13: entity ExampleMicroflows.Ticket with `on before commit call …BCo_Ticket`); deferring +# X's definition and resuming after it silently dropped every statement between the two (E5: 5 +# documents). So: on a forward-reference error, defer the first statement that mentions X without +# defining it; and resume from the earliest statement that mentions any name in the error, not after +# the deferred one — `create or modify` re-runs of statements that already landed are idempotent. +fwd = re.search(r'\b([A-Za-z0-9_]+\.[A-Za-z0-9_]+) is defined later in this script|references [a-z ]+? ([A-Za-z0-9_]+\.[A-Za-z0-9_]+) before it is created', report) +names = qns | {q for q in re.findall(r'\b([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', report) if fwd and q == (fwd.group(1) or fwd.group(2))} +def mentions(s, q): return re.search(r'\b' + re.escape(q) + r'\b', s) +lo = min([i for i, s in enumerate(stmts) if any(mentions(s, q) for q in names) or + any(re.search(r'/' + re.escape(b) + r'\b', s) for b in bare)] + [hit]) +if fwd: + x = fwd.group(1) or fwd.group(2) + for i in range(lo, len(stmts)): + m = head_re.search(stmts[i]) + if mentions(stmts[i], x) and not (m and m.group(2) == x): hit = i; break +with open(deferred, 'a', encoding='utf-8', newline='') as f: + if os.path.getsize(deferred) == 0: f.write('mdl 1;\n') + f.write('-- deferred: exec refused this statement: %s\n%s\n' % (err[:400].replace('\n', ' '), strip_deferred(stmts[hit]))) +rest = [s for i, s in enumerate(stmts) if i >= lo and i != hit] +open(src, 'w', encoding='utf-8', newline='').write('mdl 1;\n' + '\n'.join(s for s in rest if s.strip() != 'mdl 1;') + '\n') +print(1) diff --git a/pipelines/mendix-mpr/lib/fix-describe-output.py b/pipelines/mendix-mpr/lib/fix-describe-output.py new file mode 100644 index 00000000..85f25451 --- /dev/null +++ b/pipelines/mendix-mpr/lib/fix-describe-output.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""fix-describe-output.py — repair describe output that mxcli's own parser rejects. + +Only shapes that are unambiguously an emitter defect (mxcli 0.26.0, GenAI showcase app 2026-10-12): + 1. java action / javascript action parameter lines put the separating `,` AFTER the `-- comment`, + so the comment swallows it: ` Name: String not null -- The name.,` → `mismatched input + 'Name' expecting ')'`. Move the comma in front of the comment. 10 of 36 modules failed check on + this alone, and 25 more failed as a cascade. +Everything else stays byte-identical (CRLF inside literals is load-bearing). +""" +import re, sys +t = open(sys.argv[1], encoding='utf-8', newline='').read() +n = [0] +def comma(m): + n[0] += 1 + return m.group(1) + ',' + m.group(2).rstrip() + m.group(3) +t = re.sub(r'^( [A-Za-z0-9_"]+: [^\n]*?)( -- [^\n]*?),[ \t]*(\r?\n)', comma, t, flags=re.M) +open(sys.argv[2], 'w', encoding='utf-8', newline='').write(t) +sys.stderr.write('fix-describe-output: %d parameter commas moved out of comments\n' % n[0]) diff --git a/pipelines/mendix-mpr/lib/mdlsplit.py b/pipelines/mendix-mpr/lib/mdlsplit.py new file mode 100644 index 00000000..b9c68c85 --- /dev/null +++ b/pipelines/mendix-mpr/lib/mdlsplit.py @@ -0,0 +1,56 @@ +"""Split a describe script into top-level statements, shared by topo-order-mdl.py and +compare-mdl.py. A statement ends at a column-0 line that ends in ';' — but only when that line +starts and ends OUTSIDE a string literal ('…', doubled quote escapes), a $$…$$ block (Java/JS +bodies, JSON samples) or a comment (-- to end of line, /* … */). The GenAI showcase app (increment 2) +has microflows returning multi-line JSON string literals whose inner lines sit at column 0 and +end in ';', and the plain column-0 rule cut those flows into fragments that were then reordered. +Text is passed through byte-for-byte: no newline translation (CRLF inside literals is load-bearing).""" +import re +OPENER = re.compile(r'^(create|grant|revoke|alter|drop|delete|mdl|@|/\*|--)') +# after an INDENTED terminator only a keyword, an annotation or a doc comment at column 0 closes the +# statement (` );` then `@Position(380, 130)` + `create … entity`: ChatCompletions_FunctionCalling) +STRONG = re.compile(r'^(create |grant |revoke |alter |drop |delete |@[A-Za-z]|/\*\*)') +def split_statements(text): + stmts, cur, state, pending = [], [], 'code', None # state: code | str | dollar | block + for ln in text.split('\n'): + start = state + # a candidate boundary closes only when the next non-blank line opens a statement: Studio Pro + # emits multi-line expressions with continuation lines at column 0 ("else $X, Y = $Z);") + if pending and ln.strip(): + if start == 'code' and (OPENER if pending == 'col0' else STRONG).match(ln): + stmts.append('\n'.join(cur)); cur = [] + pending = None + i, n, comment = 0, len(ln), False + while i < n: + c = ln[i] + if state == 'code': + if c == "'": state = 'str' + elif ln.startswith('$$', i): state = 'dollar'; i += 1 + elif ln.startswith('--', i): comment = True; break # comment to end of line + elif ln.startswith('/*', i): state = 'block'; i += 1 + elif state == 'str': + if c == "'": state = 'code' # '' escape toggles twice, net unchanged + elif state == 'dollar': + if ln.startswith('$$', i): state = 'code'; i += 1 + elif state == 'block': + if ln.startswith('*/', i): state = 'code'; i += 1 + i += 1 + cur.append(ln) + # the line may START inside a $$ body: java/javascript actions end with a column-0 `$$;` + # (GenAI showcase app: 15 java actions of GenAICommons merged into one "statement", and check's + # statement numbers no longer matched ours). The opener lookahead above still guards the + # multi-line-literal case. + # a json structure with a CUSTOM NAME map ends on an indented ` );` (GenAI showcase app, 2026-10-13: + # 4 modules split 1-9 statements short of mxcli's count, so check's `statement N:` mapped onto + # the wrong statement or past the end — field run defect E1b). An indented terminator closes only when + # the next non-blank line opens with a column-0 keyword. + if state == 'code' and not comment and ln.rstrip().endswith(';'): + pending = 'col0' if not ln[0].isspace() else 'indent' + if pending and cur: stmts.append('\n'.join(cur)); cur = [] + return stmts, '\n'.join(cur) + +def strip_deferred(s): + """Drop the leading `-- deferred: …` reason lines a statement carries from an earlier pass.""" + ls = s.split('\n') + while ls and (ls[0].startswith('-- deferred:') or not ls[0].strip()): ls.pop(0) + return '\n'.join(ls) diff --git a/pipelines/mendix-mpr/lib/own-modules.awk b/pipelines/mendix-mpr/lib/own-modules.awk new file mode 100644 index 00000000..986a0896 --- /dev/null +++ b/pipelines/mendix-mpr/lib/own-modules.awk @@ -0,0 +1,12 @@ +# own-modules.awk — print the own modules of a `mxcli list modules` capture, one per line. +# awk -F'|' -f lib/own-modules.awk list-modules.txt +# Own module = empty Source column (Marketplace modules carry "Marketplace vX"), System excluded. +# Rows are the table lines after the `|---` separator, so a leading WARNING / "Connected to:" line +# and the trailing "(N modules)" line are skipped whether or not they are present. +# Golden input: tests/wave2/fixtures/rehost-list-modules/ (verbatim mxcli v0.25.0 and v0.26.0). +/^\|-/ { body = 1; next } +body && NF > 3 { + name = $2; src = $3 + gsub(/ /, "", name); gsub(/ /, "", src) + if (src == "" && name != "System" && name != "") print name +} diff --git a/pipelines/mendix-mpr/lib/quarantine-mdl.py b/pipelines/mendix-mpr/lib/quarantine-mdl.py new file mode 100644 index 00000000..f519d101 --- /dev/null +++ b/pipelines/mendix-mpr/lib/quarantine-mdl.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python3 +"""quarantine-mdl.py + +Move the statements that `mxcli check` rejected out of (rewritten in place) and append +them to , so the rest of the module can be executed instead of nothing at all. check is +all-or-nothing per file: one `-- line N:` syntax error or one `statement N:` reference error blocks +every other statement of the module (GenAI showcase app, 2026-10-12: 1 of 36 modules executed; 25 of +the 35 failures were cascades of 10 modules whose java actions mxcli emits with a trailing `,` inside +a `--` comment). Deferred statements are retried by the caller once the modules they reference exist; +what still fails is a ledger row with the verbatim check error, never a silent drop. + +Mapping (GenAI showcase app 2026-10-13, E1): `statement N:` reference errors map by number, verified against +the qualified name on the line and re-mapped by name when the numbered statement does not mention it; +`✗ … [CODE]` lint errors map by the document on the following `at Q.N` line, or by the `Q.N` in the +message when that line is empty. check counts statements without the leading `mdl 1;` (verified +against ChatCompletions_ComplexityReduction "statement 6" and GenAICommons "statement 233"). Each +deferred statement carries one `-- deferred:` line with the verbatim error. Prints the number of +statements moved, 0 when the check output names none it can map (the caller then falls back to +lib/exec-each.py, never drops the module). +""" +import os, re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements, strip_deferred + +src, chk, deferred = sys.argv[1], sys.argv[2], sys.argv[3] +text = open(src, encoding='utf-8', newline='').read() +stmts, tail = split_statements(text) +if tail.strip(): stmts.append(tail) +report = re.sub(r'\x1b\[[0-9;]*m', '', open(chk, encoding='utf-8', errors='replace').read()) # check colours its output +lines = report.split('\n') + +head_re = re.compile(r'^(?:@[^\n]*\n|/\*\*.*?\*/\s*)*create or modify ([a-z][a-z -]*?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M | re.S) +grant_re = re.compile(r'^(?:grant|revoke) .*? on (?:[a-z ]+?) ([A-Za-z0-9_]+\.[A-Za-z0-9_]+)\b', re.M) +qn_re = re.compile(r'\b([A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*)\b') +defined, granted = {}, {} +for i, s in enumerate(stmts): + m = head_re.search(s) + if m and m.group(1) != 'module': defined.setdefault(m.group(2), []).append(i) + g = grant_re.search(s) + if g and not m: granted.setdefault(g.group(1), []).append(i) + +# statement index (0-based into stmts) of each 1-based check statement number: check does not count +# the leading `mdl 1;` (GenAICommons: "statement 233" = the 234th split statement, `mdl 1;` first), +# nor a comment-only tail. +numbered = [i for i, s in enumerate(stmts) if s.strip() != 'mdl 1;' and not all( + l.strip().startswith('--') or not l.strip() for l in s.split('\n'))] +bad, why = set(), {} +def mark(ks, reason): + for k in ks: + bad.add(k); why.setdefault(k, reason) +def detail(at): # the error line plus its indented `- ...` continuation lines + out = [lines[at].strip()] + for l in lines[at + 1:at + 8]: + if l.strip().startswith('- '): out.append(l.strip()) + else: break + return ' '.join(out) + +for at, l in enumerate(lines): + # (1) reference errors: `statement N: 'Q.N' …`. Map by NUMBER, verified against the first + # qualified name on the line; when the numbered statement does not mention it, map by NAME (the + # create, or the grant for a grant line) — a wrong number must never quarantine the wrong statement. + m = re.match(r'^\s*statement (\d+):(.*)$', l) + if m: + n, rest = int(m.group(1)) - 1, m.group(2) + idx = numbered[n] if 0 <= n < len(numbered) else None + q = next((x for x in qn_re.findall(rest) if x in defined or x in granted), None) + if idx is not None and (q is None or re.search(r'\b' + re.escape(q) + r'\b', stmts[idx])): + mark([idx], detail(at)) + elif q: + is_grant = re.match(r'\s*(grant|revoke)\b', rest) + mark((granted.get(q) if is_grant else defined.get(q)) or defined.get(q) or granted.get(q, []), detail(at)) + continue + # (2) lint errors: `✗ [CODE]`, the document on the NEXT line (`at Q.N`) — or, when that + # line is empty (`at ` with nothing after it), named in the message itself (`✗ snippet Q.N: …`, + # `✗ layout 'Q.N' …`). GenAI 2026-10-12: 7 modules dropped because neither form mapped (E1a). + if l.lstrip().startswith('\u2717') and not re.search(r'\d+ (reference |syntax )?error', l): + nxt = lines[at + 1].strip() if at + 1 < len(lines) else '' + cands = qn_re.findall(nxt[2:]) if nxt.startswith('at') else [] + q = next((x for x in cands + qn_re.findall(l) if x in defined), None) + if q: mark(defined[q], l.strip() + ('' if q in l else ' at ' + q)) + +# syntax errors come as `line N:M` — map the line onto the statement that holds it +if re.search(r'^Syntax errors found', report, re.M): + starts, line = [], 1 + for s in stmts: + starts.append(line); line += s.count('\n') + 1 + for at, l in enumerate(lines): + m = re.match(r'^\s*- line (\d+)', l) + if not m: continue + n = int(m.group(1)) + for i in range(len(stmts) - 1, -1, -1): + if starts[i] <= n: mark([i], 'syntax error: ' + l.strip()); break +# close over the module's own references: a statement that names a document defined by a deferred +# statement would fail the next round anyway (check skips references to objects created within the +# script, so deferring the definition exposes every caller) — MxGenAIConnector took 15 rounds of one +# layer each before this. The callers go to the deferred file too and come back in the same retry. +grown = True +while grown: + grown = False + names = {n for n, ks in defined.items() if any(k in bad for k in ks)} + for n in names: + pat = re.compile(r'\b' + re.escape(n) + r'\b') + for i, s in enumerate(stmts): + if i in bad or s.strip() == 'mdl 1;': continue + m = head_re.search(s); g = grant_re.search(s.strip()) + if (m and m.group(1) == 'module'): continue + if pat.search(s) and not (m and m.group(2) == n and m.group(1) in ('entity', 'non-persistent entity')): + bad.add(i); grown = True; why.setdefault(i, 'same-module closure: names %s, deferred with it' % n) +if not bad: + print(0); sys.exit(0) +keep = [s for i, s in enumerate(stmts) if i not in bad] +open(src, 'w', encoding='utf-8', newline='').write('\n'.join(keep) + '\n') +with open(deferred, 'a', encoding='utf-8', newline='') as f: + if os.path.getsize(deferred) == 0: f.write('mdl 1;\n') + for i in sorted(bad): + # one `-- deferred:` line per statement, never nested: a retried statement arrives with its + # previous reason attached, which used to be wrapped again on every pass (E3, 7x overcount) + s = strip_deferred(stmts[i]) + f.write('-- deferred: check rejected this statement: %s\n%s\n' % (why.get(i, '?')[:400].replace('\n', ' '), s)) +print(len(bad)) diff --git a/pipelines/mendix-mpr/lib/topo-order-mdl.py b/pipelines/mendix-mpr/lib/topo-order-mdl.py new file mode 100755 index 00000000..af46fdd4 --- /dev/null +++ b/pipelines/mendix-mpr/lib/topo-order-mdl.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Reorder the statements of a `describe module … with all` script so every document is +created before any statement that references it. A unit is one `create or modify ` +statement plus the `grant … on … ` statements that follow it; a unit depends on every +other unit whose qualified name its text mentions. Kahn's sort, stable on the original order; +a cycle (recursion between flows, an entity whose event handler takes the entity) is reported +and its members are left in original order. Usage: topo-order-mdl.py in.mdl out.mdl + +Increment 1 ordered microflows only; increment 2 (GenAI showcase app, 48 modules) added nanoflows, +snippets called by pages, JSON structures used by mappings, and names with digits +(`AmazonS3Connector.Step2`), which the first pattern `[A-Za-z_]+` silently skipped.""" +import os, re, sys +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mdlsplit import split_statements +# newline='': annotation text carries CRLF inside string literals; translating it changes the flow and exec refuses the splice +stmts, tail = split_statements(open(sys.argv[1], encoding='utf-8', newline='').read()) +QN = r'[A-Za-z0-9_]+\.[A-Za-z0-9_]+' +head_re = re.compile(r'^(?:@[^\n]*\n|/\*\*.*?\*/\s*)*create or modify ([a-z][a-z -]*?) (' + QN + r')\b', re.M | re.S) +grant_re = re.compile(r'^grant .*? on (?:[a-z ]+?) (' + QN + r')\b', re.M) +# a unit = a create statement; a `grant … on ` statement is a unit of its own that depends on +# that document AND on every association of the module its member list names (bare member names: +# `grant read (Name, Credentials_Configuration) on entity M.Credentials` — exec rejects the grant when +# the association does not exist yet: "entity M.Credentials has no member(s) Credentials_Configuration", +# GenAI showcase app EchoConnector 2026-10-12; attaching grants to the entity put them before the association). +units = [] +for st in stmts: + m = head_re.search(st) + if m and m.group(1) != 'module': units.append((st, m.group(2), None)) + else: + g = grant_re.search(st.strip()) + units.append((st, None, g.group(1) if g else None)) +stmts = [u[0] for u in units] +idx_of = {} # name -> [unit indexes] (a name may carry two documents) +for k, u in enumerate(units): + if u[1]: idx_of.setdefault(u[1], []).append(k) +# a name carrying two documents (GenAI showcase app: microflow GenAICommons.ConsumedKnowledgeBase_GetCollectionOptions +# wraps a java action of the same name) must create the java/javascript action first — exec resolves +# the call in statement order and a flow created first makes the action "not found", so the flow is refused +LATE = ('microflow', 'nanoflow', 'page', 'snippet', 'layout') +for n, ks in idx_of.items(): + if len(ks) > 1: ks.sort(key=lambda k: (head_re.search(units[k][0]).group(1) in LATE, k)) +grant_units = [k for k, u in enumerate(units) if u[2]] +for k in grant_units: idx_of['grant#%d' % k] = [k] +movable = sorted(k for ks in idx_of.values() for k in ks) +pat = {n: re.compile(r'\b' + re.escape(n) + r'\b') for n in idx_of if not n.startswith('grant#')} +deps = {} +for n, ks in idx_of.items(): + body = '\n'.join(stmts[k] for k in ks) + deps[n] = {o for o in pat if o != n and pat[o].search(body)} + if n.startswith('grant#'): + on = units[ks[0]][2]; mod = on.split('.')[0] + deps[n] |= {on} if on in idx_of else set() + deps[n] |= {mod + '.' + w for w in re.findall(r'[A-Za-z_][A-Za-z0-9_]*', body) if (mod + '.' + w) in idx_of and (mod + '.' + w) != on} +movable = sorted(k for ks in idx_of.values() for k in ks) +# Kahn, stable on original order. A cycle (mutual recursion; an entity whose access rule names an +# association that names the entity) is broken by forcing the earliest remaining unit in original +# order and carrying on — never by abandoning the order of everything after it (that regressed the +# increment-1 fixture, where one entity/association cycle swallowed all 83 microflows). +order, placed, forced = [], set(), [] +remaining = sorted(idx_of, key=lambda n: idx_of[n][0]) +while remaining: + ready = [n for n in remaining if deps[n] <= placed] + if not ready: + ready = [remaining[0]]; forced.append(remaining[0]) + for n in ready: + order.append(n); placed.add(n); remaining.remove(n) +if forced: sys.stderr.write('cycle: forced ahead of an unmet reference: %s\n' % ', '.join(forced)) +seq = [k for n in order for k in idx_of[n]] # unit indexes in the new order +moved = sum(1 for a, b in zip(seq, movable) if a != b) +out = list(stmts) +for slot, k in zip(movable, seq): + out[slot] = stmts[k] +open(sys.argv[2], 'w', encoding='utf-8', newline='').write('\n'.join(out) + ('\n' + tail if tail.strip() else '\n')) +sys.stderr.write('statements=%d units=%d repositioned=%d\n' % (len(stmts), len(movable), moved)) diff --git a/pipelines/mendix-mpr/rehost-extract.sh b/pipelines/mendix-mpr/rehost-extract.sh new file mode 100755 index 00000000..2584d258 --- /dev/null +++ b/pipelines/mendix-mpr/rehost-extract.sh @@ -0,0 +1,269 @@ +#!/usr/bin/env bash +# rehost-extract.sh — read an existing Mendix .mpr, emit each own module as MDL, replay it into a +# fresh target .mpr of the same Mendix version, and write the fidelity ledger. +# +# pipelines/mendix-mpr/rehost-extract.sh [Module ...] +# +# /captures/ verbatim mxcli output (golden fixtures: list modules, show , +# project-tree.json, check/exec logs, timings.tsv per module, phases.tsv per phase) +# /mdl/ .mdl (source describe), .ordered.mdl (what was executed), +# target-.mdl (describe of the target afterwards), _settings/_userroles +# /fidelity-ledger.tsv one row per source document (denominator in the header) +# +# Modules default to every module `list modules` shows with an empty Source column except System. +# The target must already exist (`mxcli new --version --theme none`); this script +# creates nothing but model content. It does NOT run the build gate — run project-bin/exec.sh's +# mxbuild gate or `~/.mxcli/mxbuild//modeler/mx check ` afterwards. +# +# Field run (2026-10-12, mxcli v0.26.0, Linux container): a public banking sample app +# (Mendix 11.13.0, one own module of 151 documents + MyFirstModule 4) -> `mxcli new --version 11.13.0`. +# 155/155 catalog documents emitted, 154 identical on target describe, 1 documentation drift, +# `mx check` and `mxbuild --target=deploy` clean. Everything in here was exercised on that run: +# - the whole-module emitter is the MDL statement `describe module with all` run through -c; +# the `mxcli describe module ` SUBCOMMAND emits only the module and its roles (0.26.0). +# - describe emits microflows in catalog order, so `check` reported 14 MDL-ORDER01 (callee +# created after caller) and exec refused the file; lib/topo-order-mdl.py reorders flow units +# (create + its grant execute statements) and the reordered file checked clean. +# - `describe module … with all` omits image collections (MyFirstModule.Images) — described +# separately below so the ledger has the row. +# - demo users describe with Password: '***' and exec refuses the placeholder; project security +# level describes as a report, not MDL. Both are ledger hand-off rows, never executed here. +# Not exercised: nanoflows, Java/JavaScript actions, snippets, workflows, REST/OData, marketplace +# install (the blank template already carried the source's seven modules at the same versions). +# Portability: bash 3.2 (stock macOS) — no mapfile, no associative arrays; timing goes through +# Python because `date +%s.%N` and `bc` are missing there. Only Linux runs are field-proven. +set -euo pipefail + +SRC="${1:?usage: rehost-extract.sh [Module ...]}" +TGT="${2:?target.mpr}" +OUT="${3:?out-dir}" +shift 3 +HERE="$(cd "$(dirname "$0")" && pwd)" +MXCLI="${MXCLI:-$(command -v mxcli || true)}" +[ -x "$MXCLI" ] || { echo "mxcli not found: set MXCLI=/path/to/mxcli" >&2; exit 2; } +PY="${PYTHON:-python3}" +# the source is read-only: a target that resolves to the same file would be overwritten by exec +_real() { "$PY" -c 'import os, sys; print(os.path.realpath(sys.argv[1]))' "$1"; } +[ "$(_real "$SRC")" != "$(_real "$TGT")" ] || { echo "source and target are the same file: $SRC — refusing" >&2; exit 2; } +mkdir -p "$OUT/captures" "$OUT/mdl" +mx() { "$MXCLI" "$@" 2>&1 | grep -v 'vibe-coded PoC'; } # one warning line per call, drop it +log() { printf '%s\n' "$*" >&2; } + +T_START=$("$PY" -c 'import time; print("%.3f" % time.time())') +phase() { "$PY" -c 'import sys, time; t = time.time(); print("%s\t%.1f" % (sys.argv[1], t - float(open(sys.argv[2]).read() or sys.argv[3])))' "$1" "$OUT/captures/.phase-mark" "$T_START" >> "$OUT/captures/phases.tsv"; "$PY" -c 'import time; print("%.3f" % time.time())' > "$OUT/captures/.phase-mark"; } +printf 'phase\tseconds\n' > "$OUT/captures/phases.tsv"; : > "$OUT/captures/.phase-mark" +# ---- 1. inventory (golden captures) --------------------------------------------------------- +mx -p "$SRC" list modules > "$OUT/captures/list-modules.txt" +"$MXCLI" -p "$SRC" project-tree --json 2>/dev/null > "$OUT/captures/project-tree.json" +if [ $# -eq 0 ]; then + # own module = empty Source column, not System + MODULES=(); while IFS= read -r _m; do MODULES+=("$_m"); done < <(awk -F'|' -f "$HERE/lib/own-modules.awk" "$OUT/captures/list-modules.txt") +else + MODULES=("$@") +fi +# bash 3.2 under set -u dies on "${MODULES[@]}" when the array is empty +[ ${#MODULES[@]} -gt 0 ] || { log "no own modules in captures/list-modules.txt — name them as arguments"; exit 2; } +log "own modules: ${MODULES[*]}" +for M in "${MODULES[@]}"; do + for t in entities associations enumerations microflows nanoflows pages snippets javaactions; do + mx -p "$SRC" show "$t" "$M" | grep -v '^Connected to' > "$OUT/captures/show-$t-$M.txt" + done +done + +phase inventory +# ---- 2. extract ------------------------------------------------------------------------------- +# describe_module : `describe module … with all` plus every describable +# document of the module that the whole-module emitter omits, described one by one from the +# project tree. On 0.26.0 `with all` omits image collections (banking sample run) and, on a public +# GenAI showcase app (48 modules), nanoflows, snippets, JavaScript actions, JSON structures, import/export +# mappings, task queues, scheduled events and some pages. Used for the source and, in step 4, for +# the target, so both sides carry the same document set. +describe_module() { + local mpr="$1" M="$2" out="$3" + # `|| true`: on the target side the module may not exist at all (its check failed) — the ledger says so + "$MXCLI" -p "$mpr" -c "describe module $M with all" 2>/dev/null | grep -v 'vibe-coded PoC' > "$out" || true + # 0.26.0 cuts `-c` output at 10 000 lines without saying so (two modules of the GenAI showcase app + # both ended mid-page at line 10 001). Keep only complete statements; the tree fill-in + # below then describes every document the cut swallowed, one by one. + "$PY" -I - "$out" "$HERE/lib" <<'PYEOF' >&2 +import sys; sys.path.insert(0, sys.argv[2]) +from mdlsplit import split_statements +text = open(sys.argv[1], encoding='utf-8', newline='').read() +stmts, tail = split_statements(text) +if tail.strip(): + open(sys.argv[1], 'w', encoding='utf-8', newline='').write('\n'.join(stmts) + '\n') + print('describe module truncated at %d lines (incomplete last statement dropped: %s)' % (text.count('\n'), tail.strip().split('\n')[0][:80])) +PYEOF + "$MXCLI" -p "$mpr" project-tree --json 2>/dev/null > "$out.tree.json" + "$PY" -I - "$M" "$out" "$out.tree.json" <<'PYEOF' | while read -r t q; do +import json,re,sys +M, out, tree = sys.argv[1], sys.argv[2], sys.argv[3] +have = set(re.findall(r'^create or modify [a-z -]+? (' + re.escape(M) + r'\.[A-Za-z0-9_]+)', open(out, encoding='utf-8', newline='').read(), re.M)) +def walk(n): + for x in n: + q = x.get('qualifiedName', '') + if x.get('describable') and q.startswith(M + '.') and q not in have and x.get('type') not in ('modulerole',): + print(x['type'], q) + walk(x.get('children', [])) +walk(json.load(open(tree))) +PYEOF + "$MXCLI" -p "$mpr" describe "$t" "$q" 2>/dev/null | grep -v 'vibe-coded PoC' | sed '1{/^mdl 1;$/d}' >> "$out" + done + rm -f "$out.tree.json" +} +# timing in Python: `date +%s.%N` and `bc` do not exist on stock macOS +now() { "$PY" -c 'import time; print("%.3f" % time.time())'; } +since() { "$PY" -c 'import sys, time; print("%.1f" % (time.time() - float(sys.argv[1])))' "$1"; } +diff_s() { "$PY" -c 'import sys; print("%.1f" % (float(sys.argv[2]) - float(sys.argv[1])))' "$1" "$2"; } +TIMES="$OUT/captures/timings.tsv" +printf 'module\tdescribe_s\ttopo_s\tcheck_s\texec_s\tdescribe_target_s\n' > "$TIMES" +# per-module timings live in files, not associative arrays (bash 3.2 has none) +TDIR="$OUT/captures/.t"; mkdir -p "$TDIR" +tset() { printf '%s' "$3" > "$TDIR/$1.$2"; } # tset +tget() { cat "$TDIR/$1.$2" 2>/dev/null || true; } # tget +# REHOST_REUSE_MDL=1 skips the source describes when /mdl/.mdl already exists (a rerun after +# fixing the check/exec side; the describes are the slow part — 36 modules took 30 min on the GenAI showcase app). +for M in "${MODULES[@]}"; do + t0=$(now) + if [ "${REHOST_REUSE_MDL:-0}" = 1 ] && [ -s "$OUT/mdl/$M.mdl" ]; then log "reusing mdl/$M.mdl"; else describe_module "$SRC" "$M" "$OUT/mdl/$M.mdl"; fi + t1=$(now) + "$PY" -I "$HERE/lib/topo-order-mdl.py" "$OUT/mdl/$M.mdl" "$OUT/mdl/$M.ordered.mdl" 2>>"$OUT/captures/describe-$M.stderr" + "$PY" -I "$HERE/lib/fix-describe-output.py" "$OUT/mdl/$M.ordered.mdl" "$OUT/mdl/$M.ordered.mdl" 2>>"$OUT/captures/describe-$M.stderr" + tset DESC "$M" "$(diff_s "$t0" "$t1")"; tset TOPO "$M" "$(since "$t1")" +done +# module order: replay the modules so that each is created after every listed module it references +# (qualified-name mention in its describe output); REHOST_KEEP_ORDER=1 keeps the order given. A cycle +# between modules is reported and left in the given order — exec then fails on the forward reference +# and the ledger says so. +if [ "${REHOST_KEEP_ORDER:-0}" = 0 ]; then + _order=$("$PY" -I - "$OUT/mdl" "${MODULES[@]}" <<'PYEOF' +import re,sys +d, mods = sys.argv[1], sys.argv[2:] +body = {m: open('%s/%s.mdl' % (d, m), encoding='utf-8', newline='').read() for m in mods} +deps = {m: {o for o in mods if o != m and re.search(r'\b' + re.escape(o) + r'\.[A-Za-z0-9_]', body[m])} for m in mods} +order, placed, rest = [], set(), list(mods) +while rest: + ready = [m for m in rest if deps[m] <= placed] + if not ready: + sys.stderr.write('module cycle among: %s (left in catalog order)\n' % ', '.join(rest)); order += rest; break + for m in ready: order.append(m); placed.add(m); rest.remove(m) +print('\n'.join(order)) +PYEOF + ) + MODULES=(); while IFS= read -r _m; do [ -n "$_m" ] && MODULES+=("$_m"); done </dev/null | grep -v 'vibe-coded PoC' > "$OUT/mdl/_settings.mdl" +mx -p "$SRC" -c "show user roles" > "$OUT/captures/show-userroles.txt" +: > "$OUT/mdl/_userroles.mdl" +awk -F'|' 'NR>3 && NF>3 { gsub(/ /,"",$2); if ($2!="") print $2 }' "$OUT/captures/show-userroles.txt" | while read -r r; do + "$MXCLI" -p "$SRC" describe userrole "$r" 2>/dev/null | grep -v 'vibe-coded PoC' >> "$OUT/mdl/_userroles.mdl" +done + +phase describe_topo_userroles +# ---- 3. check, then exec into the target (modules in replay order) --------------------------- +# check is all-or-nothing per file, so a statement it rejects is moved to mdl/.deferred.mdl +# (lib/quarantine-mdl.py) and the rest of the module is executed; the deferred statements are retried +# below until no retry makes progress, because most rejections are forward references into a module +# that is created later (cross-module cycles) or into a statement deferred for a syntax error. +status=0 +check_exec() { # check_exec