diff --git a/CHANGELOG.md b/CHANGELOG.md index 875fb69cf..9429b41d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,147 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ## [Unreleased] +### Added — `sources` is read by all four CLI surfaces, plus `meta init --config-only` + +`.metaobjects/config.json`'s `sources` key stops being a Node-only concern. Adopter +guide: [`docs/features/metadata-sources.md`](docs/features/metadata-sources.md). + +- **`sources` is read by all four CLI surfaces**, not just the Node `meta` CLI — + the C#, Python and Java/Kotlin CLIs (Kotlin has no CLI of its own; it runs + through the same Maven plugin as Java) now resolve metadata from the + port-neutral `.metaobjects/config.json`, so one declaration serves every port + (C#'s CLI loader accepts only a single directory `path` source — see the + adopter guide). Each reads a **neutral subset** (`schema_version` + `sources`) and ignores + unknown top-level keys, so the TypeScript-owned keys in that file (`migrate`, + `scope`, `extract`, and the rest) never become a four-port change. Precedence + is a ladder — explicit CLI argument, then the port's own native surface (a + pom's ``/``, Python's `metadata` key), then `sources`, + then the default `metaobjects/` directory — and a config that exists but is + malformed errors at its own rung rather than silently falling through. Gated + by the new + [`fixtures/source-resolution-conformance/`](fixtures/source-resolution-conformance/) + corpus, which every port runs. +- **`meta init --config-only`** writes `.metaobjects/config.json` and nothing + else, so a Maven- or pip-rooted project can declare its sources for the Node + CLI (which owns `migrate` and `verify --db`, ADR-0015) without acquiring a + TypeScript scaffold it will not use. +- **`scope` / `migrate.scope` stay Node-CLI-only.** Java's shipped `` + grammar uses `*` to cross the `::` separator and `@` to match one segment — + respectively `scope`'s `**` and `*`, inverted — plus `!`-prefix exclusion and + a `.[attr]` predicate `scope` cannot express at all + (`GeneratorUtil.createRegexFromGlob` carries a `TODO` conceding its own + separator handling is wrong). Both are output filters over the same resolved + file set, so reconciling them is a separate, adopter-affecting decision + rather than a mechanical port. No cross-port behavior depends on `scope`. +- **Resolved file order, and the malformed-config error code, are deliberately + NOT cross-port contracts.** The ports' directory walks already differ and + always have (Java sorts by basename, C# by full-path ordinal, Python by + basename, TypeScript walks depth-first); the corpus compares file **sets**. + A malformed config must raise rather than silently degrade to "no config", + but which error is each port's own — verified empirically: TypeScript raises + a raw `ZodError` with no code at all, Python raises + `ERR_COLLECTION_NOT_FOUND`, C# and Java both raise `ERR_BAD_ATTR_VALUE`. +- **Directory expansion follows symlinked directories in all four ports** — + including when a declared `sources` path is itself a symlink, or a symlink + sits partway through a walked tree. TypeScript and C# already did; Java and + Python now match (a symlinked `sources` path previously resolved to zero + files in Java, silently, exit 0). A symlink cycle is a loud error rather + than a hang. Gated by two new `symlinks`-bearing corpus cases. +- **Behavior change (Java/Maven only): a `` naming neither + `` nor ``, with no `.metaobjects/config.json` `sources` + and no default `metaobjects/` directory, now FAILS the build** + (`ERR_COLLECTION_NOT_FOUND`) instead of silently producing an empty model + and passing. This is the one behavior change here that can break an + existing `mvn metaobjects:generate`/`:verify` — most likely to bite a + multi-module reactor where a parent pom configures `` and one child + module never adds its own ``. To restore the old outcome, declare + ``/`` explicitly in that module's pom, or give it a real + metadata source (a `metaobjects/` directory or a `.metaobjects/config.json` + `sources` entry). + +### Changed — a committed migration chain must replay from empty, and `meta migrate` stops writing chains that cannot ([#313](https://github.com/metaobjectsdev/metaobjects/issues/313)) + +**`meta migrate --from-db` now REFUSES a drop for a table or view the committed schema +snapshot never contained**, exiting 2 and naming each object. This is the one change here +that can fail an existing project's `meta migrate`, so it leads. Pass +**`--allow drop-unmanaged`** when the drop is genuinely intended. + +The refusal exists because the drop it blocks produces a migration nobody can replay. The +live migrate path diffs metadata against introspection and never reads the snapshot, so a +table another tool owns reads as "in the database, not in the model" and is proposed for a +`DROP TABLE`. Every incremental migrate then keeps succeeding against the database that +already has that table — the chain only fails the day someone provisions a fresh one, which +for the reporter was **three months later**, by which point the only working database left +was a leftover CI container. `drift/classify.ts` has always said objects present in the DB +but not the snapshot "must never be treated as actionable drift or auto-dropped"; this is +the first place that doctrine is enforced where it mattered. + +It does not false-fire on brownfield projects, and the reason is structural rather than +special-cased: **both mechanisms ADD to the snapshot.** A `baseline --from-db` snapshot +contains the foreign table; a project declaring `migrate.scope` carries its out-of-scope +entries forward. The guard fires precisely when nothing ever claimed the object. It fails +OPEN with no snapshot on disk — refusing there would break the first `meta migrate` of every +greenfield project — and it lives on the live path only, because the offline path diffs +against the snapshot and so cannot propose a snapshot-absent drop at all. + +**Emitted forward drops now carry `IF EXISTS`** — `drop-table`, `drop-view` (plain and +CASCADE), `drop-index` (both the plain form and #285's constraint-backed +`ALTER TABLE … DROP CONSTRAINT`), `drop-fk` and `drop-check` — in both dialects, so an +already-absent object cannot break a replay. **Down statements stay bare, deliberately:** +`rollbackTo` runs `down.sql` and the ledger delete in ONE transaction, so a guarded down +would no-op and still record the rollback as done. Rollback is the one place a loud failure +is load-bearing. Also left bare on purpose: the sqlite recreate-and-copy rebuild's +`DROP TABLE` and d1-cascade's, each of which drops a table the same recipe just +`INSERT…SELECT`ed from, where `IF EXISTS` converts a caught corruption into a silent one. +`drop-column` is excluded as the one genuine dialect limit — sqlite has no +`DROP COLUMN IF EXISTS` — and the new refusal covers it instead. D1 inherits the sqlite +change, since `emit/d1.ts` renders through `renderSqlite`. + +**A chain creating a table or view in a non-default schema now emits +`CREATE SCHEMA IF NOT EXISTS`** ahead of it. `CREATE SCHEMA` was emitted nowhere in either +emitter — only by the ledger's own setup — so an `@schema` project's chain could never apply +to a virgin database. Views count, not only tables: a first migration creating just a view in +a non-default schema failed identically. The down does not drop the schema; it may hold +objects this tool does not own and cannot restore. + +### Added — `meta verify --replay` and `--replay-snapshot` + +Two new verify subverbs that answer the question the toolchain was already promising an +answer to. `docs/features/migrations-and-drift.md` and `meta migrate --help` both said +`apply-pending` "is the way to provision a fresh or CI database"; that is true only of a +chain that builds the schema, and nothing checked. + +- **`--replay`** replays the committed chain into an empty throwaway database and asserts it + **applies**. This is the #313 gate. +- **`--replay-snapshot`** additionally asserts the replayed schema **equals the committed + snapshot**, finally wiring `verifyReplay` — built, exported, and without a CLI caller since + the 2026-05-31 design retained it as "the optional `verify --replay` integrity aid". It + catches a different defect: hand-edited structural DDL that still applies but no longer + builds the recorded schema. + +They are two tiers rather than one gate because the populations differ. A project adopted via +`migrate baseline --from-db` passes the first trivially and **cannot** pass the second by +construction — its snapshot is the whole introspected database against an empty chain. The +reporter's failure was an *apply* error, so the weaker assertion is the one that answers the +bug and is immune to that class. The limitation is documented rather than auto-detected: the +only candidate signal has no production caller and would live in the *target* database's +ledger, while the gate runs against a fresh engine with no ledger at all. + +Neither needs a `--db`. The engine is local and disposable — real Postgres in-process via +**PGlite**, a throwaway temp file for sqlite — so there is nothing to provision, no +credentials, and no scratch database to collide with or drop by mistake. **`@electric-sql/pglite` +is a new OPTIONAL peer dependency of `@metaobjectsdev/migrate-ts`** (~22 MB of WASM, so it is +not forced on every adopter): install it to replay a postgres chain. With no URL to infer from, +the dialect precedence is `--dialect` > `migrate.dialect` > refuse naming `--dialect`. +`--migration-format flyway` and `--dialect d1` are refused, mirroring `apply-pending`. An empty +chain and a missing snapshot both pass and **say which**, because a gate that is silent when it +checked nothing cannot be told apart from one that passed. + +`verifyReplay` also gains an optional `governed` so a project declaring `migrate.scope` can use +the second tier at all: such a project carries the other owner's tables into its snapshot on +purpose and its chain never creates them, so without this they were reported as missing on +every replay. + ### Added — pre-release publishing to a private registry (no more real releases just to test a change) Trying an unreleased change against a downstream project required cutting a real release on diff --git a/bun.lock b/bun.lock index d241e0129..d8e0d292e 100644 --- a/bun.lock +++ b/bun.lock @@ -330,6 +330,7 @@ "@metaobjectsdev/metadata": "workspace:*", }, "devDependencies": { + "@electric-sql/pglite": "^0.5.0", "@libsql/kysely-libsql": "^0.4.1", "@types/pg": "^8.20.0", "bun-types": "latest", @@ -339,8 +340,14 @@ "typescript": "^5.6.0", }, "peerDependencies": { + "@electric-sql/pglite": ">=0.3.0 <0.6.0", + "@libsql/kysely-libsql": ">=0.4.0 <0.5.0", "kysely": ">=0.27.0 <0.30.0", }, + "optionalPeers": [ + "@electric-sql/pglite", + "@libsql/kysely-libsql", + ], }, "server/typescript/packages/render": { "name": "@metaobjectsdev/render", @@ -531,6 +538,8 @@ "@csstools/css-tokenizer": ["@csstools/css-tokenizer@4.0.0", "", {}, "sha512-QxULHAm7cNu72w97JUNCBFODFaXpbDg+dP8b/oWFAZ2MTRppA3U00Y2L1HqaS4J6yBqxwa/Y3nMBaxVKbB/NsA=="], + "@electric-sql/pglite": ["@electric-sql/pglite@0.5.5", "", {}, "sha512-QeZ+oB7QaU4EJj2awrB8hwFZ5TXxLRBQ9Gfq0dQauYDdWsRtuWRCERgtri35vRfL07KQHlVlc/4Qxvn7a5AXeA=="], + "@exodus/bytes": ["@exodus/bytes@1.15.1", "", { "peerDependencies": { "@noble/hashes": "^1.8.0 || ^2.0.0" }, "optionalPeers": ["@noble/hashes"] }, "sha512-S6mL0yNB/Abt9Ei4tq8gDhcczc4S3+vQ4ra7vxnAf+YHC02srtqxKKZghx2Dq6p0e66THKwR6r8N6P95wEty7Q=="], "@fastify/ajv-compiler": ["@fastify/ajv-compiler@3.6.0", "", { "dependencies": { "ajv": "^8.11.0", "ajv-formats": "^2.1.1", "fast-uri": "^2.0.0" } }, "sha512-LwdXQJjmMD+GwLOkP7TVC68qa+pSSogeWWmznRJ/coyTcfe9qA05AHFSe1eZFwK6q+xVRpChnvFUkf1iYaSZsQ=="], diff --git a/docs/CONFORMANCE.md b/docs/CONFORMANCE.md index 941ece9df..d2dbfc967 100644 --- a/docs/CONFORMANCE.md +++ b/docs/CONFORMANCE.md @@ -1,6 +1,6 @@ # Conformance coverage -The MetaObjects standard ships **20 shared conformance corpora** under +The MetaObjects standard ships **21 shared conformance corpora** under [`fixtures/`](../fixtures/). Every port runs every corpus that is *applicable to it* and asserts the same expected behaviour against the same fixtures. **This page is the inverse index**: fixture → feature doc + per-port pass status, and it is the @@ -42,6 +42,7 @@ regenerate with `ls -d fixtures//*/ | wc -l`. | [`fixtures/template-output-render-conformance/`](../fixtures/template-output-render-conformance/) | 5 | ✓ | ✓ | ✓ | ✓ | ✓ | | [`fixtures/generator-registry-conformance/`](../fixtures/generator-registry-conformance/) | 1 canonical manifest | ✓ | ✓ | ✓ | ✓ | ✓ | | [`fixtures/provider-composition-conformance/`](../fixtures/provider-composition-conformance/) | 9 (5 error-shape + 4 compose-load) | ✓ | ✓ | — (JVM registry via Java) | ✓ | ✓ | +| [`fixtures/source-resolution-conformance/`](../fixtures/source-resolution-conformance/) | 25 cases | ✓ (reference implementation) | ✓ | inherits via Java | ✓ | ✓ | | [`fixtures/scope-conformance/`](../fixtures/scope-conformance/) | 10 cases | ✓ (reference implementation) | — | — | — | — | | [`fixtures/agent-context-conformance/`](../fixtures/agent-context-conformance/) | 4 | ✓ (the emitter is TS-owned) | — | — | — | — | | [`fixtures/metamodel-docs/`](../fixtures/metamodel-docs/) | 1 | ✓ (docs emit is TS-owned) | — | — | — | — | @@ -182,6 +183,31 @@ inheritance), `m2m/` (3), `jsonb/` (2, typed value-object columns) and Kotlin, C#, Python — run it in BOTH lanes: a hand-rolled reference server and the port's own GENERATED API artifact booted over HTTP. +### `fixtures/source-resolution-conformance/` (25 cases) + +All 25 cases → [features/metadata-sources.md](features/metadata-sources.md) (how a +declared `sources` set resolves to a file list). Companion to +`scope-conformance/` below — `sources` decides which files are read, `scope` +filters what is emitted from them. The corpus is file-shaped: one committed +`cases.json`, read directly by every port's runner, with no per-port fixture +and no ledger. + +It pins the resolved file **SET** for a declared `sources` list — the default +directory, replacement-not-merge, the relative-path base (the directory +HOLDING `.metaobjects/`, never the process cwd), recursive directory walking, +case-insensitive extension matching, union-with-de-duplication, and every +error condition (an unresolvable path, an unsupported `resource`/`package` +kind, and a malformed config — `"expectError": true` pins only that +resolution RAISES, since which error code it raises with is deliberately NOT +a cross-port contract; see the corpus README). **All four CLI surfaces run +it** — TypeScript (the reference implementation), C#, Python, and Java (Kotlin +inherits it, since Kotlin has no CLI entry point of its own and runs through +the same Maven plugin as Java): +`server/typescript/packages/sdk/test/source-resolution-conformance.test.ts`, +`server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs`, +`server/python/tests/conformance/test_source_resolution_conformance.py`, and +`server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java`. + ### `fixtures/scope-conformance/` (10 cases) All 10 cases → [features/metadata-sources.md](features/metadata-sources.md) (the @@ -207,9 +233,9 @@ grammar rather than four. ## Orphaned fixtures (tested but not yet documented) -The fixtures in the seven corpora mapped above (metamodel 255 + yaml 15 + verify 31 -+ render 15 + persistence 33 + api-contract 41 + scope 10) each map to a feature doc. None -are orphaned today. The remaining corpora in the totals table gate tooling +The fixtures in the eight corpora mapped above (metamodel 255 + yaml 15 + verify 31 ++ render 15 + persistence 33 + api-contract 41 + source-resolution 25 + scope 10) each +map to a feature doc. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider composition, agent context, docs emit) rather than user-facing metamodel behaviour, so they have no feature-doc row. diff --git a/docs/features/metadata-sources.md b/docs/features/metadata-sources.md index 930a63044..d10d1a027 100644 --- a/docs/features/metadata-sources.md +++ b/docs/features/metadata-sources.md @@ -34,12 +34,74 @@ exactly that set. A `path` is read **in place and never installed** or copied. `"sources": []`, so every existing project takes the default and resolves the same files it always did. -**Port support.** `sources` / `scope` / `migrate.scope` are read by the **Node -`meta` CLI** today. The Java, Kotlin, C# and Python CLIs still take their metadata -location their own way (a Maven `` element, a positional directory, a -`metadata` config key). The cross-port pattern grammar is already pinned by -[`fixtures/scope-conformance/`](../../fixtures/scope-conformance/); wiring the other -four CLIs to the same config file is future work, not yet planned or scheduled. +**Port support.** `sources` is read by **all four CLI surfaces** — the Node `meta` +CLI, `dotnet meta` (C#), `metaobjects` (Python) and `metaobjects:generate` (Java +and Kotlin, via Maven — Kotlin has no CLI entry point of its own; it runs through +the same Maven plugin as Java). Each resolves the same files from the same +declaration; that promise is gated by +[`fixtures/source-resolution-conformance/`](../../fixtures/source-resolution-conformance/). + +Each port reaches it its own way, and the ladder is the same everywhere — first +match wins: + +1. An explicit CLI argument (a positional metadata directory, `--config`, `--cwd`). +2. The port's own native surface, where it has one — Java and Kotlin's pom + ``/``, Python's `metadata` key in `metaobjects.config.yaml`. + If the pom names either element it owns the concern outright and the neutral + file is not consulted. +3. `sources` in `.metaobjects/config.json`. +4. The built-in default — a `metaobjects/` directory beside that config. + +A config file that exists but is malformed is an error at its own rung; it never +falls through to the next one. + +**`dotnet meta` (C#) accepts only a single DIRECTORY source.** Its loader +(`MetaDataLoader.FromDirectory`) takes one directory, so `sources` declaring more +than one entry, or a single entry that resolves to a FILE rather than a +directory, is refused with a clear diagnostic naming the limit — rather than +silently loading a subset of the declared sources, or (for the single-file case) +failing deep inside the loader with an opaque, uncoded error. Declare exactly one +directory `path` to use `dotnet meta gen`/`verify`/`docs`, or route around the +limit with an explicit `` CLI argument. +`MetaDataLoader.Load(IReadOnlyList)` (`MetaDataLoader.cs`) is the +documented follow-up that would lift both restrictions. The other three CLI +surfaces have no such limit. + +The non-TypeScript ports read a **neutral subset** of that file — +`schema_version` and `sources` — and ignore every other top-level key, so the +TypeScript-owned keys beside them (`migrate`, `extract`, and the rest) never +become a four-port concern. The Node CLI remains the file's only writer; +`meta init --config-only` writes it into a Maven- or pip-rooted project without +adding a TypeScript scaffold. + +**`scope` and `migrate.scope` remain Node-CLI-only.** Java's shipped `` +grammar uses `*` to cross the `::` package separator and `@` to match one +segment — respectively `scope`'s `**` and `*`, **inverted** — plus `!`-prefix +exclusion and a `.[attr]` predicate that `scope` cannot express at all +(`GeneratorUtil.createRegexFromGlob` even carries a `TODO` conceding its own +separator handling is wrong). Both are output filters over the same resolved +file set, so they compete rather than layer, and reconciling them changes +behavior for existing Java consumers — a separate, adopter-affecting decision +rather than a mechanical port. No cross-port behavior depends on `scope`. + +**Two things are deliberately not cross-port contracts**, because the corpus +that gates this feature says so explicitly: + +- **Resolved file order.** The ports' directory walks already differ and always + have — Java sorts by basename, C# by full-path ordinal, Python by basename, + TypeScript walks depth-first with files before subdirectories — and the + loader's overlay partition discards caller order regardless + (super-resolution is order-independent, #188). Every port resolves the same + file **set**; only the order within it is each port's own. +- **The error code for a malformed config.** A `.metaobjects/config.json` that + exists but is malformed (e.g. `sources` declared as an object instead of an + array) must raise rather than silently degrade to "no config" — but which + error is each port's own. Verified empirically: TypeScript raises a raw + `ZodError` carrying no MetaObjects error code at all, Python raises + `ERR_COLLECTION_NOT_FOUND`, and C# and Java both raise `ERR_BAD_ATTR_VALUE`. + The shared corpus expresses this with `"expectError": true` ("must raise, + code unpinned") alongside the existing string-code form for cases that DO + pin one. --- @@ -575,8 +637,11 @@ Phase 1 ships the spine. Explicitly **not** built yet, so you do not go looking: - **`url` sources** and **named `collection` references**. - **Per-generator declarative `scope`.** The TypeScript `filter` function is the escape hatch and is unchanged. -- **The other four ports' CLIs reading `.metaobjects/config.json`.** The pattern - grammar is corpus-gated so they cannot diverge when they land. +- **Cross-port `scope` and `migrate.scope`.** Java's own `` grammar + collides with `scope`'s pattern characters (see "Port support" near the top of + this document), so reconciling the two is a separate, adopter-affecting decision + rather than a mechanical port. They remain Node-CLI-only for now; `sources` + itself is no longer in this list — all four CLI surfaces read it today. - **Database and other runtime metadata sources.** Ruled a runtime-metadata concern rather than a build-time one. @@ -587,17 +652,33 @@ Design rationale and the full phase plan: ## Verified by +**Cross-port source resolution** + +- [`fixtures/source-resolution-conformance/`](../../fixtures/source-resolution-conformance/) — + pins the resolved file SET for a declared `sources` list (the default, the + relative-path base, recursion, extension matching, union/de-dup, and every error + condition, including the deliberately-unpinned malformed-config case — see the + corpus's own README). All four CLI surfaces run it: + `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts`, + `server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs`, + `server/python/tests/conformance/test_source_resolution_conformance.py`, and + `server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java` + (Kotlin runs through the Java/Maven loader, so it shares the Java runner). + **Cross-port pattern semantics** - [`fixtures/scope-conformance/`](../../fixtures/scope-conformance/) — 10 cases pinning `*` / `**`, include-union, exclude-after-include, literal metacharacters, and case sensitivity. TypeScript runs it today; the other four ports have no runner - yet. See [`CONFORMANCE.md`](../CONFORMANCE.md). + yet, because `scope` itself stays Node-CLI-only (see "Port support" above). See + [`CONFORMANCE.md`](../CONFORMANCE.md). **TypeScript gates** - `server/typescript/packages/sdk/test/scope.test.ts` — the pattern engine - `server/typescript/packages/sdk/test/scope-conformance.test.ts` — the corpus runner +- `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts` — the + cross-port sources corpus runner - `server/typescript/packages/sdk/test/sources.test.ts` — `path` resolution, `_pending` exclusion, unsupported kinds - `server/typescript/packages/sdk/test/discovery.test.ts` — nearest-ancestor walk and diff --git a/docs/features/migrations-and-drift.md b/docs/features/migrations-and-drift.md index 446de0380..3149d4901 100644 --- a/docs/features/migrations-and-drift.md +++ b/docs/features/migrations-and-drift.md @@ -57,12 +57,100 @@ configured in `metaobjects.config.ts` (typically `./migrations/__";` ahead of it. Without that, an `@schema` +project's chain could never apply to a virgin database, because nothing ever created +the schema. The down does **not** drop the schema: it may hold objects this tool does +not own and cannot restore. + #### D1: rebuilding a foreign-key-referenced table Cloudflare D1 applies each migration inside its own implicit transaction, and SQLite diff --git a/docs/superpowers/plans/2026-08-19-cross-port-metadata-sources.md b/docs/superpowers/plans/2026-08-19-cross-port-metadata-sources.md new file mode 100644 index 000000000..aaafb5c19 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-cross-port-metadata-sources.md @@ -0,0 +1,1881 @@ +# Cross-port metadata `sources` Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the Java/Kotlin, C# and Python CLIs read the `sources` key from the port-neutral `.metaobjects/config.json`, so all four CLI surfaces resolve the same metadata files the Node `meta` CLI already does. + +**Architecture:** Each port gains two small units — a **neutral-subset config reader** (parse `schema_version` + `sources`, ignore unknown top-level keys) and a **source resolver** (spec set → metadata file set, relative to the declaring config's directory). Both are then wired behind the port's existing metadata-location argument as a *fallback*, never an override. A new filesystem-based conformance corpus gates the resolved file **set** across all four ports plus TypeScript. + +**Tech Stack:** Java 17 + Maven (`metaobjects-maven-plugin`, JUnit), C# (.NET, xUnit), Python 3 (`uv` + pytest), TypeScript (Bun test). No new third-party dependencies in any port. + +**Spec:** [`docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md`](../specs/2026-08-19-cross-port-metadata-sources-design.md) + +## Global Constraints + +- **`scope` is OUT of scope.** Only `sources` ships. No port may read, parse, or act on the `scope` or `migrate.scope` keys. Reason: Java's shipped `` grammar collides on `*` and `@` (spec §2). +- **The neutral subset is exactly `schema_version` and `sources`.** Validate these strictly; **ignore unknown top-level keys** — the file carries TypeScript-owned keys (`pending_in_git`, `confidence_thresholds`, `extract`, `migrate`) that no other port models (spec §4). +- **`schema_version` must equal `1`.** Any other value is an error. +- **Relative `path` resolves against the directory holding the `.metaobjects/` folder** — never ambient cwd. Absolute paths as-is (spec §3). +- **Default when `sources` is absent or empty:** exactly one `path` source, the literal `metaobjects` (spec §3, `metadata-files.ts:33`). +- **Metadata file extensions:** `.json`, `.yaml`, `.yml`, matched case-insensitively. All four ports already agree — do not change any port's existing set. +- **File ORDER is NOT a contract.** Each port keeps its existing `DirectorySource` walk order. Corpus comparisons are order-insensitive (set equality) (spec §3). +- **Error codes, already registered in all five ledgers — do not add any:** `ERR_SOURCE_UNRESOLVED` (a declared path does not exist), `ERR_SOURCE_KIND_UNSUPPORTED` (`resource`/`package`), `ERR_COLLECTION_NOT_FOUND` (nothing declared and no default dir). +- **Precedence ladder, first match wins** (spec §5): explicit CLI argument → port's native config surface → `.metaobjects/config.json` `sources` → built-in default. **A file that exists but is malformed is an ERROR at its own rung — never a fall-through.** +- **Public repo.** No absolute home paths, no other-project names, in code, tests, fixtures, docs or commit messages. +- **All four readers ship in ONE changeset.** Non-TS lanes do not run on PRs (`AGENTS.md:91-92`), so a per-port deferral goes green locally and turns ports red on the next push to `main`. + +--- + +## File Structure + +**New shared corpus** +- `fixtures/source-resolution-conformance/cases.json` — the single source of truth for every port's expectations +- `fixtures/source-resolution-conformance/README.md` — grammar, semantics, runner contract + +**TypeScript** (reference implementation already exists; adds a runner only) +- Create: `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts` + +**Python** +- Create: `server/python/src/metaobjects/config/neutral_config.py` — neutral-subset reader +- Create: `server/python/src/metaobjects/config/source_resolver.py` — spec set → file list +- Create: `server/python/src/metaobjects/config/__init__.py` +- Modify: `server/python/src/metaobjects/cli.py` — fallback wiring +- Create: `server/python/tests/config/test_neutral_config.py` +- Create: `server/python/tests/config/test_source_resolver.py` +- Create: `server/python/tests/conformance/test_source_resolution_conformance.py` + +**C#** +- Create: `server/csharp/MetaObjects/Config/NeutralConfig.cs` +- Create: `server/csharp/MetaObjects/Config/SourceResolver.cs` +- Modify: `server/csharp/MetaObjects.Cli/Program.cs` — fallback wiring +- Create: `server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs` +- Create: `server/csharp/MetaObjects.Tests/Config/NeutralConfigTests.cs` + +**Java** (Kotlin inherits — it has no CLI entry point of its own) +- Create: `server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java` +- Create: `server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java` +- Modify: `server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java` — fallback wiring +- Create: `server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java` +- Create: `server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java` + +**Node writer** +- Modify: `server/typescript/packages/cli/src/commands/init.ts` — config-only mode +- Modify: `server/typescript/packages/cli/test/init.test.ts` + +**Docs** +- Modify: `docs/features/metadata-sources.md:36-42` — the "Port support" paragraph +- Modify: `CHANGELOG.md` — under `## [Unreleased]` + +--- + +## Task 1: The shared corpus and the TypeScript reference runner + +The corpus is filesystem-based, unlike `scope-conformance` which is pure string matching. Each case declares a tree to materialize, a config to write, and either an expected file set or an expected error code. + +**Files:** +- Create: `fixtures/source-resolution-conformance/cases.json` +- Create: `fixtures/source-resolution-conformance/README.md` +- Create: `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts` + +**Interfaces:** +- Produces: the case-file schema every later task's runner consumes — + `{ cases: [{ name, tree: {relPath: "content"}, config: object | null, expectFiles?: string[], expectError?: string }] }`. + `tree` keys are paths relative to the materialized project root. `config` is written to `.metaobjects/config.json` when non-null; when null, no config file is created. `expectFiles` are project-root-relative paths, compared as an **unordered set**. `expectError` is an error code string. + +- [ ] **Step 1: Write the corpus cases file** + +Create `fixtures/source-resolution-conformance/cases.json`: + +```json +{ + "cases": [ + { + "name": "no-config-uses-default-directory", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": null, + "expectFiles": ["metaobjects/meta.users.json"] + }, + { + "name": "empty-sources-uses-default-directory", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [] }, + "expectFiles": ["metaobjects/meta.users.json"] + }, + { + "name": "declared-path-replaces-the-default-entirely", + "tree": { + "metaobjects/ignored.json": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.orders.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.orders.json"] + }, + { + "name": "directory-is-walked-recursively", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/nested/meta.b.yaml": "metadata.root:\n children: []\n", + "model/nested/deep/meta.c.yml": "metadata.root:\n children: []\n" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [ + "model/meta.a.json", + "model/nested/meta.b.yaml", + "model/nested/deep/meta.c.yml" + ] + }, + { + "name": "non-metadata-extensions-are-ignored", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/README.md": "not metadata", + "model/notes.txt": "not metadata", + "model/script.ts": "export {}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.a.json"] + }, + { + "name": "a-single-file-path-resolves-to-that-file", + "tree": { + "vendor/meta.catalog.json": "{\"metadata.root\":{\"children\":[]}}", + "vendor/meta.other.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "vendor/meta.catalog.json" }] }, + "expectFiles": ["vendor/meta.catalog.json"] + }, + { + "name": "two-sources-union", + "tree": { + "a/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "b/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "a" }, { "path": "b" }] }, + "expectFiles": ["a/meta.a.json", "b/meta.b.json"] + }, + { + "name": "overlapping-sources-yield-each-file-once", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/nested/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "model" }, { "path": "model/nested" }] + }, + "expectFiles": ["model/meta.a.json", "model/nested/meta.b.json"] + }, + { + "name": "source-order-does-not-change-the-resolved-set", + "tree": { + "a/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "b/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "b" }, { "path": "a" }] }, + "expectFiles": ["a/meta.a.json", "b/meta.b.json"] + }, + { + "name": "a-parent-relative-path-resolves-against-the-config-directory", + "tree": { + "shared/meta.shared.json": "{\"metadata.root\":{\"children\":[]}}", + "app/.keep": "" + }, + "config": { "schema_version": 1, "sources": [{ "path": "shared" }] }, + "expectFiles": ["shared/meta.shared.json"] + }, + { + "name": "a-declared-path-that-does-not-exist-is-an-error", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "nope" }] }, + "expectError": "ERR_SOURCE_UNRESOLVED" + }, + { + "name": "an-empty-directory-source-resolves-to-no-files", + "tree": { + "model/.keep": "" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [] + }, + { + "name": "resource-kind-is-unsupported", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "resource": "com/acme/model" }] }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "package-kind-is-unsupported", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "package": "acme-model" }] }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "no-config-and-no-default-directory-is-an-error", + "tree": { + "src/.keep": "" + }, + "config": null, + "expectError": "ERR_COLLECTION_NOT_FOUND" + }, + { + "name": "unknown-top-level-keys-are-ignored", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "model" }], + "pending_in_git": true, + "confidence_thresholds": { "pending_promote": 0.8 }, + "extract": { "metaignore": ".metaignore" }, + "migrate": { "dialect": "postgres" } + }, + "expectFiles": ["model/meta.a.json"] + } + ] +} +``` + +- [ ] **Step 2: Write the corpus README** + +Create `fixtures/source-resolution-conformance/README.md`: + +```markdown +# source-resolution-conformance + +Pins how a consumer's `sources` set in `.metaobjects/config.json` resolves to a +set of metadata files. Every port's CLI must resolve the SAME FILES from the +same declaration — that is the cross-port promise this corpus exists to keep. + +Companion to `scope-conformance/`, which pins the (currently TypeScript-only) +`scope` pattern grammar. The two are independent: `sources` decides which files +are read, `scope` filters what is emitted from them. + +## Shape + +``` +cases.json # { cases: [{ name, tree, config, expectFiles? , expectError? }] } +README.md +``` + +- **`tree`** — a map of project-root-relative path → file content. The runner + materializes it in a fresh temporary directory. A `.keep` entry exists only to + force an otherwise-empty directory to be created. +- **`config`** — written verbatim to `.metaobjects/config.json` under the project + root. When `null`, no config file is created at all. +- **`expectFiles`** — project-root-relative paths, compared as an **UNORDERED + SET**. See "Order is deliberately not pinned" below. +- **`expectError`** — an error code the resolution must fail with. Exactly one of + `expectFiles` / `expectError` is present per case. + +## Semantics pinned here + +- **Default.** `sources` absent or empty ⇒ exactly one `path` source, the literal + `metaobjects`. It is a default VALUE, never a requirement. +- **Replacement, not merge.** A declared `sources` replaces the default entirely — + the default directory is not implicitly appended. +- **Relative base.** A relative `path` resolves against the directory HOLDING the + `.metaobjects/` folder, never against the process working directory. +- **Recursion.** A directory `path` is walked recursively; a file `path` resolves + to that one file. +- **Extensions.** `.json`, `.yaml`, `.yml`, matched case-insensitively. Nothing else. +- **Union with de-duplication.** Overlapping sources yield each file exactly once. +- **A declared path that does not exist is `ERR_SOURCE_UNRESOLVED`** — never a + silent skip. Only the DEFAULT may be absent, and then it is + `ERR_COLLECTION_NOT_FOUND`. +- **`resource` and `package` kinds are declared but resolve nowhere yet:** + `ERR_SOURCE_KIND_UNSUPPORTED`. +- **Unknown top-level config keys are IGNORED.** The file carries + TypeScript-owned keys no other port models. `schema_version` and `sources` are + the neutral subset; each port validates those strictly and ignores the rest. + +## Order is deliberately NOT pinned + +`expectFiles` is a set. The ports' directory walks already differ and always +have — Java sorts by basename (`DirectorySource.java:105`), C# by full-path +ordinal (`DirectorySource.cs:64`), Python by basename +(`directory_source.py:40-48`), TypeScript walks depth-first with files before +subdirectories (`metadata-files.ts:101-121`). Making order a contract would be a +behavior change in three ports for no benefit: super-resolution is +order-independent (#188) and the loader's overlay partition discards caller +order anyway. + +A port MAY have a stable internal order — several do, and their own generated +output depends on it. It just is not a cross-port promise. + +## Behavioral contract + +Each port's runner reads `cases.json`, and for every case: materializes `tree` +in a fresh temp directory, writes `config` when non-null, resolves sources +against that root, then asserts either that the resolved file set equals +`expectFiles` (as a set, project-root-relative, path separators normalized to +`/`) or that resolution failed with `expectError`. + +## Reference implementation + +`server/typescript/packages/sdk/src/sources.ts` (`resolveSources`) and +`server/typescript/packages/sdk/src/collection.ts` (`resolveCollection`). +``` + +- [ ] **Step 3: Write the failing TypeScript runner** + +Create `server/typescript/packages/sdk/test/source-resolution-conformance.test.ts`: + +```ts +// Runs the shared source-resolution corpus against the TypeScript reference +// implementation. Every port ships an equivalent runner reading this same file. +import { describe, expect, test } from "bun:test"; +import { mkdtemp, mkdir, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, relative, resolve, sep } from "node:path"; +import { resolveCollection } from "../src/collection.js"; + +interface Case { + readonly name: string; + readonly tree: Record; + readonly config: unknown | null; + readonly expectFiles?: readonly string[]; + readonly expectError?: string; +} + +const CORPUS = resolve(import.meta.dir, "../../../../../fixtures/source-resolution-conformance/cases.json"); + +async function materialize(c: Case): Promise { + const root = await mkdtemp(join(tmpdir(), "mo-src-conf-")); + for (const [rel, content] of Object.entries(c.tree)) { + const abs = join(root, rel); + await mkdir(dirname(abs), { recursive: true }); + await writeFile(abs, content); + } + if (c.config !== null) { + await mkdir(join(root, ".metaobjects"), { recursive: true }); + await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify(c.config, null, 2)); + } + return root; +} + +const cases: Case[] = JSON.parse(await readFile(CORPUS, "utf8")).cases; + +describe("source-resolution conformance", () => { + for (const c of cases) { + test(c.name, async () => { + const root = await materialize(c); + if (c.expectError !== undefined) { + let code: string | undefined; + try { + await resolveCollection(root, { explicitDir: root }); + } catch (e) { + code = (e as { code?: string }).code; + } + expect(code).toBe(c.expectError); + return; + } + const collection = await resolveCollection(root, { explicitDir: root }); + const got = collection.files.map((f) => relative(root, f).split(sep).join("/")).sort(); + expect(got).toEqual([...(c.expectFiles ?? [])].sort()); + }); + } +}); +``` + +- [ ] **Step 4: Run it and confirm every case passes** + +Run: `cd server/typescript/packages/sdk && bun test test/source-resolution-conformance.test.ts` +Expected: **16 pass, 0 fail.** The TypeScript side is the already-shipped reference — this runner asserts the corpus describes real behavior, so a failure here means the CORPUS is wrong, not the code. Fix `cases.json` until it matches, and do not change `sources.ts` or `collection.ts` in this task. + +- [ ] **Step 5: Typecheck** + +Run: `cd server/typescript && bun run --filter '*' typecheck` +Expected: clean. (`bun test` does not typecheck — a break here would otherwise sit red through later tasks.) + +- [ ] **Step 6: Commit** + +```bash +git add fixtures/source-resolution-conformance server/typescript/packages/sdk/test/source-resolution-conformance.test.ts +git commit -m "test(conformance): a corpus for cross-port source resolution + +Pins the resolved file SET — not its order, which already differs by port +and is not a contract. TypeScript is the reference and ships the first +runner; the other three follow in this changeset." +``` + +--- + +## Task 2: Python — neutral config reader and source resolver + +**Files:** +- Create: `server/python/src/metaobjects/config/__init__.py` +- Create: `server/python/src/metaobjects/config/neutral_config.py` +- Create: `server/python/src/metaobjects/config/source_resolver.py` +- Test: `server/python/tests/config/test_neutral_config.py` +- Test: `server/python/tests/config/test_source_resolver.py` + +**Interfaces:** +- Consumes: `fixtures/source-resolution-conformance/cases.json` (Task 1); the existing `metaobjects.errors.ErrorCode` members `ERR_SOURCE_UNRESOLVED`, `ERR_SOURCE_KIND_UNSUPPORTED`, `ERR_COLLECTION_NOT_FOUND` (`errors.py:108,110,114`). +- Produces: + - `read_neutral_config(config_dir: Path) -> NeutralConfig | None` — `None` when no `.metaobjects/config.json` exists; raises on malformed. + - `NeutralConfig` dataclass with field `sources: list[dict[str, str]]`. + - `DEFAULT_METADATA_DIR: str = "metaobjects"`. + - `resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path]`. + - `resolve_collection(root: Path) -> list[Path]` — the full ladder: config-or-default, then resolve. + +- [ ] **Step 1: Write the failing config-reader test** + +Create `server/python/tests/config/test_neutral_config.py`: + +```python +import json +import pytest +from pathlib import Path + +from metaobjects.config.neutral_config import read_neutral_config +from metaobjects.errors import MetaObjectsError + + +def _write_config(root: Path, payload: object) -> None: + d = root / ".metaobjects" + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(payload)) + + +def test_absent_config_returns_none(tmp_path: Path) -> None: + assert read_neutral_config(tmp_path) is None + + +def test_reads_sources(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": "model"}]}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "model"}] + + +def test_unknown_top_level_keys_are_ignored(tmp_path: Path) -> None: + # The file carries TypeScript-owned keys this port must not model. + _write_config( + tmp_path, + { + "schema_version": 1, + "sources": [{"path": "model"}], + "pending_in_git": True, + "confidence_thresholds": {"pending_promote": 0.8}, + "extract": {"metaignore": ".metaignore"}, + "migrate": {"dialect": "postgres"}, + }, + ) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "model"}] + + +def test_absent_sources_key_yields_empty_list(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [] + + +def test_malformed_json_raises_not_none(tmp_path: Path) -> None: + d = tmp_path / ".metaobjects" + d.mkdir(parents=True) + (d / "config.json").write_text("{ not json") + # A file that EXISTS but cannot be read must never look like no config at all. + with pytest.raises(MetaObjectsError): + read_neutral_config(tmp_path) + + +def test_wrong_schema_version_raises(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 2, "sources": []}) + with pytest.raises(MetaObjectsError): + read_neutral_config(tmp_path) +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/python && uv run pytest tests/config/test_neutral_config.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'metaobjects.config'` + +- [ ] **Step 3: Implement the reader** + +Create `server/python/src/metaobjects/config/__init__.py`: + +```python +"""Port-neutral `.metaobjects/config.json` reading and source resolution. + +Reads only the NEUTRAL SUBSET (`schema_version`, `sources`). The file also +carries TypeScript-owned keys; those are ignored rather than modeled, so a new +TS-only key never becomes a four-port change. See +`docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §4. +""" +from .neutral_config import DEFAULT_METADATA_DIR, NeutralConfig, read_neutral_config +from .source_resolver import resolve_collection, resolve_sources + +__all__ = [ + "DEFAULT_METADATA_DIR", + "NeutralConfig", + "read_neutral_config", + "resolve_collection", + "resolve_sources", +] +``` + +Create `server/python/src/metaobjects/config/neutral_config.py`: + +```python +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path + +from metaobjects.errors import ErrorCode, MetaObjectsError + +#: The DEFAULT value of `sources` when the key is absent or empty — never a +#: requirement, and never assumed to exist by any other code path. +DEFAULT_METADATA_DIR = "metaobjects" + +_METAOBJECTS_DIR = ".metaobjects" +_CONFIG_FILE = "config.json" + + +@dataclass(frozen=True) +class NeutralConfig: + """The port-neutral subset of `.metaobjects/config.json`.""" + + #: Raw source specs, each a single-key mapping (`path` / `resource` / `package`). + sources: list[dict[str, str]] + + +def read_neutral_config(config_dir: Path) -> NeutralConfig | None: + """Read the neutral subset from ``config_dir/.metaobjects/config.json``. + + Returns ``None`` when the file does not exist. A file that EXISTS but is + malformed raises — swallowing it would make a typo'd config behave + identically to no config at all, silently resolving a possibly-stale + default directory with no diagnostic. + """ + path = config_dir / _METAOBJECTS_DIR / _CONFIG_FILE + if not path.is_file(): + return None + + try: + raw = json.loads(path.read_text()) + except (OSError, ValueError) as e: + raise MetaObjectsError( + f"{path} exists but could not be read as JSON: {e}", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) from e + + if not isinstance(raw, dict): + raise MetaObjectsError( + f"{path} must contain a JSON object", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + version = raw.get("schema_version") + if version != 1: + raise MetaObjectsError( + f"{path}: unsupported schema_version {version!r} (expected 1)", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + sources = raw.get("sources", []) + if not isinstance(sources, list) or not all( + isinstance(s, dict) and len(s) == 1 for s in sources + ): + raise MetaObjectsError( + f"{path}: 'sources' must be an array of single-key objects", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + # Unknown top-level keys are IGNORED by design — see the module docstring. + return NeutralConfig(sources=[dict(s) for s in sources]) +``` + +- [ ] **Step 4: Run the reader tests** + +Run: `cd server/python && uv run pytest tests/config/test_neutral_config.py -v` +Expected: 6 passed. If `MetaObjectsError`'s constructor signature differs, adapt the raise sites to the real one — check with `grep -n "class MetaObjectsError" -A12 server/python/src/metaobjects/errors.py` and use whatever the existing code passes. + +- [ ] **Step 5: Write the failing resolver test** + +Create `server/python/tests/config/test_source_resolver.py`: + +```python +import json +import pytest +from pathlib import Path + +from metaobjects.config.source_resolver import resolve_collection, resolve_sources +from metaobjects.errors import ErrorCode, MetaObjectsError + + +def _rel(root: Path, files: list[Path]) -> set[str]: + return {p.relative_to(root).as_posix() for p in files} + + +def test_directory_is_walked_recursively(tmp_path: Path) -> None: + (tmp_path / "model" / "nested").mkdir(parents=True) + (tmp_path / "model" / "a.json").write_text("{}") + (tmp_path / "model" / "nested" / "b.yaml").write_text("{}") + (tmp_path / "model" / "README.md").write_text("x") + got = resolve_sources(tmp_path, [{"path": "model"}]) + assert _rel(tmp_path, got) == {"model/a.json", "model/nested/b.yaml"} + + +def test_single_file_spec(tmp_path: Path) -> None: + (tmp_path / "vendor").mkdir() + (tmp_path / "vendor" / "one.json").write_text("{}") + (tmp_path / "vendor" / "two.json").write_text("{}") + got = resolve_sources(tmp_path, [{"path": "vendor/one.json"}]) + assert _rel(tmp_path, got) == {"vendor/one.json"} + + +def test_overlapping_sources_dedupe(tmp_path: Path) -> None: + (tmp_path / "model" / "nested").mkdir(parents=True) + (tmp_path / "model" / "a.json").write_text("{}") + (tmp_path / "model" / "nested" / "b.json").write_text("{}") + got = resolve_sources(tmp_path, [{"path": "model"}, {"path": "model/nested"}]) + assert _rel(tmp_path, got) == {"model/a.json", "model/nested/b.json"} + assert len(got) == 2 + + +def test_missing_path_raises_unresolved(tmp_path: Path) -> None: + with pytest.raises(MetaObjectsError) as e: + resolve_sources(tmp_path, [{"path": "nope"}]) + assert e.value.code == ErrorCode.ERR_SOURCE_UNRESOLVED + + +def test_resource_kind_unsupported(tmp_path: Path) -> None: + with pytest.raises(MetaObjectsError) as e: + resolve_sources(tmp_path, [{"resource": "com/acme"}]) + assert e.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED + + +def test_collection_falls_back_to_default_dir(tmp_path: Path) -> None: + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "a.json").write_text("{}") + got = resolve_collection(tmp_path) + assert _rel(tmp_path, got) == {"metaobjects/a.json"} + + +def test_collection_with_no_config_and_no_default_raises(tmp_path: Path) -> None: + with pytest.raises(MetaObjectsError) as e: + resolve_collection(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND + + +def test_declared_sources_replace_the_default(tmp_path: Path) -> None: + (tmp_path / ".metaobjects").mkdir() + (tmp_path / ".metaobjects" / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "ignored.json").write_text("{}") + (tmp_path / "model").mkdir() + (tmp_path / "model" / "used.json").write_text("{}") + got = resolve_collection(tmp_path) + assert _rel(tmp_path, got) == {"model/used.json"} +``` + +- [ ] **Step 6: Run it to verify it fails** + +Run: `cd server/python && uv run pytest tests/config/test_source_resolver.py -v` +Expected: FAIL — `ImportError: cannot import name 'resolve_sources'` + +- [ ] **Step 7: Implement the resolver** + +Create `server/python/src/metaobjects/config/source_resolver.py`: + +```python +from __future__ import annotations + +from pathlib import Path + +from metaobjects.errors import ErrorCode, MetaObjectsError + +from .neutral_config import DEFAULT_METADATA_DIR, read_neutral_config + +_SUPPORTED_SUFFIXES = (".json", ".yaml", ".yml") + + +def _list_metadata_files(directory: Path) -> list[Path]: + """Recursively list metadata files under ``directory``. + + Mirrors `DirectorySource`'s extension set (`.json`/`.yaml`/`.yml`, + case-insensitive). Order is this port's own and is deliberately NOT a + cross-port contract — see the corpus README. + """ + return sorted( + (p for p in directory.rglob("*") if p.is_file() and p.suffix.lower() in _SUPPORTED_SUFFIXES), + key=lambda p: p.name, + ) + + +def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path]: + """Resolve a declared source SET to a de-duplicated list of metadata files. + + A relative ``path`` resolves against ``config_dir`` — the directory HOLDING + the ``.metaobjects/`` folder — never against the process working directory. + """ + seen: dict[Path, None] = {} + + for spec in specs: + if "path" not in spec: + kind = next(iter(spec), "") + raise MetaObjectsError( + f'source kind "{kind}" is not supported by this toolchain yet; use a "path" source', + code=ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, + ) + + raw = Path(spec["path"]) + target = raw if raw.is_absolute() else (config_dir / raw) + + if not target.exists(): + raise MetaObjectsError( + f'source path "{spec["path"]}" does not exist ' + f"(resolved to {target}, relative to {config_dir})", + code=ErrorCode.ERR_SOURCE_UNRESOLVED, + ) + + found = _list_metadata_files(target) if target.is_dir() else [target] + for f in found: + seen.setdefault(f.resolve(), None) + + return list(seen) + + +def resolve_collection(root: Path) -> list[Path]: + """The full ladder: declared `sources`, else the default directory. + + Only the DEFAULT may be absent — a declared source that does not resolve is + `ERR_SOURCE_UNRESOLVED`, a louder failure. + """ + root = root.resolve() + cfg = read_neutral_config(root) + specs = cfg.sources if cfg is not None and cfg.sources else [] + + if not specs: + default_dir = root / DEFAULT_METADATA_DIR + if not default_dir.is_dir(): + raise MetaObjectsError( + f'no metadata sources declared in {root} and no default ' + f'"{DEFAULT_METADATA_DIR}" directory found. Declare "sources" in ' + f".metaobjects/config.json, or run 'meta init' to scaffold.", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + specs = [{"path": DEFAULT_METADATA_DIR}] + + return resolve_sources(root, specs) +``` + +- [ ] **Step 8: Run the resolver tests** + +Run: `cd server/python && uv run pytest tests/config/ -v` +Expected: 14 passed (6 reader + 8 resolver). + +- [ ] **Step 9: Commit** + +```bash +git add server/python/src/metaobjects/config server/python/tests/config +git commit -m "feat(python): read the port-neutral sources key + +Neutral subset only (schema_version + sources); unknown top-level keys are +ignored so a TypeScript-owned key never becomes a four-port change." +``` + +--- + +## Task 3: Python — conformance runner and CLI fallback + +**Files:** +- Create: `server/python/tests/conformance/test_source_resolution_conformance.py` +- Modify: `server/python/src/metaobjects/cli.py` + +**Interfaces:** +- Consumes: `resolve_collection` / `resolve_sources` (Task 2); `cases.json` (Task 1). +- Produces: CLI behavior — a `gen`/`verify` invocation with no positional `metadata_dir` and no `metadata` key in `metaobjects.config.yaml` falls back to `.metaobjects/config.json`. + +- [ ] **Step 1: Write the failing conformance runner** + +Create `server/python/tests/conformance/test_source_resolution_conformance.py`: + +```python +"""Runs the shared source-resolution corpus against this port. + +Reads `fixtures/source-resolution-conformance/cases.json` — the single +committed source of truth. There is no per-port fixture. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from metaobjects.config.source_resolver import resolve_collection +from metaobjects.errors import MetaObjectsError + +_CORPUS = ( + Path(__file__).resolve().parents[3] + / "fixtures" + / "source-resolution-conformance" + / "cases.json" +) + +_CASES = json.loads(_CORPUS.read_text())["cases"] + + +def _materialize(case: dict, root: Path) -> None: + for rel, content in case["tree"].items(): + p = root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(content) + if case["config"] is not None: + d = root / ".metaobjects" + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(case["config"], indent=2)) + + +@pytest.mark.parametrize("case", _CASES, ids=[c["name"] for c in _CASES]) +def test_source_resolution_conformance(case: dict, tmp_path: Path) -> None: + _materialize(case, tmp_path) + + if "expectError" in case: + with pytest.raises(MetaObjectsError) as e: + resolve_collection(tmp_path) + assert e.value.code.value == case["expectError"] + return + + got = {p.relative_to(tmp_path.resolve()).as_posix() for p in resolve_collection(tmp_path)} + assert got == set(case["expectFiles"]) +``` + +- [ ] **Step 2: Run it** + +Run: `cd server/python && uv run pytest tests/conformance/test_source_resolution_conformance.py -v` +Expected: 16 passed. If `e.value.code` is already a plain string rather than an enum, drop the `.value`. If the corpus path resolution is wrong, print `_CORPUS` and correct the `parents[N]` index — the file must resolve to the repository's `fixtures/` directory. + +- [ ] **Step 3: Write the failing CLI fallback test** + +Add to `server/python/tests/conformance/test_source_resolution_conformance.py`: + +```python +def test_cli_falls_back_to_neutral_config(tmp_path: Path, monkeypatch) -> None: + """No positional metadata_dir and no YAML `metadata` key ⇒ neutral config wins.""" + (tmp_path / "model").mkdir() + (tmp_path / "model" / "meta.a.json").write_text('{"metadata.root":{"children":[]}}') + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + + from metaobjects.cli import resolve_metadata_location + + monkeypatch.chdir(tmp_path) + got = resolve_metadata_location(explicit=None, config=None, root=tmp_path) + assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { + "model/meta.a.json" + } +``` + +- [ ] **Step 4: Run it to verify it fails** + +Run: `cd server/python && uv run pytest tests/conformance/test_source_resolution_conformance.py::test_cli_falls_back_to_neutral_config -v` +Expected: FAIL — `ImportError: cannot import name 'resolve_metadata_location'` + +- [ ] **Step 5: Add the ladder function to the CLI** + +Add to `server/python/src/metaobjects/cli.py` (near the other resolution helpers, around the existing `gen_state_dir_for` at line 387): + +```python +def resolve_metadata_location( + explicit: str | None, + config: "ProjectConfig | None", + root: Path, +) -> list[str]: + """The precedence ladder for where metadata lives. First match wins. + + 1. An explicit CLI argument (the positional ``metadata_dir``). + 2. This port's native surface — ``metadata`` in ``metaobjects.config.yaml``. + 3. ``sources`` in the port-neutral ``.metaobjects/config.json``. + 4. The built-in default directory. + + A file that EXISTS at any rung but is malformed raises rather than falling + through to the next rung. See + `docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §5. + """ + from metaobjects.config.source_resolver import resolve_collection, resolve_sources + + if explicit is not None: + return [str(p) for p in resolve_sources(Path(explicit).resolve().parent, [{"path": explicit}])] + + if config is not None: + return [str(p) for p in resolve_sources(Path(config.metadata_dir()).parent, [{"path": config.metadata_dir()}])] + + # Rungs 3 and 4 both live in `resolve_collection`. + return [str(p) for p in resolve_collection(root)] +``` + +- [ ] **Step 6: Run the test** + +Run: `cd server/python && uv run pytest tests/conformance/test_source_resolution_conformance.py -v` +Expected: 17 passed. + +- [ ] **Step 7: Run the whole Python suite for regressions** + +Run: `cd server/python && uv run --extra integration pytest -q` +Expected: no new failures versus the pre-task baseline. (Capture the baseline first with the same command on a clean tree if you have not already — `uv.lock` re-dirties on any `uv run`; do NOT commit it.) + +- [ ] **Step 8: Commit** + +```bash +git add server/python/src/metaobjects/cli.py server/python/tests/conformance/test_source_resolution_conformance.py +git commit -m "feat(python): the neutral config is the fallback when nothing else names a location + +Ladder: explicit arg > metaobjects.config.yaml > .metaobjects/config.json > +default dir. Gated by the shared source-resolution corpus." +``` + +--- + +## Task 4: C# — neutral config reader, resolver, conformance runner, CLI fallback + +**Files:** +- Create: `server/csharp/MetaObjects/Config/NeutralConfig.cs` +- Create: `server/csharp/MetaObjects/Config/SourceResolver.cs` +- Create: `server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs` +- Modify: `server/csharp/MetaObjects.Cli/Program.cs` + +**Interfaces:** +- Consumes: `cases.json` (Task 1); `MetaObjects.Errors` members `ERR_SOURCE_UNRESOLVED`, `ERR_SOURCE_KIND_UNSUPPORTED`, `ERR_COLLECTION_NOT_FOUND` (`Errors.cs:132,134,138`). +- Produces: + - `MetaObjects.Config.NeutralConfig.Read(string configDir) -> NeutralConfig?` + - `NeutralConfig.Sources -> IReadOnlyList>` + - `MetaObjects.Config.SourceResolver.ResolveCollection(string root) -> IReadOnlyList` + - `MetaObjects.Config.SourceResolver.ResolveSources(string configDir, IReadOnlyList> specs) -> IReadOnlyList` + - `NeutralConfig.DefaultMetadataDir -> "metaobjects"` + +- [ ] **Step 1: Write the failing conformance runner** + +Create `server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs`: + +```csharp +// Runs the shared source-resolution corpus against this port. Reads the single +// committed fixtures/source-resolution-conformance/cases.json — no per-port fixture. +using System.Text.Json; +using MetaObjects.Config; +using Xunit; + +namespace MetaObjects.Conformance.Tests; + +public class SourceResolutionConformanceTests +{ + private sealed record Case( + string Name, + Dictionary Tree, + JsonElement? Config, + string[]? ExpectFiles, + string? ExpectError); + + public static TheoryData CaseNames() + { + var data = new TheoryData(); + foreach (var c in LoadCases()) data.Add(c.Name); + return data; + } + + private static string CorpusPath() + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !Directory.Exists(Path.Combine(dir.FullName, "fixtures"))) + dir = dir.Parent; + Assert.NotNull(dir); + return Path.Combine(dir!.FullName, "fixtures", "source-resolution-conformance", "cases.json"); + } + + private static List LoadCases() + { + using var doc = JsonDocument.Parse(File.ReadAllText(CorpusPath())); + var cases = new List(); + foreach (var el in doc.RootElement.GetProperty("cases").EnumerateArray()) + { + var tree = new Dictionary(); + foreach (var p in el.GetProperty("tree").EnumerateObject()) + tree[p.Name] = p.Value.GetString() ?? ""; + + var cfgEl = el.GetProperty("config"); + JsonElement? cfg = cfgEl.ValueKind == JsonValueKind.Null ? null : cfgEl.Clone(); + + string[]? expectFiles = el.TryGetProperty("expectFiles", out var ef) + ? ef.EnumerateArray().Select(x => x.GetString()!).ToArray() + : null; + string? expectError = el.TryGetProperty("expectError", out var ee) ? ee.GetString() : null; + + cases.Add(new Case(el.GetProperty("name").GetString()!, tree, cfg, expectFiles, expectError)); + } + return cases; + } + + [Theory] + [MemberData(nameof(CaseNames))] + public void ResolvesTheSameFileSet(string name) + { + var c = LoadCases().Single(x => x.Name == name); + var root = Path.Combine(Path.GetTempPath(), "mo-src-conf-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(root); + try + { + foreach (var (rel, content) in c.Tree) + { + var abs = Path.Combine(root, rel.Replace('/', Path.DirectorySeparatorChar)); + Directory.CreateDirectory(Path.GetDirectoryName(abs)!); + File.WriteAllText(abs, content); + } + if (c.Config is not null) + { + var d = Path.Combine(root, ".metaobjects"); + Directory.CreateDirectory(d); + File.WriteAllText(Path.Combine(d, "config.json"), c.Config.Value.GetRawText()); + } + + if (c.ExpectError is not null) + { + var ex = Assert.ThrowsAny(() => SourceResolver.ResolveCollection(root)); + Assert.Equal(c.ExpectError, ex.Code.ToString()); + return; + } + + var got = SourceResolver.ResolveCollection(root) + .Select(f => Path.GetRelativePath(root, f).Replace(Path.DirectorySeparatorChar, '/')) + .ToHashSet(); + Assert.Equal(c.ExpectFiles!.ToHashSet(), got); + } + finally + { + Directory.Delete(root, recursive: true); + } + } +} +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/csharp && dotnet test MetaObjects.Conformance.Tests/MetaObjects.Conformance.Tests.csproj --nologo --verbosity quiet` +Expected: FAIL to COMPILE — `MetaObjects.Config` does not exist. If `MetaObjectsException` is named differently, check with `grep -n "class .*Exception" server/csharp/MetaObjects/Errors.cs` and use the real type and its code property. + +- [ ] **Step 3: Implement the reader** + +Create `server/csharp/MetaObjects/Config/NeutralConfig.cs`: + +```csharp +// Port-neutral `.metaobjects/config.json` reading. +// +// Reads only the NEUTRAL SUBSET (`schema_version`, `sources`). The file also +// carries TypeScript-owned keys; those are IGNORED rather than modeled, so a new +// TS-only key never becomes a four-port change. See +// docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md §4. +using System.Text.Json; + +namespace MetaObjects.Config; + +public sealed class NeutralConfig +{ + /// The DEFAULT value of `sources` when the key is absent or empty — never a + /// requirement, and never assumed to exist by any other code path. + public const string DefaultMetadataDir = "metaobjects"; + + private const string MetaObjectsDir = ".metaobjects"; + private const string ConfigFile = "config.json"; + + public IReadOnlyList> Sources { get; } + + private NeutralConfig(IReadOnlyList> sources) => Sources = sources; + + /// Returns null when the file does not exist. A file that EXISTS but is + /// malformed throws — swallowing it would make a typo'd config behave + /// identically to no config at all. + public static NeutralConfig? Read(string configDir) + { + var path = Path.Combine(configDir, MetaObjectsDir, ConfigFile); + if (!File.Exists(path)) return null; + + JsonDocument doc; + try + { + doc = JsonDocument.Parse(File.ReadAllText(path)); + } + catch (Exception e) + { + throw new MetaObjectsException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + $"{path} exists but could not be read as JSON: {e.Message}"); + } + + using (doc) + { + var root = doc.RootElement; + if (root.ValueKind != JsonValueKind.Object) + throw new MetaObjectsException(ErrorCode.ERR_COLLECTION_NOT_FOUND, $"{path} must contain a JSON object"); + + if (!root.TryGetProperty("schema_version", out var v) || v.ValueKind != JsonValueKind.Number || v.GetInt32() != 1) + throw new MetaObjectsException(ErrorCode.ERR_COLLECTION_NOT_FOUND, $"{path}: unsupported schema_version (expected 1)"); + + var specs = new List>(); + if (root.TryGetProperty("sources", out var srcs) && srcs.ValueKind == JsonValueKind.Array) + { + foreach (var s in srcs.EnumerateArray()) + { + if (s.ValueKind != JsonValueKind.Object) + throw new MetaObjectsException(ErrorCode.ERR_COLLECTION_NOT_FOUND, $"{path}: each 'sources' entry must be an object"); + var d = new Dictionary(); + foreach (var p in s.EnumerateObject()) d[p.Name] = p.Value.GetString() ?? ""; + if (d.Count != 1) + throw new MetaObjectsException(ErrorCode.ERR_COLLECTION_NOT_FOUND, $"{path}: each 'sources' entry must have exactly one key"); + specs.Add(d); + } + } + + // Unknown top-level keys are IGNORED by design — see the file header. + return new NeutralConfig(specs); + } + } +} +``` + +- [ ] **Step 4: Implement the resolver** + +Create `server/csharp/MetaObjects/Config/SourceResolver.cs`: + +```csharp +namespace MetaObjects.Config; + +public static class SourceResolver +{ + private static readonly HashSet SupportedExtensions = + new(StringComparer.OrdinalIgnoreCase) { ".json", ".yaml", ".yml" }; + + /// Resolve a declared source SET to a de-duplicated list of metadata files. + /// A relative `path` resolves against `configDir` — the directory HOLDING the + /// `.metaobjects/` folder — never against the process working directory. + public static IReadOnlyList ResolveSources( + string configDir, + IReadOnlyList> specs) + { + var seen = new List(); + var known = new HashSet(StringComparer.Ordinal); + + foreach (var spec in specs) + { + if (!spec.TryGetValue("path", out var rawPath)) + { + var kind = spec.Keys.FirstOrDefault() ?? ""; + throw new MetaObjectsException( + ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, + $"source kind \"{kind}\" is not supported by this toolchain yet; use a \"path\" source"); + } + + var target = Path.IsPathRooted(rawPath) ? rawPath : Path.GetFullPath(Path.Combine(configDir, rawPath)); + + var isDir = Directory.Exists(target); + if (!isDir && !File.Exists(target)) + throw new MetaObjectsException( + ErrorCode.ERR_SOURCE_UNRESOLVED, + $"source path \"{rawPath}\" does not exist (resolved to {target}, relative to {configDir})"); + + // Order is this port's own and is deliberately NOT a cross-port + // contract — see the corpus README. + var found = isDir + ? Directory.EnumerateFiles(target, "*", SearchOption.AllDirectories) + .Where(p => SupportedExtensions.Contains(Path.GetExtension(p))) + .OrderBy(p => p, StringComparer.Ordinal) + : new[] { target }.AsEnumerable(); + + foreach (var f in found) + { + var full = Path.GetFullPath(f); + if (known.Add(full)) seen.Add(full); + } + } + + return seen; + } + + /// The full ladder: declared `sources`, else the default directory. + /// Only the DEFAULT may be absent — a declared source that does not resolve + /// is ERR_SOURCE_UNRESOLVED, a louder failure. + public static IReadOnlyList ResolveCollection(string root) + { + root = Path.GetFullPath(root); + var cfg = NeutralConfig.Read(root); + var specs = cfg?.Sources ?? Array.Empty>(); + + if (specs.Count == 0) + { + var defaultDir = Path.Combine(root, NeutralConfig.DefaultMetadataDir); + if (!Directory.Exists(defaultDir)) + throw new MetaObjectsException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + $"no metadata sources declared in {root} and no default \"{NeutralConfig.DefaultMetadataDir}\" " + + "directory found. Declare \"sources\" in .metaobjects/config.json, or run 'meta init' to scaffold."); + specs = new[] { (IReadOnlyDictionary)new Dictionary { ["path"] = NeutralConfig.DefaultMetadataDir } }; + } + + return ResolveSources(root, specs); + } +} +``` + +- [ ] **Step 5: Run the conformance runner** + +Run: `cd server/csharp && dotnet test MetaObjects.Conformance.Tests/MetaObjects.Conformance.Tests.csproj --nologo --verbosity quiet` +Expected: all 16 corpus cases pass. + +- [ ] **Step 6: Wire the CLI fallback** + +Modify `server/csharp/MetaObjects.Cli/Program.cs`. The positional `` is currently required for `gen` (`:76`), `docs` (`:124`) and `verify` (`:204`). Make it OPTIONAL, falling back to the neutral config: + +```csharp +// Rung 1 is the explicit positional argument; rungs 3-4 live in ResolveCollection. +// C# has no native config surface (rung 2), so the ladder is two rungs here. +static string ResolveMetadataDirOrExit(string? metadataDir) +{ + if (metadataDir is not null) return metadataDir; + try + { + // Proves a collection resolves from cwd, and reports the same errors the + // other ports do. The loader still takes a directory, so hand it the + // resolved root rather than the file list — widening the loader call to a + // source SET is a separate change. + _ = MetaObjects.Config.SourceResolver.ResolveCollection(Directory.GetCurrentDirectory()); + return Path.Combine(Directory.GetCurrentDirectory(), MetaObjects.Config.NeutralConfig.DefaultMetadataDir); + } + catch (MetaObjectsException e) + { + Console.Error.WriteLine($"error: {e.Code}: {e.Message}"); + Environment.Exit(2); + throw; + } +} +``` + +Then at each of the three `if (metadataDir is null || outDir is null)` guards, replace the `metadataDir is null` half with a call to `ResolveMetadataDirOrExit(metadataDir)` assigned before the guard, leaving the `outDir is null` check intact. + +> **Note for the implementer:** the C# loader's `FromDirectory` takes one directory, so this task's CLI wiring resolves the *root*, not the file set. Widening `MetaDataLoader.FromDirectory` to accept a resolved source SET is deliberately NOT in this plan — `Load(IReadOnlyList)` already exists (`MetaDataLoader.cs:334`) and wiring it is a follow-up. What ships here is the reader, the resolver and the corpus. + +- [ ] **Step 7: Run the C# suite** + +Run: `cd server/csharp && dotnet test MetaObjects.Cli.Tests/MetaObjects.Cli.Tests.csproj --nologo --verbosity quiet && dotnet test MetaObjects.Conformance.Tests/MetaObjects.Conformance.Tests.csproj --nologo --verbosity quiet` +Expected: no new failures. + +- [ ] **Step 8: Commit** + +```bash +git add server/csharp/MetaObjects/Config server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs server/csharp/MetaObjects.Cli/Program.cs +git commit -m "feat(csharp): read the port-neutral sources key + +Neutral subset only; the positional metadataDir becomes optional and falls +back to .metaobjects/config.json. Gated by the shared corpus." +``` + +--- + +## Task 5: Java — neutral config reader, resolver, conformance runner, mojo fallback + +Kotlin needs no separate work: it has no CLI entry point of its own (`fun main` appears in neither `metadata-ktx` nor `codegen-kotlin`) and runs through this same Maven plugin. + +**Files:** +- Create: `server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java` +- Create: `server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java` +- Create: `server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java` +- Modify: `server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java` + +**Interfaces:** +- Consumes: `cases.json` (Task 1); `com.metaobjects.ErrorCode` members `ERR_SOURCE_UNRESOLVED`, `ERR_SOURCE_KIND_UNSUPPORTED`, `ERR_COLLECTION_NOT_FOUND` (`ErrorCode.java:287,290,296`). +- Produces: + - `NeutralConfig.read(Path configDir) -> Optional` + - `NeutralConfig.getSources() -> List>` + - `NeutralConfig.DEFAULT_METADATA_DIR -> "metaobjects"` + - `SourceResolver.resolveCollection(Path root) -> List` + - `SourceResolver.resolveSources(Path configDir, List> specs) -> List` + +- [ ] **Step 1: Write the failing conformance runner** + +Create `server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java`: + +```java +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.MethodSource; + +import javax.json.Json; // use the JSON API already on this module's classpath +import java.nio.file.*; +import java.util.*; +import java.util.stream.Collectors; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * Runs the shared source-resolution corpus against this port. Reads the single + * committed fixtures/source-resolution-conformance/cases.json — no per-port fixture. + */ +class SourceResolutionConformanceTest { + + private static Path corpus() { + Path dir = Paths.get("").toAbsolutePath(); + while (dir != null && !Files.isDirectory(dir.resolve("fixtures"))) dir = dir.getParent(); + assertNotNull(dir, "could not locate the repository fixtures/ directory"); + return dir.resolve("fixtures/source-resolution-conformance/cases.json"); + } + + static List> cases() throws Exception { + // Parse with whatever JSON facility this module already depends on. + return CorpusReader.read(corpus()); + } + + @ParameterizedTest(name = "{0}") + @MethodSource("cases") + void resolvesTheSameFileSet(Map c, @TempDir Path root) throws Exception { + CorpusReader.materialize(c, root); + + Object expectError = c.get("expectError"); + if (expectError != null) { + MetaDataException e = assertThrows(MetaDataException.class, + () -> SourceResolver.resolveCollection(root)); + assertEquals(expectError, e.getErrorCode().name()); + return; + } + + Set got = SourceResolver.resolveCollection(root).stream() + .map(p -> root.toRealPath_unchecked().relativize(p).toString().replace('\\', '/')) + .collect(Collectors.toSet()); + + @SuppressWarnings("unchecked") + Set want = new HashSet<>((List) c.get("expectFiles")); + assertEquals(want, got); + } +} +``` + +> **Implementer note:** `CorpusReader` and `toRealPath_unchecked()` above are placeholders for whatever this module already has. Before writing this file, run `grep -rn "cases.json" server/java/metadata/src/test --include=*.java | head` and copy the corpus-reading approach an existing conformance test uses (`ConformanceTest`, `RegistryManifestConformanceTest`). Do not add a new JSON dependency; reuse the module's existing one. Materialization is: for each `tree` entry create parent dirs and write the content; when `config` is non-null, write it to `/.metaobjects/config.json`. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/java && mvn -pl metadata test -Dtest=SourceResolutionConformanceTest -q` +Expected: compile failure — `com.metaobjects.config` does not exist. + +- [ ] **Step 3: Implement the reader** + +Create `server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java`: + +```java +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; + +import java.nio.file.*; +import java.util.*; + +/** + * The port-neutral subset of {@code .metaobjects/config.json}. + * + *

Reads only {@code schema_version} and {@code sources}. The file also carries + * TypeScript-owned keys ({@code pending_in_git}, {@code confidence_thresholds}, + * {@code extract}, {@code migrate}); those are IGNORED rather than modeled, so a new + * TS-only key never becomes a four-port change. See + * {@code docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md} §4. + */ +public final class NeutralConfig { + + /** + * The DEFAULT value of {@code sources} when the key is absent or empty — never a + * requirement, and never assumed to exist by any other code path. + */ + public static final String DEFAULT_METADATA_DIR = "metaobjects"; + + private static final String METAOBJECTS_DIR = ".metaobjects"; + private static final String CONFIG_FILE = "config.json"; + + private final List> sources; + + private NeutralConfig(List> sources) { + this.sources = List.copyOf(sources); + } + + public List> getSources() { + return sources; + } + + /** + * Returns empty when the file does not exist. A file that EXISTS but is malformed + * throws — swallowing it would make a typo'd config behave identically to no + * config at all. + */ + public static Optional read(Path configDir) { + Path path = configDir.resolve(METAOBJECTS_DIR).resolve(CONFIG_FILE); + if (!Files.isRegularFile(path)) return Optional.empty(); + + Map raw; + try { + raw = JsonSupport.readObject(path); + } catch (Exception e) { + throw new MetaDataException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + path + " exists but could not be read as JSON: " + e.getMessage()); + } + + Object version = raw.get("schema_version"); + if (!(version instanceof Number) || ((Number) version).intValue() != 1) { + throw new MetaDataException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + path + ": unsupported schema_version " + version + " (expected 1)"); + } + + List> specs = new ArrayList<>(); + Object srcs = raw.getOrDefault("sources", List.of()); + if (srcs instanceof List list) { + for (Object o : list) { + if (!(o instanceof Map m) || m.size() != 1) { + throw new MetaDataException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + path + ": each 'sources' entry must be an object with exactly one key"); + } + Map spec = new LinkedHashMap<>(); + m.forEach((k, v) -> spec.put(String.valueOf(k), String.valueOf(v))); + specs.add(spec); + } + } + + // Unknown top-level keys are IGNORED by design — see the class javadoc. + return Optional.of(new NeutralConfig(specs)); + } +} +``` + +> **Implementer note:** `JsonSupport.readObject` is a placeholder. Use whatever JSON reader `metadata` already depends on — find it with `grep -rn "import.*json" server/java/metadata/src/main/java/com/metaobjects/loader/parser/*.java | head`. Do not add a dependency. Likewise confirm `MetaDataException`'s constructor takes an `ErrorCode` and a message; adapt if it does not. + +- [ ] **Step 4: Implement the resolver** + +Create `server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java`: + +```java +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; + +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.file.*; +import java.util.*; +import java.util.stream.Stream; + +/** Resolves a declared source SET to a de-duplicated list of metadata files. */ +public final class SourceResolver { + + private static final Set EXTENSIONS = Set.of(".json", ".yaml", ".yml"); + + private SourceResolver() {} + + /** + * A relative {@code path} resolves against {@code configDir} — the directory + * HOLDING the {@code .metaobjects/} folder — never against the process working + * directory. + */ + public static List resolveSources(Path configDir, List> specs) { + LinkedHashSet seen = new LinkedHashSet<>(); + + for (Map spec : specs) { + String rawPath = spec.get("path"); + if (rawPath == null) { + String kind = spec.keySet().stream().findFirst().orElse(""); + throw new MetaDataException( + ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, + "source kind \"" + kind + "\" is not supported by this toolchain yet; use a \"path\" source"); + } + + Path raw = Paths.get(rawPath); + Path target = raw.isAbsolute() ? raw : configDir.resolve(raw).normalize(); + + boolean isDir = Files.isDirectory(target); + if (!isDir && !Files.isRegularFile(target)) { + throw new MetaDataException( + ErrorCode.ERR_SOURCE_UNRESOLVED, + "source path \"" + rawPath + "\" does not exist (resolved to " + target + + ", relative to " + configDir + ")"); + } + + if (isDir) { + // Order is this port's own and is deliberately NOT a cross-port + // contract — see the corpus README. + try (Stream walk = Files.walk(target)) { + walk.filter(Files::isRegularFile) + .filter(p -> hasSupportedExtension(p.getFileName().toString())) + .sorted(Comparator.comparing(p -> p.getFileName().toString())) + .forEach(p -> seen.add(p.toAbsolutePath().normalize())); + } catch (IOException e) { + throw new UncheckedIOException("Failed to list " + target, e); + } + } else { + seen.add(target.toAbsolutePath().normalize()); + } + } + + return new ArrayList<>(seen); + } + + /** + * The full ladder: declared {@code sources}, else the default directory. Only the + * DEFAULT may be absent — a declared source that does not resolve is + * {@code ERR_SOURCE_UNRESOLVED}, a louder failure. + */ + public static List resolveCollection(Path root) { + Path base = root.toAbsolutePath().normalize(); + List> specs = NeutralConfig.read(base) + .map(NeutralConfig::getSources) + .orElse(List.of()); + + if (specs.isEmpty()) { + Path defaultDir = base.resolve(NeutralConfig.DEFAULT_METADATA_DIR); + if (!Files.isDirectory(defaultDir)) { + throw new MetaDataException( + ErrorCode.ERR_COLLECTION_NOT_FOUND, + "no metadata sources declared in " + base + " and no default \"" + + NeutralConfig.DEFAULT_METADATA_DIR + "\" directory found. Declare \"sources\" in " + + ".metaobjects/config.json, or run 'meta init' to scaffold."); + } + specs = List.of(Map.of("path", NeutralConfig.DEFAULT_METADATA_DIR)); + } + + return resolveSources(base, specs); + } + + private static boolean hasSupportedExtension(String name) { + String lower = name.toLowerCase(Locale.ROOT); + for (String ext : EXTENSIONS) if (lower.endsWith(ext)) return true; + return false; + } +} +``` + +- [ ] **Step 5: Run the conformance runner** + +Run: `cd server/java && mvn -pl metadata test -Dtest=SourceResolutionConformanceTest -q` +Expected: all 16 corpus cases pass. + +- [ ] **Step 6: Wire the mojo fallback** + +Modify `server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java`. Rung 2 is the pom: if the `` names **either** `` or ``, the pom owns the concern and the neutral file is NOT consulted. Only when the pom names neither does the mojo fall back: + +```java +/** + * The precedence ladder for where metadata lives (spec §5). First match wins. + * + *

1. The pom — {@code } or {@code }. If EITHER is + * present the pom owns the whole concern and the neutral file is not consulted; + * precedence is whole-concern, not a per-entry merge. + *
2. {@code sources} in the port-neutral {@code .metaobjects/config.json}, read from + * the module basedir. + *
3. The built-in default directory. + * + *

A neutral file that EXISTS but is malformed throws rather than falling through. + */ +protected List resolveNeutralSourcesIfPomIsSilent(LoaderParam loaderConfig) { + boolean pomNamesLocation = + (loaderConfig.getSourceDir() != null && !loaderConfig.getSourceDir().isBlank()) + || (loaderConfig.getSources() != null && !loaderConfig.getSources().isEmpty()); + if (pomNamesLocation) return List.of(); + + return com.metaobjects.config.SourceResolver + .resolveCollection(getProjectBaseDir().toPath()) + .stream() + .map(java.nio.file.Path::toString) + .toList(); +} +``` + +Call it where the loader's sources are assembled and pass the result through the existing `setSourceURIs` / `sources` path. Use the mojo's existing accessor for the module base directory (find it with `grep -n "basedir\|getBasedir\|MavenProject" server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java | head`). + +**Do not touch ``.** It keeps its current semantics — `scope` is out of scope for this plan (Global Constraints). + +- [ ] **Step 7: Run the Java build** + +Run: `cd server/java && mvn -pl metadata,maven-plugin -am install -DskipTests -q && mvn -pl metadata test -q` +Expected: BUILD SUCCESS, no new failures. Do **not** pipe through `tail` — that reports `tail`'s exit status, not Maven's. + +- [ ] **Step 8: Commit** + +```bash +git add server/java/metadata/src/main/java/com/metaobjects/config server/java/metadata/src/test/java/com/metaobjects/config server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java +git commit -m "feat(java): read the port-neutral sources key when the pom is silent + +Whole-concern precedence: a pom naming sourceDir or sources owns the +concern outright. is untouched — scope stays TypeScript-only." +``` + +--- + +## Task 6: `meta init --config-only`, the Node-side writer + +The gap the phase-1 design named: a JVM- or pip-rooted adopter has no `.metaobjects/config.json` at all, so the Node CLI's `migrate` has nothing to discover no matter which ports can read. `meta init` today scaffolds the whole TypeScript project and has no flag to write only the config (`init.ts:635-650`). + +**Files:** +- Modify: `server/typescript/packages/cli/src/commands/init.ts` +- Test: `server/typescript/packages/cli/test/init.test.ts` + +**Interfaces:** +- Consumes: the existing `parseInitArgs` flag parser and `InitOptions` (`init.ts:635-650`). +- Produces: `meta init --config-only` writes exactly `.metaobjects/config.json` and nothing else. + +- [ ] **Step 1: Write the failing test** + +Add to `server/typescript/packages/cli/test/init.test.ts`: + +```ts +test("--config-only writes just the config, no TypeScript scaffold", async () => { + const root = await mkdtemp(join(tmpdir(), "mo-init-config-only-")); + + const result = await runInit({ cwd: root, configOnly: true }); + + // The one file it writes. + expect(result.created).toContain(".metaobjects/config.json"); + const cfg = JSON.parse(await readFile(join(root, ".metaobjects", "config.json"), "utf8")); + expect(cfg.schema_version).toBe(1); + expect(cfg.sources).toEqual([]); + + // None of the TypeScript scaffold — this is the whole point of the flag: a + // Maven- or pip-rooted project declares its sources for the Node CLI without + // acquiring a TS project it will not use. + for (const unwanted of [ + "metaobjects.config.ts", + "codegen/generators/entity.ts", + "package.json", + ".gitignore", + ]) { + expect(existsSync(join(root, unwanted))).toBe(false); + } +}); + +test("--config-only leaves an existing config untouched", async () => { + const root = await mkdtemp(join(tmpdir(), "mo-init-config-only-existing-")); + await mkdir(join(root, ".metaobjects"), { recursive: true }); + const existing = { schema_version: 1, sources: [{ path: "model" }] }; + await writeFile(join(root, ".metaobjects", "config.json"), JSON.stringify(existing)); + + const result = await runInit({ cwd: root, configOnly: true }); + + expect(result.preserved).toContain(".metaobjects/config.json"); + const cfg = JSON.parse(await readFile(join(root, ".metaobjects", "config.json"), "utf8")); + expect(cfg.sources).toEqual([{ path: "model" }]); +}); +``` + +> **Implementer note:** match the existing tests' helper names in that file — if the entry point is not `runInit`, or options are shaped differently, adapt. Check with `grep -n "runInit\|export async function init\|InitOptions" server/typescript/packages/cli/src/commands/init.ts | head`. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript/packages/cli && bun test test/init.test.ts -t "config-only"` +Expected: FAIL — `configOnly` is not a recognized option. + +- [ ] **Step 3: Implement the flag** + +In `init.ts`: add `configOnly?: boolean` to the options type, parse `--config-only` in `parseInitArgs`, thread it through to the entry point, and take an early-return branch that runs only the existing `.metaobjects/config.json` block (`:374-412`) — reusing that code rather than duplicating the write, so the two paths cannot drift on the config's default content. Add `--config-only` to the command's `--help` text with a one-line description naming its purpose (declaring sources for the Node CLI from a non-TypeScript project). + +- [ ] **Step 4: Run the test** + +Run: `cd server/typescript/packages/cli && bun test test/init.test.ts` +Expected: all pass, including the two new cases. + +- [ ] **Step 5: Typecheck** + +Run: `cd server/typescript && bun run --filter '*' typecheck` +Expected: clean. + +- [ ] **Step 6: Commit** + +```bash +git add server/typescript/packages/cli/src/commands/init.ts server/typescript/packages/cli/test/init.test.ts +git commit -m "feat(cli): meta init --config-only + +Writes .metaobjects/config.json and nothing else, so a Maven- or pip-rooted +project can declare its sources for the Node CLI without acquiring a +TypeScript scaffold it will not use." +``` + +--- + +## Task 7: Documentation and changelog + +**Files:** +- Modify: `docs/features/metadata-sources.md:36-42` +- Modify: `CHANGELOG.md` + +- [ ] **Step 1: Replace the "Port support" paragraph** + +`docs/features/metadata-sources.md` currently tells adopters the other four CLIs do not read the config. Replace that paragraph with: + +```markdown +**Port support.** `sources` is read by **all four CLI surfaces** — the Node `meta` +CLI, `dotnet meta` (C#), `metaobjects` (Python) and `metaobjects:generate` (Java +and Kotlin, via Maven). Each resolves the same files from the same declaration; +that promise is gated by +[`fixtures/source-resolution-conformance/`](../../fixtures/source-resolution-conformance/). + +Each port reaches it its own way, and the ladder is the same everywhere — first +match wins: + +1. An explicit CLI argument (a positional metadata directory, `--config`, `--cwd`). +2. The port's own native surface, where it has one — Java and Kotlin's pom + ``/``, Python's `metadata` key in `metaobjects.config.yaml`. + If the pom names either element it owns the concern outright and the neutral + file is not consulted. +3. `sources` in `.metaobjects/config.json`. +4. The built-in default — a `metaobjects/` directory beside that config. + +A config file that exists but is malformed is an error at its own rung; it never +falls through to the next one. + +The non-TypeScript ports read a **neutral subset** of that file — +`schema_version` and `sources` — and ignore every other top-level key, so the +TypeScript-owned keys beside them (`migrate`, `extract`, and the rest) never +become a four-port concern. The Node CLI remains the file's only writer; +`meta init --config-only` writes it into a Maven- or pip-rooted project without +adding a TypeScript scaffold. + +**`scope` and `migrate.scope` remain Node-CLI-only.** Java ships its own +`` grammar whose `*` and `@` mean different things from `scope`'s, so +reconciling them is a separate, adopter-affecting decision rather than a +mechanical port. + +**File order is not a cross-port promise.** Every port resolves the same file +SET; the order within it is each port's own and always has been. +``` + +- [ ] **Step 2: Add the changelog entry** + +Under `## [Unreleased]` in `CHANGELOG.md`: + +```markdown +### Added + +- **`sources` is read by all four CLI surfaces**, not just the Node `meta` CLI — + the C#, Python and Java/Kotlin CLIs now resolve metadata from the + port-neutral `.metaobjects/config.json`, so one declaration serves every port. + Each reads a **neutral subset** (`schema_version` + `sources`) and ignores + unknown top-level keys, so the TypeScript-owned keys in that file never become + a four-port change. Precedence is a ladder — explicit CLI argument, then the + port's native surface (a pom's ``/``, Python's `metadata` + key), then `sources`, then the default `metaobjects/` directory — and a config + that exists but is malformed errors at its rung rather than silently falling + through. Gated by the new `fixtures/source-resolution-conformance/` corpus, + which every port runs. +- **`meta init --config-only`** writes `.metaobjects/config.json` and nothing + else, so a Maven- or pip-rooted project can declare its sources for the Node + CLI (which owns `migrate` and `verify --db`, ADR-0015) without acquiring a + TypeScript scaffold it will not use. + +### Notes + +- **`scope` / `migrate.scope` stay Node-CLI-only for now.** Java has shipped a + `` grammar for years in which `*` crosses the `::` separator and `@` + matches one segment — the exact inverse of `scope`'s `*` and `**` — plus + `!`-prefix exclusion and a `.[attr]` predicate `scope` cannot express. + Reconciling them changes behavior for existing Java consumers, so it is its + own decision rather than a mechanical port. No cross-port behavior depends on + `scope`. +- **Resolved file ORDER is explicitly not a cross-port contract.** The ports' + directory walks already differ and always have; the corpus compares file SETS. + Super-resolution is order-independent (#188) and the loader's overlay + partition discards caller order regardless. +``` + +- [ ] **Step 3: Verify no leaked private names or absolute home paths** + +Run: `scripts/ci-local.sh --quick` +Expected: green. This is the same leak-scan that gates every PR, and it is the +authority — it checks staged content against the configured private-name +denylist as well as absolute user-home path patterns, which an ad-hoc grep here +would not. If it blocks a commit, **genericize the offending line** rather than +bypassing with `--no-verify`. + +- [ ] **Step 4: Commit** + +```bash +git add docs/features/metadata-sources.md CHANGELOG.md +git commit -m "docs: sources is read by all four CLI surfaces + +Records the precedence ladder, the neutral-subset rule, why scope stays +Node-only, and that file order is deliberately not a contract." +``` + +--- + +## Task 8: Full cross-port gate + +- [ ] **Step 1: Run the affected-port lanes** + +```bash +scripts/ci-local.sh --only typescript --strict-toolchains +scripts/ci-local.sh --only python --strict-toolchains +scripts/ci-local.sh --only csharp --strict-toolchains +scripts/ci-local.sh --only java --strict-toolchains +``` + +Expected: all four green. **Do not pipe any of these through `tail`** — the shell reports `tail`'s exit status, so a red lane reads as green. + +- [ ] **Step 2: Typecheck and build the TypeScript workspace** + +Run: `cd server/typescript && bun run --filter '*' build && bun run --filter '*' typecheck` +Expected: clean. + +- [ ] **Step 3: Confirm the corpus is genuinely gating** + +Prove the gate by breaking it, not by its silence. Temporarily change one `expectFiles` entry in `cases.json` to a wrong path, re-run **all four** runners, and confirm **each** goes red: + +```bash +cd server/typescript/packages/sdk && bun test test/source-resolution-conformance.test.ts # expect FAIL +cd server/python && uv run pytest tests/conformance/test_source_resolution_conformance.py # expect FAIL +cd server/csharp && dotnet test MetaObjects.Conformance.Tests/MetaObjects.Conformance.Tests.csproj # expect FAIL +cd server/java && mvn -pl metadata test -Dtest=SourceResolutionConformanceTest # expect FAIL +``` + +Then revert `cases.json`. **A runner that stays green here is not wired to the corpus** — fix it before proceeding. Four ports byte-matching one manifest is the entire point. + +- [ ] **Step 4: Verify the working tree is clean of incidental changes** + +Run: `git status --short` +Expected: empty. In particular `server/python/uv.lock` must NOT be staged — it re-dirties on any `uv run` and is genuine pre-existing drift on `main`. + +- [ ] **Step 5: Final commit if anything was fixed** + +```bash +git add -- +git commit -m "fix: close gaps found by the full cross-port gate" +``` + +Stage explicit paths — never `git add -A`. The stash stack and worktrees are shared across sessions. + +--- + +## Self-Review + +**Spec coverage.** §2 (`sources` ships, `scope` deferred) → Global Constraints + Tasks 2-5, deferral recorded in Task 7. §3 identical column → corpus cases in Task 1; may-differ column → the corpus README's order section and each resolver's order comment. §4 neutral subset → the `unknown-top-level-keys-are-ignored` case plus each reader's ignore-by-default behavior. §5 precedence ladder → Task 3 step 5 (Python), Task 4 step 6 (C#), Task 5 step 6 (Java), documented in Task 7. §6 writer → Task 6. §7 corpus → Task 1, with the CI-latency caveat carried into the README. §8 deferrals → Task 7's changelog Notes. + +**Known plan limitations, stated rather than hidden.** Three tasks carry an explicit *implementer note* where the exact local API could not be verified without opening files this plan does not otherwise touch: Java's JSON facility and `MetaDataException` constructor (Task 5 steps 1, 3), C#'s exception type name (Task 4 step 2), and the CLI test helper names in `init.test.ts` (Task 6 step 1). Each note names the exact `grep` that resolves it. These are look-ups, not design gaps. + +**Scope boundary worth flagging to a reviewer.** Tasks 4 and 5 wire the *resolver* and the *corpus*, and the C# CLI hands the loader a resolved root rather than a resolved file SET, because `MetaDataLoader.FromDirectory` takes one directory. Every port's set-accepting loader entry already exists (spec §2 table); widening the CLI call sites to use it is a deliberate follow-up, not part of this changeset. What ships here is that all four ports **read the same declaration and resolve the same files** — which is the stated requirement. diff --git a/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md b/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md new file mode 100644 index 000000000..7d7c13c08 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-migrate-chain-replayability.md @@ -0,0 +1,1800 @@ +# Migration Chain Replayability Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make a committed migration chain provably replayable from an empty database, and stop `meta migrate` writing statements that cannot replay. + +**Architecture:** Three independent layers. The emitter stops writing landmines (`IF EXISTS` on forward drops; `CREATE SCHEMA IF NOT EXISTS` ahead of a non-default schema). A new `meta verify --replay` gate replays the committed chain into an **in-process** database (PGlite for postgres, `:memory:` libsql for sqlite) and asserts it applies; `--replay-snapshot` additionally asserts it reproduces the committed snapshot via the already-built-but-unwired `verifyReplay`. An emit-time provenance guard refuses to drop an object the committed snapshot never contained, so the bad SQL is never authored. + +**Tech Stack:** TypeScript, Bun test runner, Kysely, `@electric-sql/pglite` (new, optional peer), `@libsql/kysely-libsql` (already a hard dependency of `cli`, a devDependency of `migrate-ts`). + +**Spec:** [`docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md`](../specs/2026-08-19-migrate-chain-replayability-design.md) + +## Global Constraints + +- **Scope is TypeScript only.** Schema migration is TS-owned ([ADR-0015](../../../spec/decisions/ADR-0015-single-shared-migrate-engine.md)). No Java/Kotlin/C#/Python work, no conformance-corpus fan-out. +- **`IF EXISTS` goes on FORWARD drops only.** Down statements stay bare. `rollbackTo` runs `down.sql` and the ledger delete in ONE transaction (`apply/apply.ts:185-189`), so a down that no-ops would still record the rollback as done. +- **Never `instanceof` a metadata node from another package** — use the exported guards (`isMetaObject`, `isWritableSource`, …). Two physical copies of `@metaobjectsdev/metadata` in one process make `instanceof` silently false. +- **No `any`.** Use `unknown` and narrow. A bare `let x;` is an implicit evolving `any`. **No `as never` / `as unknown as T` in committed test fixtures** — every literal in this plan is written against the real type; if one does not compile, the type is the authority, not the cast. +- **Never call `own*()` accessors** (ADR-0039) except where surrounding code documents a sanctioned reason. +- **Errors are `ParseError` with a structured `code`** and `codeSource(...)` — never a message prefix. +- **Backward compatibility is absolute:** a project that declares no new flag must emit byte-identical migrations except for the `IF EXISTS` and `CREATE SCHEMA` tokens this plan adds, and `SNAPSHOT_FORMAT_VERSION` must remain 3. +- **Every new gate FAILS OPEN on a missing input.** A project that has never generated a snapshot, or has no committed migrations, is not in an error state — but it must SAY so, never pass silently. A gate that is quiet when it checked nothing is indistinguishable from a gate that passed. +- **Run tests scoped:** `cd server/typescript && bun test packages/`. NEVER a bare `bun test` at the repository root — it walks java/python/csharp and takes many minutes. +- **`bun test` does NOT typecheck.** Run `bun run --filter '*' typecheck` from the repository root before every commit and confirm all 18 packages exit 0. +- **Every task ends GREEN.** No task may leave `bun test packages/migrate-ts` or `packages/cli` red for a later task to inherit. Test churn caused by a task is fixed inside that task. +- **Public repository.** No private project names, no absolute home paths, in code, tests, fixtures, or commit messages. The committed `.githooks/pre-commit` enforces this (`git config core.hooksPath .githooks`) using the denylist at `git config hooks.denyListPath`. +- **Stage explicit paths only.** Never `git add -A` — other worktrees share this repository. + +--- + +## Task ordering (a DAG, not a line) + +``` +Task 1 (emit: IF EXISTS) ─┐ +Task 2 (emit: CREATE SCHEMA) ─┤ +Task 7 (provenance guard) ─┼─→ Task 8 (docs + CHANGELOG) +Task 3 (replay engine) ─┬─ Task 4 (verifyReplay governed) ─┐ + └─ Task 5 (--replay) ──────────────┴─→ Task 6 (--replay-snapshot) +``` + +- **Tasks 1, 2, 3 and 7 are independent** of each other and of everything else, and may run in any order or in parallel. +- **Task 4 needs Task 3** — its test imports `openReplayEngine`. +- **Task 5 needs Task 3** — the gate calls `openReplayEngine`. +- **Task 6 needs Tasks 3, 4 AND 5** — it extends `runReplayVerify` (Task 5) and passes `governed` (Task 4). +- **Task 8 needs 1, 2, 5, 6, 7** — it documents what they ship. + +--- + +## File Structure + +**Modified — `server/typescript/packages/migrate-ts/src/`** +- `emit/postgres.ts` — `IF EXISTS` on forward drops; `CREATE SCHEMA IF NOT EXISTS` emission +- `emit/sqlite.ts` — `IF EXISTS` on forward drops (two lines only; see Task 1) +- `verify/replay.ts` — thread scope inputs into the snapshot comparison +- `types.ts` — `AllowOptions.dropUnmanaged` +- `index.ts` — export the new replay-engine surface +- `package.json` — `@electric-sql/pglite` as an optional peer + devDependency + +**New — `server/typescript/packages/migrate-ts/src/`** +- `verify/replay-engine.ts` — provision an in-process database (PGlite / `:memory:` libsql), hand back a Kysely instance and a disposer. One responsibility: engine lifecycle. No replay logic, no comparison. + +**Modified — `server/typescript/packages/cli/src/`** +- `lib/args.ts` — `--replay` / `--replay-snapshot` verify flags; `drop-unmanaged` allow token +- `lib/allow.ts` — `drop-unmanaged` → `dropUnmanaged` grant mapping +- `commands/verify.ts` — the replay gate +- `commands/migrate.ts` — the emit-time provenance guard +- `package.json` — `--external @electric-sql/pglite` on `build:binary` + +**Modified — `server/typescript/packages/sdk/src/`** +- `config.ts` — `drop-unmanaged` in `AllowTokenEnum` + +**Modified — existing tests (churn, fixed inside the task that causes it)** +- `packages/migrate-ts/test/unit/emit-postgres.test.ts` (Task 1) +- `packages/migrate-ts/test/unit/emit-sqlite.test.ts` (Task 1) +- `packages/migrate-ts/test/check/emit-postgres-check.test.ts` (Task 1) +- `packages/migrate-ts/test/check-evolution/drop-check-down.test.ts` (Task 1) +- `packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts` (Task 1 — **both** lines 142 and 143) + +**New — tests** +- `packages/migrate-ts/test/emit-drop-if-exists.test.ts` +- `packages/migrate-ts/test/emit-postgres-create-schema.test.ts` +- `packages/migrate-ts/test/unit/replay-engine.test.ts` +- `packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts` — the #313 regression, emitter→apply +- `packages/migrate-ts/test/integrity/replay-scoped.test.ts` +- `packages/cli/test/verify-replay.test.ts` +- `packages/cli/test/migrate-drop-unmanaged.test.ts` + +--- + +## Task 1: `IF EXISTS` on forward drops, both dialects + +**Files:** +- Modify: `server/typescript/packages/migrate-ts/src/emit/postgres.ts` (`renderUp`, `renderDropView`) +- Modify: `server/typescript/packages/migrate-ts/src/emit/sqlite.ts` (`renderUpNative`) +- Test: `server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts` (create) +- Fix churn: `test/unit/emit-postgres.test.ts`, `test/unit/emit-sqlite.test.ts`, `test/check/emit-postgres-check.test.ts`, `test/check-evolution/drop-check-down.test.ts`, `test/integration/pg-constraint-backed-index-285.test.ts` + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: no new exports. Behaviour change only. + +**Context the implementer needs.** `renderUp` (postgres) / `renderUpNative` (sqlite) are the FORWARD +renderers. `renderDown` / `renderDownNative` are out of scope — see the Global Constraint. + +The exact sites, all verified bare at HEAD: + +| File:line | Change kind | Action | +|---|---|---| +| `postgres.ts:66` | `drop-table` | add `IF EXISTS` | +| `postgres.ts:95` | `drop-index`, constraint-backed arm | `DROP CONSTRAINT IF EXISTS` | +| `postgres.ts:96` | `drop-index`, plain arm | `DROP INDEX IF EXISTS` | +| `postgres.ts:98` | `drop-fk` | `DROP CONSTRAINT IF EXISTS` | +| `postgres.ts:104` | `drop-check` | `DROP CONSTRAINT IF EXISTS` | +| `postgres.ts:375` | `renderDropView`, plain | `DROP VIEW IF EXISTS` | +| `postgres.ts:388` | `renderDropView`, CASCADE | `DROP VIEW IF EXISTS … CASCADE` | +| `sqlite.ts:219` | `drop-table` | add `IF EXISTS` | +| `sqlite.ts:225` | `drop-index` | add `IF EXISTS` | + +**SQLite is two lines, not more.** `sqlite.ts:240` (`drop-view`) and `:241` (`replace-view`) already +emit `DROP VIEW IF EXISTS` on the forward side. The sqlite `drop-fk`/`drop-check`/`add-check` arms +**throw** (`sqlite.ts:226-235`) — those change kinds are folded into a table recreate, so there is no +standalone statement to guard, which is why the Postgres-only guard on `drop-fk`/`drop-check` is not +a dialect split (spec §3.1). + +**Deliberately left bare — do not "finish the job":** +- `sqlite.ts:197` — the recreate-and-copy rebuild's `DROP TABLE`. It drops a table the same recipe + just `INSERT…SELECT`ed from; `IF EXISTS` there converts a caught corruption into a silent one. +- `emit/d1-cascade.ts:126` — same reason. +- `postgres.ts:431` — inside `renderRestoreView`, reached only from `postgres.ts:178`/`:179`, both in + `renderDown`. Guarding it violates the forward-only rule in the same change that states it. +- Every down site: `postgres.ts:113`, `:147`, `:164`, `:171`, `:176`; `sqlite.ts:256`, `:262`, `:275`. + +**`drop-check` IS reachable — the code comment saying otherwise is wrong.** `emit/postgres.ts:99-102` +claims `add-check`/`drop-check` are "declared but NOT yet produced by the diff". That comment is +false: `diff/index.ts:579` and `:592` both push `drop-check`, an evolved `field.enum @values` is a +live producer, and `test/check/emit-postgres-check.test.ts:25` plus +`test/check-evolution/drop-check-down.test.ts:12` already assert the emitted forward statement. +**Delete the false half of that comment as part of this task** and assert `drop-check` in the new +test like every other forward drop. + +`emit/d1.ts:21` renders through `renderSqlite`, so the sqlite edits also change D1's committed +migrations. That is accepted and expected (spec §3.1). + +- [ ] **Step 1: Write the failing test** + +```ts +// server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts +// +// Forward drops must tolerate an absent object so a committed chain replays into +// an empty database (#313). Down statements must NOT — `rollbackTo` runs down.sql +// and the ledger delete in one transaction, so a silently-no-op down would record +// the rollback as done. +import { describe, test, expect } from "bun:test"; +import { renderPostgres } from "../src/emit/postgres.js"; +import { renderSqlite } from "../src/emit/sqlite.js"; +import type { Change, ChangeStatus, TableDescriptor } from "../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +const GONE: TableDescriptor = { + name: "gone", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], +}; + +describe("forward drops tolerate an absent object (#313)", () => { + test("postgres drop-table", () => { + const { up } = renderPostgres([{ kind: "drop-table", table: "gone", status: ALLOWED }]); + expect(up).toContain('DROP TABLE IF EXISTS "gone";'); + }); + + test("postgres drop-view", () => { + const { up } = renderPostgres([{ kind: "drop-view", view: "v_gone", status: ALLOWED }]); + expect(up).toContain('DROP VIEW IF EXISTS "v_gone";'); + }); + + test("postgres drop-index, plain", () => { + const { up } = renderPostgres([{ kind: "drop-index", table: "t", index: "idx_gone", status: ALLOWED }]); + expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); + }); + + test("postgres drop-index, constraint-backed (#285)", () => { + const { up } = renderPostgres([{ + kind: "drop-index", table: "t", index: "uq_gone", status: ALLOWED, + restore: { name: "uq_gone", columns: ["a"], unique: true, constraint: "unique" }, + }]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "uq_gone";'); + }); + + test("postgres drop-fk", () => { + const { up } = renderPostgres([{ kind: "drop-fk", table: "t", fk: "fk_gone", status: ALLOWED }]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "fk_gone";'); + }); + + // drop-check IS produced by the diff (diff/index.ts:579, :592) — an evolved + // `field.enum @values` is a live producer. The `renderUp` comment claiming + // otherwise is deleted by this task. + test("postgres drop-check", () => { + const { up } = renderPostgres([{ kind: "drop-check", table: "t", check: "t_qty_chk", status: ALLOWED }]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "t_qty_chk";'); + }); + + test("sqlite drop-table", () => { + const { up } = renderSqlite([{ kind: "drop-table", table: "gone", status: ALLOWED }]); + expect(up).toContain('DROP TABLE IF EXISTS "gone";'); + }); + + test("sqlite drop-index", () => { + const { up } = renderSqlite([{ kind: "drop-index", table: "t", index: "idx_gone", status: ALLOWED }]); + expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); + }); + + // Already guarded before this task; pinned so a later sweep cannot un-guard it. + test("sqlite drop-view was already guarded", () => { + const { up } = renderSqlite([{ kind: "drop-view", view: "v_gone", status: ALLOWED }]); + expect(up).toContain('DROP VIEW IF EXISTS "v_gone";'); + }); +}); + +describe("down statements stay bare — a rollback must fail loudly", () => { + test("postgres create-table down", () => { + const { down } = renderPostgres([{ kind: "create-table", table: GONE, status: ALLOWED }]); + expect(down).toContain('DROP TABLE "gone";'); + expect(down).not.toContain("DROP TABLE IF EXISTS"); + }); + + test("postgres create-view down", () => { + const { down } = renderPostgres([{ + kind: "create-view", status: ALLOWED, + view: { name: "v", sql: "SELECT 1 AS one", columns: ["one"] }, + }]); + expect(down).toContain('DROP VIEW "v";'); + expect(down).not.toContain("DROP VIEW IF EXISTS"); + }); + + test("sqlite create-table down", () => { + const { down } = renderSqlite([{ kind: "create-table", table: GONE, status: ALLOWED }]); + expect(down).toContain('DROP TABLE "gone";'); + expect(down).not.toContain("DROP TABLE IF EXISTS"); + }); +}); + +describe("the recreate-and-copy rebuild drop stays bare — deliberately", () => { + test("sqlite recreate emits a bare DROP TABLE for the table it just copied from", () => { + const expectedSchema = { + tables: [{ + name: "orders", + columns: [ + { name: "id", sqlType: { kind: "integer" as const, bits: 64 as const }, nullable: false }, + { name: "amount", sqlType: { kind: "integer" as const, bits: 64 as const }, nullable: false }, + ], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], + }], + views: [], + }; + const { up } = renderSqlite( + [{ + kind: "change-column-type", table: "orders", column: "amount", + from: { kind: "real" }, to: { kind: "integer", bits: 64 }, status: ALLOWED, + }], + expectedSchema, + ); + // IF EXISTS here would turn a caught corruption into a silent one: the recipe + // just INSERT…SELECTed out of this exact table. + expect(up).toContain('DROP TABLE "orders";'); + expect(up).not.toContain('DROP TABLE IF EXISTS "orders";'); + }); +}); +``` + +**If a literal above does not compile**, read the real shape in `src/types.ts` (`Change`, +`ChangeStatus`, `TableDescriptor`, `IndexDescriptor`, `ViewDescriptor`) and correct the literal. +Do not add a cast. `renderSqlite(changes, expectedSchema?, actualMeta?)` and +`renderPostgres(changes)` are the real signatures. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-drop-if-exists.test.ts` +Expected: the six postgres + two sqlite "forward drops" tests FAIL. The "sqlite drop-view was +already guarded", the three "down statements stay bare" and the rebuild test PASS already — they pin +behaviour this task must not change. + +- [ ] **Step 3: Implement — postgres forward drops** + +In `emit/postgres.ts`, inside `renderUp`: + +```ts + case "drop-table": return `DROP TABLE IF EXISTS ${quoteQualified(c.table, c.schema)};`; +``` + +```ts + case "drop-index": + return c.restore?.constraint !== undefined + ? `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.index)};` + : `DROP INDEX IF EXISTS ${quoteIndexQualified(c.index, c.schema)};`; +``` + +```ts + case "drop-fk": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.fk)};`; +``` + +```ts + case "drop-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.check)};`; +``` + +Replace the false comment above the check arms (`postgres.ts:99-102`) with the truth: + +```ts + // `drop-check` IS produced by the diff — diff/index.ts:579 and :592 push it, + // and an evolved `field.enum @values` is a live producer. (An earlier comment + // here claimed these arms were unreachable; they are not.) `add-check` is the + // paired ADD and rides the same passes. +``` + +In `renderDropView` (`:375` and `:388`), change both forms: + +```ts + if (dependents.length === 0) return `DROP VIEW IF EXISTS ${qualified};`; +``` + +```ts + `DROP VIEW IF EXISTS ${qualified} CASCADE;`, +``` + +**Do NOT touch `renderRestoreView` (`:431`).** + +- [ ] **Step 4: Implement — sqlite forward drops** + +In `emit/sqlite.ts`, inside `renderUpNative`, exactly two lines: + +```ts + case "drop-table": return `DROP TABLE IF EXISTS ${quote(c.table)};`; +``` + +```ts + case "drop-index": return `DROP INDEX IF EXISTS ${quote(c.index)};`; +``` + +Leave `sqlite.ts:197` and `renderDownNative` untouched. + +- [ ] **Step 5: Run the new test to verify it passes** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-drop-if-exists.test.ts` +Expected: PASS, all thirteen. + +- [ ] **Step 6: Fix the expected churn — five named files** + +These assertions pin the exact bare statement and will go RED. Each is a FORWARD (`up`) assertion; +update it to the `IF EXISTS` form. **Read each one first**: if an assertion is on a `down`, the +correct fix is to leave it alone and check you did not edit a down renderer. + +| File:line | Current | Becomes | +|---|---|---| +| `test/unit/emit-postgres.test.ts:136` | `DROP TABLE "legacy";` | `DROP TABLE IF EXISTS "legacy";` | +| `test/unit/emit-postgres.test.ts:164` | `DROP INDEX "old_idx";` | `DROP INDEX IF EXISTS "old_idx";` | +| `test/unit/emit-postgres.test.ts:187` | `ALTER TABLE "weeks" DROP CONSTRAINT "weeks_program_id_fk";` | `… DROP CONSTRAINT IF EXISTS "weeks_program_id_fk";` | +| `test/unit/emit-sqlite.test.ts:61` | `DROP TABLE "old";` | `DROP TABLE IF EXISTS "old";` | +| `test/unit/emit-sqlite.test.ts:77` | `DROP INDEX "i";` | `DROP INDEX IF EXISTS "i";` | +| `test/check/emit-postgres-check.test.ts:25` | `r.up` ⊃ `ALTER TABLE "orders" DROP CONSTRAINT "orders_status_chk";` | `… DROP CONSTRAINT IF EXISTS …` | +| `test/check-evolution/drop-check-down.test.ts:12` | `r.up` ⊃ `ALTER TABLE "orders" DROP CONSTRAINT "orders_qty_numeric_chk";` | `… DROP CONSTRAINT IF EXISTS …` | +| `test/integration/pg-constraint-backed-index-285.test.ts:143` | `/ALTER TABLE .*DROP CONSTRAINT "work_item_message_id_unique"/` | `/ALTER TABLE .*DROP CONSTRAINT IF EXISTS "work_item_message_id_unique"/` | + +**Do NOT change these — they must stay green, and their staying green is the evidence:** +`test/unit/emit-postgres.test.ts:318` and `test/unit/emit-sqlite.test.ts:237` (down assertions), +`test/check/emit-postgres-check.test.ts:21` (down), `test/unit/emit-sqlite.test.ts:128` (the +recreate-and-copy rebuild drop), `test/write-migration*.test.ts` (hand-written SQL, not emitter +output), `test/unit/emit-views.test.ts:20` and `test/unit/diff.test.ts:264` (`/DROP VIEW/i`, which +still matches). + +- [ ] **Step 7: Re-anchor the one negative assertion that would go VACUOUSLY green** + +`test/integration/pg-constraint-backed-index-285.test.ts:142` is a *negative* assertion: + +```ts + expect(sqlText).not.toMatch(/DROP INDEX "?work_item_message_id_unique/); +``` + +Adding `IF EXISTS` puts ` IF EXISTS ` between `DROP INDEX` and the quote, so the regex stops matching +for a reason that has nothing to do with #285 — it would keep passing even if #285 fully regressed. +Re-anchor it so it still tests what it was written to test: + +```ts + // Anchored to tolerate the #313 `IF EXISTS` token: without `(IF EXISTS )?` this + // negative assertion passes because the SPELLING changed, not because the + // constraint-backed index is correctly dropped via ALTER TABLE. + expect(sqlText).not.toMatch(/DROP INDEX (IF EXISTS )?"?work_item_message_id_unique/); +``` + +Then **prove the re-anchoring works**: temporarily revert the `drop-index` constraint-backed arm to +`DROP INDEX IF EXISTS …`, confirm line 142 goes RED, and restore. A negative assertion you have not +seen fail is not a test. + +- [ ] **Step 8: Run the affected suites** + +Run: `cd server/typescript && bun test packages/migrate-ts && bun test packages/cli` +Expected: PASS, both. If anything is still red, it is churn Step 6 missed — fix it here, not later. + +- [ ] **Step 9: Typecheck** + +Run: `bun run --filter '*' typecheck` (from the repository root) +Expected: all 18 packages exit 0. + +- [ ] **Step 10: Commit** + +```bash +git add server/typescript/packages/migrate-ts/src/emit/postgres.ts \ + server/typescript/packages/migrate-ts/src/emit/sqlite.ts \ + server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts \ + server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts \ + server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts \ + server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts \ + server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts \ + server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts +git commit -m "fix(migrate): forward drops tolerate an absent object so a chain can replay" +``` + +--- + +## Task 2: `CREATE SCHEMA IF NOT EXISTS` for non-default schemas + +**Files:** +- Modify: `server/typescript/packages/migrate-ts/src/emit/postgres.ts` (`renderPostgres`) +- Test: `server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from Task 1. +- Produces: no new exports. + +**Context:** `CREATE SCHEMA` is emitted nowhere in `migrate-ts/src` or `cli/src` today except the +ledger's own (`apply/ledger.ts:126`). A chain containing `CREATE TABLE "reporting"."x"` therefore +cannot apply to a virgin database. SQLite has no schema namespacing +(`emit-sqlite-schema-rejected.test.ts` pins that a schema is rejected there), so this is +Postgres-only and is NOT a dialect split. + +**Collect from `create-view` too, not only `create-table`.** Spec §3.3 says "ahead of the first +**object** in a non-default schema", and a first migration that creates only a *view* in +`"reporting"` fails identically. `create-view` carries the schema in **two** places — the change's +own `schema?` and `view.schema` — so read `c.schema ?? c.view.schema`, the same precedence +`renderCreateView(c.view, c.schema, …)` already uses. + +- [ ] **Step 1: Write the failing test** + +```ts +// server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts +import { describe, test, expect } from "bun:test"; +import { renderPostgres } from "../src/emit/postgres.js"; +import type { ChangeStatus, TableDescriptor, ViewDescriptor } from "../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +const t = (name: string, schema?: string): TableDescriptor => ({ + name, + ...(schema !== undefined ? { schema } : {}), + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], +}); + +const v = (name: string, schema?: string): ViewDescriptor => ({ + name, + ...(schema !== undefined ? { schema } : {}), + sql: "SELECT 1 AS one", + columns: ["one"], +}); + +describe("a chain that creates an object in a non-default schema creates the schema first", () => { + test("emits CREATE SCHEMA IF NOT EXISTS before the table", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "reporting"), status: ALLOWED }]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')) + .toBeLessThan(up.indexOf("CREATE TABLE")); + }); + + // Spec §3.3 says "the first OBJECT", not "the first table". A chain whose first + // migration creates only a view in a non-default schema fails identically. + test("emits it for a create-view too", () => { + const { up } = renderPostgres([{ kind: "create-view", view: v("v_x", "reporting"), status: ALLOWED }]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')) + .toBeLessThan(up.indexOf("CREATE VIEW")); + }); + + test("emits it once for two objects in the same schema", () => { + const { up } = renderPostgres([ + { kind: "create-table", table: t("x", "reporting"), status: ALLOWED }, + { kind: "create-table", table: t("y", "reporting"), status: ALLOWED }, + { kind: "create-view", view: v("v_x", "reporting"), status: ALLOWED }, + ]); + expect(up.match(/CREATE SCHEMA IF NOT EXISTS "reporting";/g)).toHaveLength(1); + }); + + test("emits one per distinct schema, in sorted order", () => { + const { up } = renderPostgres([ + { kind: "create-table", table: t("x", "zeta"), status: ALLOWED }, + { kind: "create-table", table: t("y", "alpha"), status: ALLOWED }, + ]); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "alpha";')) + .toBeLessThan(up.indexOf('CREATE SCHEMA IF NOT EXISTS "zeta";')); + }); + + test("emits nothing for the default schema", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x"), status: ALLOWED }]); + expect(up).not.toContain("CREATE SCHEMA"); + }); + + test("emits nothing for an explicit 'public'", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "public"), status: ALLOWED }]); + expect(up).not.toContain("CREATE SCHEMA"); + }); + + test("the down does NOT drop the schema", () => { + const { down } = renderPostgres([{ kind: "create-table", table: t("x", "reporting"), status: ALLOWED }]); + expect(down).not.toContain("DROP SCHEMA"); + }); +}); +``` + +The last case is a real decision, not filler: dropping a schema on rollback would destroy objects +this tool does not own and cannot restore. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-postgres-create-schema.test.ts` +Expected: FAIL — the first four cases find no `CREATE SCHEMA`. The last three PASS already. + +- [ ] **Step 3: Implement** + +In `renderPostgres`, after the changes are sorted and before the returned `up` is joined: + +```ts + // A chain must be appliable to a VIRGIN database (#313). `CREATE TABLE "s"."x"` + // fails there unless the schema exists, and no migration has ever created one. + // IF NOT EXISTS because a later migration in the same chain, or an operator, may + // have created it already. Views count as objects too, so a first migration that + // creates only a view in a non-default schema is covered. Deliberately NOT dropped + // in `down`: the schema may hold objects this tool does not own and cannot restore. + const createdSchemas = new Set(); + for (const c of sorted) { + const s = + c.kind === "create-table" ? c.table.schema + : c.kind === "create-view" ? (c.schema ?? c.view.schema) + : undefined; + if (s !== undefined && s !== DEFAULT_DB_SCHEMA_POSTGRES) createdSchemas.add(s); + } + // Sorted so output is deterministic — the snapshot and golden tests rely on it. + const schemaStmts = [...createdSchemas].sort().map((s) => `CREATE SCHEMA IF NOT EXISTS ${quote(s)};`); +``` + +then prepend to the returned `up`: + +```ts + up: [...schemaStmts, ...upStmts].join("\n\n"), +``` + +`DEFAULT_DB_SCHEMA_POSTGRES` is exported from `@metaobjectsdev/metadata` — `qualified-name.ts` and +`diff/index.ts:17` both already import it from there. If `postgres.ts` already has a local notion of +the default schema, use that rather than adding a second one. + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/emit-postgres-create-schema.test.ts` +Expected: PASS, all seven. + +- [ ] **Step 5: Run the affected suites** + +Run: `cd server/typescript && bun test packages/migrate-ts && bun test packages/cli` +Expected: PASS. `emit-postgres-schema-namespacing.test.ts` is the file most likely to need updating; +read its assertions before changing them. Any churn is fixed here, in this task. + +- [ ] **Step 6: Typecheck and commit** + +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/migrate-ts/src/emit/postgres.ts \ + server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts +git commit -m "fix(migrate): a chain creates the schema it needs, so it applies to a virgin database" +``` + +--- + +## Task 3: The in-process replay engine + +**Files:** +- Create: `server/typescript/packages/migrate-ts/src/verify/replay-engine.ts` +- Modify: `server/typescript/packages/migrate-ts/src/index.ts` (export it) +- Modify: `server/typescript/packages/migrate-ts/package.json` (optional peer + devDependency) +- Modify: `server/typescript/packages/cli/package.json` (`build:binary` external) +- Test: `server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: + ```ts + export interface ReplayEngine { + db: Kysely>; + dispose: () => Promise; + } + export function openReplayEngine(dialect: "postgres" | "sqlite"): Promise; + ``` + Tasks 4, 5 and 6 call `openReplayEngine` and must `await engine.dispose()` in a `finally`. + +### Two decisions this task settles, both already verified + +**Where it lives, and how the dependency is declared.** `openReplayEngine` goes in **`migrate-ts`**, +not `cli`: `migrate-ts`'s own tests (Tasks 4 and 5) need it, and `cli` depends on `migrate-ts`, so +putting it in `cli` would be a cycle. But `migrate-ts` today declares only `@iarna/toml` and +`@metaobjectsdev/metadata` as dependencies — `kysely` is a peer and every driver is dev-only, +deliberately. **PGlite is 22 MB of WASM**; a hard dependency would make every CLI adopter download it +to run `meta gen`. So: + +- `@electric-sql/pglite` → **optional peerDependency** `">=0.3.0 <0.6.0"` + **devDependency** + `"^0.5.0"`. Both 0.3.16 and 0.5.5 were verified against the adapter below. +- `@libsql/kysely-libsql` → **optional peerDependency** `">=0.4.0 <0.5.0"` + keep the existing + devDependency. It is already a hard `dependencies` entry of `cli` (`^0.4.0`), so the CLI path + always resolves it and only a direct `migrate-ts` embedder can miss it. +- Both are imported **lazily** inside the opener, with an install hint on failure, mirroring + `cli/src/lib/kysely.ts`'s `buildKyselyFromUrl`. +- Peer ranges must be **bounded** or `scripts/check-peer-ranges.ts` fails the `gates` lane (its test + is: a range is unbounded exactly when it accepts `9999.0.0`). Set + `peerDependenciesMeta..optional = true` for both. +- `cli`'s `build:binary` bundles the CLI (`bun build ./bin/meta.ts --compile`). Add + `--external @electric-sql/pglite` beside the two existing `--external @biomejs/*` flags, so the + standalone binary does not embed 22 MB of WASM and instead resolves it from the adopter's project + at run time. + +**PGlite is NOT `pg`-compatible — a shim is required.** PGlite exposes `query`/`exec`/`close`; +kysely's `PostgresDialect` wants a `pg.Pool` (`connect()` → a client with `query()`/`release()`, plus +`end()`). The ~20-line shim below was **executed against PGlite 0.3.16 and 0.5.5** and verified to +handle: `CREATE SCHEMA`, a CHECK constraint, `information_schema` reads, kysely transactions, +transaction *rollback*, `pg_advisory_lock`/`pg_advisory_unlock` (which `applyPending` takes on +postgres), and — the one that matters — rejecting `DROP TABLE "theirs"` with exactly +`table "theirs" does not exist`, the reporter's error. + +Also verified: `LibsqlDialect({ url: ":memory:" })` works, and two `:memory:` instances are fully +isolated from each other. + +- [ ] **Step 1: Declare the dependencies** + +In `server/typescript/packages/migrate-ts/package.json`: + +```jsonc + "peerDependencies": { + "kysely": ">=0.27.0 <0.30.0", + "@electric-sql/pglite": ">=0.3.0 <0.6.0", + "@libsql/kysely-libsql": ">=0.4.0 <0.5.0" + }, + "peerDependenciesMeta": { + "@electric-sql/pglite": { "optional": true }, + "@libsql/kysely-libsql": { "optional": true } + }, +``` + +and add `"@electric-sql/pglite": "^0.5.0"` to `devDependencies` (`@libsql/kysely-libsql` is already +there). Then from the repository root: `bun install`. + +In `server/typescript/packages/cli/package.json`, extend `build:binary`: + +``` +bun build ./bin/meta.ts --compile --outfile dist/meta --external @biomejs/wasm-bundler --external @biomejs/wasm-web --external @electric-sql/pglite +``` + +Run `bun run scripts/check-peer-ranges.ts` (or the `gates` lane's equivalent) and confirm it passes. + +- [ ] **Step 2: Write the failing test** + +```ts +// server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts +import { describe, test, expect } from "bun:test"; +import { sql } from "kysely"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; + +describe("openReplayEngine", () => { + test("sqlite: gives an empty, usable database", async () => { + const engine = await openReplayEngine("sqlite"); + try { + await sql`CREATE TABLE t (id integer primary key)`.execute(engine.db); + await sql`INSERT INTO t (id) VALUES (1)`.execute(engine.db); + const rows = await sql<{ id: number }>`SELECT id FROM t`.execute(engine.db); + expect(rows.rows).toHaveLength(1); + } finally { + await engine.dispose(); + } + }); + + test("postgres: gives an empty, usable database with real PG DDL", async () => { + const engine = await openReplayEngine("postgres"); + try { + // Schema namespacing + a CHECK — neither is expressible in sqlite, so this + // proves the postgres engine is really Postgres. + await sql`CREATE SCHEMA IF NOT EXISTS "reporting"`.execute(engine.db); + await sql`CREATE TABLE "reporting"."t" (id integer primary key, n integer CHECK (n > 0))`.execute(engine.db); + const rows = await sql<{ table_name: string }>` + SELECT table_name FROM information_schema.tables WHERE table_schema = 'reporting' + `.execute(engine.db); + expect(rows.rows.map((r) => r.table_name)).toContain("t"); + } finally { + await engine.dispose(); + } + }); + + // applyPending runs each migration file inside a kysely transaction and takes a + // pg advisory lock on postgres. Both must work through the shim, or the gate + // fails for a reason that has nothing to do with the chain under test. + test("postgres: transactions roll back, and advisory locks work", async () => { + const engine = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE t (id integer primary key)`.execute(engine.db); + await expect( + engine.db.transaction().execute(async (trx) => { + await sql`INSERT INTO t (id) VALUES (1)`.execute(trx); + throw new Error("boom"); + }), + ).rejects.toThrow(/boom/); + const after = await sql<{ c: string }>`SELECT count(*)::text AS c FROM t`.execute(engine.db); + expect(after.rows[0]?.c).toBe("0"); + + await sql`SELECT pg_advisory_lock(hashtext('meta'))`.execute(engine.db); + await sql`SELECT pg_advisory_unlock(hashtext('meta'))`.execute(engine.db); + } finally { + await engine.dispose(); + } + }); + + // The whole gate rests on this: a statement against a missing object must REJECT. + test("postgres: dropping a missing table rejects — the #313 signal", async () => { + const engine = await openReplayEngine("postgres"); + try { + await expect(sql`DROP TABLE "theirs"`.execute(engine.db)).rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + } + }); + + test("sqlite: dropping a missing table rejects — the #313 signal", async () => { + const engine = await openReplayEngine("sqlite"); + try { + await expect(sql`DROP TABLE "theirs"`.execute(engine.db)).rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + } + }); + + test("two engines of the same dialect do not share state", async () => { + const a = await openReplayEngine("sqlite"); + const b = await openReplayEngine("sqlite"); + try { + await sql`CREATE TABLE only_in_a (id integer)`.execute(a.db); + const rows = await sql<{ name: string }>`SELECT name FROM sqlite_master WHERE name = 'only_in_a'`.execute(b.db); + expect(rows.rows).toHaveLength(0); + } finally { + await a.dispose(); + await b.dispose(); + } + }); + + test("dispose is idempotent", async () => { + const engine = await openReplayEngine("sqlite"); + await engine.dispose(); + await engine.dispose(); + }); +}); +``` + +- [ ] **Step 3: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/unit/replay-engine.test.ts` +Expected: FAIL — `openReplayEngine` is not defined. + +- [ ] **Step 4: Implement** + +```ts +// server/typescript/packages/migrate-ts/src/verify/replay-engine.ts +// +// An empty, throwaway database that lives INSIDE this process. +// +// The replay gate has to apply a whole committed chain from nothing. Doing that +// against the user's server would mean CREATE DATABASE — which needs CREATEDB, +// breaks behind a connection pooler, is restricted on managed Postgres, collides +// between parallel CI jobs sharing one server, and puts a DROP DATABASE next to a +// name derived from a real one (Postgres truncates identifiers at 63 bytes, so a +// long enough target derives a scratch name that truncates back ONTO the target). +// None of that is worth it when the engines run in-process: PGlite is real Postgres +// compiled to WASM, and libsql runs sqlite in memory. Nothing to provision, nothing +// to clean up, nothing to drop by mistake. +// +// Both drivers are OPTIONAL peers imported lazily: PGlite is 22 MB of WASM and must +// not land in every `meta gen` adopter's node_modules. The install hints mirror +// `cli/src/lib/kysely.ts`'s. +import { Kysely } from "kysely"; + +export interface ReplayEngine { + /** An empty database. The caller owns applying migrations into it. */ + db: Kysely>; + /** Release the engine. Safe to call more than once. */ + dispose: () => Promise; +} + +export async function openReplayEngine( + dialect: "postgres" | "sqlite", +): Promise { + return dialect === "postgres" ? openPglite() : openMemorySqlite(); +} + +async function openMemorySqlite(): Promise { + type LibsqlDialectCtor = new (opts: { url: string }) => + ConstructorParameters>>[0]["dialect"]; + let LibsqlDialect: LibsqlDialectCtor; + try { + const mod = await import("@libsql/kysely-libsql"); + LibsqlDialect = mod.LibsqlDialect as unknown as LibsqlDialectCtor; + } catch { + throw new Error( + `the sqlite replay engine requires '@libsql/kysely-libsql'; install it to run 'meta verify --replay'`, + ); + } + const db = new Kysely>({ dialect: new LibsqlDialect({ url: ":memory:" }) }); + return disposable(db, async () => {}); +} + +async function openPglite(): Promise { + let PGliteCtor: new () => PgliteInstance; + try { + const mod = await import("@electric-sql/pglite"); + PGliteCtor = mod.PGlite as unknown as new () => PgliteInstance; + } catch { + throw new Error( + `the postgres replay engine requires '@electric-sql/pglite' (in-process WASM Postgres); ` + + `install it to run 'meta verify --replay' against a postgres chain`, + ); + } + const { PostgresDialect } = await import("kysely"); + const pg = new PGliteCtor(); + const db = new Kysely>({ + dialect: new PostgresDialect({ pool: pgliteAsPool(pg) as never }), + }); + return disposable(db, () => pg.close()); +} + +/** The slice of PGlite's surface this file uses. */ +interface PgliteInstance { + query(sql: string, params?: unknown[]): Promise<{ rows: unknown[]; affectedRows?: number; statement?: string }>; + close(): Promise; +} + +/** + * Kysely's PostgresDialect wants a `pg.Pool`: `connect()` returning a client with + * `query()`/`release()`, plus `end()`. PGlite offers `query`/`close` and is a SINGLE + * session, so every `connect()` hands back the same underlying instance — which is + * correct here because a replay is strictly sequential, and it is what makes a + * session advisory lock taken on one kysely connection visible to the next. + * + * `command` is only read by kysely to decide whether to report numAffectedRows; the + * replay path never reads it, so PGlite's `statement` (or a SELECT default) suffices. + */ +function pgliteAsPool(pg: PgliteInstance): unknown { + return { + async connect() { + return { + async query(sqlText: unknown, params?: readonly unknown[]) { + if (typeof sqlText !== "string") { + throw new Error("the PGlite replay engine does not support cursors"); + } + const r = await pg.query(sqlText, params ? [...params] : []); + return { + command: r.statement ?? "SELECT", + rowCount: r.affectedRows ?? r.rows.length, + rows: r.rows, + }; + }, + release() { /* single session — nothing to return to a pool */ }, + }; + }, + async end() { await pg.close(); }, + }; +} + +function disposable( + db: Kysely>, + closeEngine: () => Promise, +): ReplayEngine { + let disposed = false; + return { + db, + dispose: async () => { + if (disposed) return; + disposed = true; + try { await db.destroy(); } catch { /* the engine is throwaway */ } + try { await closeEngine(); } catch { /* idem */ } + }, + }; +} +``` + +- [ ] **Step 5: Export it** + +In `server/typescript/packages/migrate-ts/src/index.ts`, beside the existing +`export { verifyReplay } from "./verify/replay.js";`: + +```ts +export { openReplayEngine, type ReplayEngine } from "./verify/replay-engine.js"; +``` + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/unit/replay-engine.test.ts` +Expected: PASS, all seven. PGlite's first boot takes ~1s; that is expected. + +- [ ] **Step 7: Prove `introspect` works against PGlite** + +`--replay-snapshot` (Task 6) introspects the replayed database. `introspect/postgres.ts` reads +`information_schema` and `pg_catalog` and calls `pg_get_viewdef` — all real Postgres, so this should +just work, but Task 6 stalls if it does not. Add one case to the same file: + +```ts + test("postgres: introspect reads back a table and a view", async () => { + const engine = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE t (id integer primary key, n integer NOT NULL)`.execute(engine.db); + await sql`CREATE VIEW v AS SELECT id FROM t`.execute(engine.db); + const snap = await introspect(engine.db, "postgres"); + expect(snap.tables.map((x) => x.name)).toContain("t"); + expect(snap.views.map((x) => x.name)).toContain("v"); + } finally { + await engine.dispose(); + } + }); +``` + +(import `introspect` from `../../src/introspect/index.js`.) + +**If PGlite cannot execute the postgres cases**, STOP and report it. The spec's engine tiering rests +on PGlite being real Postgres; if it is not sufficient, that is a design question, not something to +work around by weakening the test. + +- [ ] **Step 8: Typecheck and commit** + +Run: `bun run --filter '*' typecheck` — all 18 exit 0. +Run: `cd server/typescript && bun test packages/migrate-ts` — PASS. + +```bash +git add server/typescript/packages/migrate-ts/src/verify/replay-engine.ts \ + server/typescript/packages/migrate-ts/src/index.ts \ + server/typescript/packages/migrate-ts/package.json \ + server/typescript/packages/cli/package.json \ + server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts \ + bun.lock +git commit -m "feat(migrate): an in-process replay engine, so the gate provisions nothing" +``` + +(The lockfile is at the repository root — confirm the path before staging.) + +--- + +## Task 4: Thread scope inputs into `verifyReplay` + +**Files:** +- Modify: `server/typescript/packages/migrate-ts/src/verify/replay.ts` +- Test: `server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts` (create) + +**Interfaces:** +- Consumes: `openReplayEngine` (Task 3). +- Produces: `VerifyReplayArgs` gains one optional field: + ```ts + /** Out-of-scope names to exclude from the SNAPSHOT side, as `scopeExpectedSchema` produces. */ + governed?: GovernedScope; + ``` + Task 6 passes it. + +**Context:** `verifyReplay` (`verify/replay.ts:31`) compares a replayed database against the +committed snapshot. A project declaring `migrate.scope` writes the *other* owner's tables into that +snapshot on purpose (`carryForwardOutOfScope`, `scope.ts:93`), and the chain never creates them — so +today the comparison reports them as missing. `excludeFromSnapshot` (`scope.ts:130`) exists for +exactly this and is already used by the committed-snapshot gate at `verify.ts:659`. + +**`excludeFromSnapshot` returns a `ScopedExpectedSchema`, not a `SchemaSnapshot`.** Its shape is +`{ snapshot, outOfScope, declaredSchemas? }` — take `.snapshot`. An empty `outOfScope` returns the +SAME snapshot object, so an unscoped caller's comparison is byte-for-byte what it was. + +**`GovernedScope`** (`scope.ts:149`) is `{ outOfScope: readonly string[]; declaredSchemas?: readonly string[] }`, +and names are **qualified**: `.` with an absent schema normalized to Postgres `public` +(`qualifiedDbName`, `qualified-name.ts:20`). SQLite objects normalize to the same `public.` prefix. +So the fixture below says `"public.theirs"`, not `"theirs"`. + +- [ ] **Step 1: Write the failing test** + +```ts +// server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts +import { describe, test, expect } from "bun:test"; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { verifyReplay } from "../../src/verify/replay.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import type { SchemaSnapshot } from "../../src/types.js"; + +function chainWith(upSql: string): string { + const dir = mkdtempSync(join(tmpdir(), "replay-scoped-")); + mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); + writeFileSync(join(dir, "20260101000000-init", "up.sql"), upSql, "utf8"); + writeFileSync(join(dir, "20260101000000-init", "down.sql"), 'DROP TABLE "mine";', "utf8"); + return dir; +} + +// `id INTEGER NOT NULL PRIMARY KEY`, not a bare `INTEGER PRIMARY KEY`: sqlite reports +// notnull=0 for the latter, so `nullable: false` here would read as drift and the +// test would fail for a reason unrelated to scope. `test/integrity/replay.test.ts:47` +// writes it the same way, for the same reason. +const CHAIN = 'CREATE TABLE "mine" (id INTEGER NOT NULL PRIMARY KEY);'; + +const table = (name: string) => ({ + name, + columns: [{ name: "id", sqlType: { kind: "integer" as const, bits: 64 as const }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], +}); + +// `theirs` belongs to another owner: `carryForwardOutOfScope` put it in the snapshot, +// and the chain — correctly — never creates it. +const SNAPSHOT: SchemaSnapshot = { tables: [table("mine"), table("theirs")], views: [] }; + +describe("verifyReplay honours migrate.scope", () => { + test("an out-of-scope table in the snapshot is not reported as missing", async () => { + const dir = chainWith(CHAIN); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, + dialect: "sqlite", + migrationsDir: dir, + snapshot: SNAPSHOT, + // Qualified: `.`, absent schema normalized to `public`. + governed: { outOfScope: ["public.theirs"] }, + }); + expect(result.ok).toBe(true); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + test("without `governed`, the same case reports drift — the control", async () => { + const dir = chainWith(CHAIN); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, dialect: "sqlite", migrationsDir: dir, snapshot: SNAPSHOT, + }); + expect(result.ok).toBe(false); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); +``` + +The control case is what makes the first test non-vacuous: it proves the difference comes from +`governed` and not from the fixture being trivially green. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/integrity/replay-scoped.test.ts` +Expected: the first test FAILS (`governed` is not accepted / not honoured); the control PASSES. + +- [ ] **Step 3: Implement** + +In `verify/replay.ts`, add the optional field to `VerifyReplayArgs`: + +```ts + /** + * The scope decision the run made, as `scopeExpectedSchema` reports it. A project + * declaring `migrate.scope` carries the OTHER owner's tables into the committed + * snapshot on purpose (`carryForwardOutOfScope`), and the chain never creates them — + * so without this they read as missing on every replay. Excluded from the SNAPSHOT + * side only; the replayed database never had them either. + */ + governed?: GovernedScope; +``` + +and apply the exclusion before comparing: + +```ts + const expected = args.governed !== undefined + ? excludeFromSnapshot(args.snapshot, args.governed).snapshot + : args.snapshot; + const classification = await driftAgainstSnapshot(expected, actual, args.dialect); +``` + +Import `excludeFromSnapshot` and `type GovernedScope` from `../scope.js`. Do not change the +signature's required fields — an existing caller passing no `governed` must behave exactly as before. + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/integrity/replay-scoped.test.ts` +Expected: PASS, both. + +- [ ] **Step 5: Run the suite, typecheck, commit** + +Run: `cd server/typescript && bun test packages/migrate-ts` — PASS. +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/migrate-ts/src/verify/replay.ts \ + server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts +git commit -m "fix(migrate): verifyReplay honours migrate.scope on the snapshot side" +``` + +--- + +## Task 5: `meta verify --replay` — the chain applies from empty + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`VerifyFlags`, `parseVerifyArgs`) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` +- Test: `server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts` (create) +- Test: `server/typescript/packages/cli/test/verify-replay.test.ts` (create) + +**Interfaces:** +- Consumes: `openReplayEngine` (Task 3). +- Produces: the `--replay` flag on `VerifyFlags`; Task 6 adds `--replay-snapshot` beside it. + +**Context:** `verify` composes gates and returns `Math.max(...)` of their exit codes +(`verify.ts:239`). `anyExplicit` (`args.ts:290`) decides whether the bare-`verify` default fires — +`--replay` must be included in it, or passing `--replay` alone would also silently run the template +gate. Refuse `--migration-format flyway` and `--dialect d1`, mirroring `apply-pending` +(`migrate.ts:419-426`, `:449-457`). + +### The dialect, when there is no `--db` + +`--replay` has no connection URL to infer a dialect from, and `--replay-snapshot` needs one to name +the snapshot file (`snapshotPath(dir, dialect)`). **Precedence, stated so it is not invented twice:** + +1. an explicit `--dialect`, +2. else `resolveMigrateConfig(EMPTY_MIGRATE_FLAGS, projectRoot).dialect`, +3. else refuse with **exit 2**, naming `--dialect`. + +Step 2 requires amending the comment on `EMPTY_MIGRATE_FLAGS` (`verify.ts:81-87`), which today says +verify "consumes only `outDir`… reading any of them here would be reaching into migrate's +decisions". Amend it, do not quietly contradict it: + +```ts + * `verify` consumes `outDir` (#292) and — for the replay gate only — `dialect`. The + * #292 restriction was about the DRIFT gate, whose dialect comes from the live `--db` + * URL; the replay gate has no `--db`, and the dialect a committed chain was emitted + * for IS a migrate decision, so migrate's own resolution is the only correct source. + * Everything else here exists to satisfy the shared shape. +``` + +### Zero committed migrations + +`discoverMigrations` is **not exported** from `apply/apply.ts` (it is a module-private function at +`:316`), so the gate cannot call it. Use `applyPending`'s return value instead: +`ApplyPendingResult` is `{ pending: string[]; applied: string[] }`, and on a fresh in-process +database with no ledger every discovered migration is pending — so `pending.length === 0` means the +directory held none. Report it and return 0; do not pass silently. + +- [ ] **Step 1: Write the RED-first regression — the emitter's own SQL, through `applyPending`** + +This is the test the previous draft of this plan was missing. Hand-writing the SQL proves nothing +about the emitter: such a test stays green if Task 1 is reverted. This one builds a `drop-table` +change, runs it through `emit()` and `writeMigration()`, and applies the resulting file — so it is +RED before Task 1 and GREEN after, and it is the only assertion that proves an *emitted* chain +replays. + +```ts +// server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts +// +// #313, end to end: the EMITTER's output, written by writeMigration, applied by +// applyPending into an empty in-process database. Every prior defect in this area +// (#226/#241, #243, #255, #285, 0.21.4's BEGIN TRANSACTION finding) shared one shape +// — SQL proven statement-by-statement and never proven through the tool that applies +// it. applyPending rewrites statements via prepareForRunnerTransaction before +// executing, so an emit-level assertion cannot see this class of bug. +import { describe, test, expect } from "bun:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { emit } from "../../src/emit/index.js"; +import { writeMigration } from "../../src/write-migration.js"; +import { applyPending } from "../../src/apply/apply.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import type { Change, ChangeStatus } from "../../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +// Exactly the reported shape: another tool owned `theirs`, so the diff proposed +// dropping it, and no migration in the chain ever created it. +const REPORTED: Change[] = [ + { + kind: "create-table", status: ALLOWED, + table: { + name: "mine", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], + }, + }, + { kind: "drop-table", table: "theirs", status: ALLOWED }, +]; + +describe("an EMITTED chain applies to an empty database (#313)", () => { + for (const dialect of ["sqlite", "postgres"] as const) { + test(`${dialect}: emit → writeMigration → applyPending, from empty`, async () => { + const dir = mkdtempSync(join(tmpdir(), `replay-emitted-${dialect}-`)); + const engine = await openReplayEngine(dialect); + try { + const result = emit(REPORTED, { dialect }); + await writeMigration(dir, "init", result); + const applied = await applyPending(engine.db, dir, { dryRun: false, dialect }); + expect(applied.applied).toHaveLength(1); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + } + + test("the control: a HAND-WRITTEN bare drop still fails, so the assertion has teeth", async () => { + const dir = mkdtempSync(join(tmpdir(), "replay-emitted-control-")); + const engine = await openReplayEngine("sqlite"); + try { + await writeMigration(dir, "init", { + up: 'CREATE TABLE "mine" (id INTEGER NOT NULL PRIMARY KEY);\n\nDROP TABLE "theirs";', + down: 'DROP TABLE "mine";', + recreatedTables: new Set(), + }); + await expect(applyPending(engine.db, dir, { dryRun: false, dialect: "sqlite" })) + .rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); +``` + +**Adjust `writeMigration`'s call shape to the real signature** — read +`migrate-ts/src/write-migration.ts` first; it may take an options object rather than +`(dir, slug, result)`, and its `EmitResult` may carry more fields than the three above. +`ApplyPendingResult` names its list `applied`. + +- [ ] **Step 2: Run it — it must be RED unless Task 1 has landed** + +Run: `cd server/typescript && bun test packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts` +Expected **if Task 1 has landed**: PASS, all three. +Expected **if Task 1 has not landed**: the two emitted-chain cases FAIL with `theirs`, the control +passes. Either outcome is informative — record which one you saw. If Task 1 has landed and a case +still fails, STOP: the emitter fix is incomplete. + +- [ ] **Step 3: Add the flag** + +In `cli/src/lib/args.ts`, add to `VerifyFlags`: + +```ts + /** Replay the committed migration chain into an in-process database and assert it applies. */ + replay: boolean; +``` + +add to the `parseArgs` options: + +```ts + replay: { type: "boolean", default: false }, +``` + +include it in the parsed object (`replay: !!values.replay`), and add it to `anyExplicit`: + +```ts + const anyExplicit = templates || codegen || values.db !== undefined || dialect === "d1" || !!values.replay; +``` + +- [ ] **Step 4: Write the CLI tests** + +```ts +// server/typescript/packages/cli/test/verify-replay.test.ts +import { describe, test, expect } from "bun:test"; +import { parseVerifyArgs } from "../src/lib/args.js"; + +describe("verify --replay flag", () => { + test("is parsed", () => { + expect(parseVerifyArgs(["--replay"]).replay).toBe(true); + }); + + test("counts as an explicit subverb, so it does not also run the template gate", () => { + expect(parseVerifyArgs(["--replay"]).anyExplicit).toBe(true); + }); + + test("defaults off", () => { + expect(parseVerifyArgs([]).replay).toBe(false); + }); +}); +``` + +- [ ] **Step 5: Implement the gate** + +In `cli/src/commands/verify.ts`, add a `runReplayVerify()` beside the existing gates and include it +in the `Math.max`: + +```ts + const replayExit = (flags.replay || flags.replaySnapshot) ? await runReplayVerify() : 0; + … + return Math.max(templateExit, schemaExit, codegenExit, requirementExit, replayExit); +``` + +**Write the condition as `(flags.replay || flags.replaySnapshot)` now, in this task**, even though +`replaySnapshot` does not exist yet — add the field in Task 6 and this line needs no second edit. +(If TypeScript objects to the missing field, add `replaySnapshot: boolean` to `VerifyFlags` here and +wire its parsing in Task 6; do not leave the condition reading `flags.replay` alone, which is how +`--replay-snapshot` would ship dead.) + +`runReplayVerify` must: + +1. **Resolve the dialect** by the precedence above. Refuse `d1` and `--migration-format flyway` with + `log.error` and **exit 2**, matching `apply-pending`'s wording at `migrate.ts:419-426` and + `:449-457`. Refuse with exit 2 and a message naming `--dialect` when no dialect can be resolved. +2. **Resolve the migrations directory through migrate's OWN precedence** — the same + `resolveMigrateConfig(EMPTY_MIGRATE_FLAGS, projectRoot)` → `resolvePath(projectRoot, migrateConfig.outDir)` + pair `checkCommittedSnapshot` uses (`verify.ts:639-641`). Do not re-derive it. +3. `openReplayEngine(dialect)` inside a `try`, with `await engine.dispose()` in the `finally`. + An engine that cannot start (including a missing optional peer) is **operational → return 2**, + and the error's own message already carries the install hint. +4. `applyPending(engine.db, dir, { dryRun: false, dialect })`. +5. **Zero migrations is not a silent pass.** When the result's `pending` is empty, print + `meta verify --replay: no committed migrations — nothing to replay` and return 0. +6. On an apply failure, print the failing statement and the remediation. An already-applied chain + cannot be repaired by hand-editing (`apply/apply.ts:88-99` makes migrations checksum-immutable), + so the message must name the compensating-migration path: + `meta verify --replay: the committed chain does not apply to an empty database. Applied migrations are immutable, so fix this with a NEW migration that creates the missing object — not by editing a committed up.sql.` + Return **1** (drift), not 2. + +- [ ] **Step 6: Add a behavioural CLI test, not only a flag-parse test** + +Three flag-parse assertions do not prove the gate runs. Add one case that drives `verifyCommand` +against a temporary project whose committed chain contains a bare drop for an object it never +creates, and assert the command returns **1**. Model the project fixture on +`packages/cli/test/migrate-scope.test.ts`'s harness — read it first — but note it drives the +**offline** path (`runOfflineGenerate`/`runBaseline`); what you need here is a `metaobjects/` +directory, a `.metaobjects/config.json` declaring `migrate.dialect`, and a committed +`migrations/-init/up.sql`. No database is involved, which is the point. + +Also assert the zero-migrations case returns 0 and says so. + +- [ ] **Step 7: Run the tests** + +Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts && bun test packages/migrate-ts/test/integrity` +Expected: PASS. + +- [ ] **Step 8: Run the suites, typecheck, commit** + +Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts` — PASS. +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/cli/src/lib/args.ts \ + server/typescript/packages/cli/src/commands/verify.ts \ + server/typescript/packages/cli/test/verify-replay.test.ts \ + server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts +git commit -m "feat(cli): meta verify --replay asserts the committed chain applies from empty" +``` + +--- + +## Task 6: `meta verify --replay-snapshot` — the chain reproduces the snapshot + +**Files:** +- Modify: `server/typescript/packages/cli/src/lib/args.ts` +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` +- Test: `server/typescript/packages/cli/test/verify-replay.test.ts` (extend) + +**Interfaces:** +- Consumes: `openReplayEngine` (Task 3), `verifyReplay` with `governed` (Task 4), the + `runReplayVerify` structure and the `(flags.replay || flags.replaySnapshot)` condition (Task 5). +- Produces: nothing later tasks depend on. + +**Context:** This is a **separate subverb, not `--strict`**. `verify` already owns a `--lax` flag on +a different axis (ADR-0023 attribute strictness, `args.ts:244`), and `--strict` beside it would read +as that flag's opposite rather than as a replay depth. + +**This tier cannot pass for baseline-adopted projects and does not try to detect them.** The only +candidate signal, `BASELINE_NAME`/`recordBaseline` (`ledger.ts:205-227`), has no production caller +and would live in the target database's ledger while this gate runs against a fresh in-process +database with no ledger at all. The limitation is documented in Task 8, and the failure message +names it. + +### `governed` has to be derived OFFLINE + +`verify.ts:659` gets its `GovernedScope` from `driftResult` (`verify.ts:498`), which comes from +`computeDriftFromActual` and needs a live `--db`. This gate has none. Derive it instead: + +- `collection.inMigrateScope` (already read at `verify.ts:217` as `schemaScope`) is the predicate. +- **When it is `undefined`, pass no `governed` at all** — an unscoped project's comparison is then + byte-for-byte unchanged, which is most projects and keeps the added surface confined. +- When it is defined, build the provenance-bearing expected schema and scope it, exactly as + `verify.ts:435-441` already does inside `migrateScopeMismatch`: + ```ts + const viewStrategy = forgeConfig?.columnNamingStrategy ?? "snake_case"; + const built = buildExpectedSchemaWithProvenance(root, { + dialect, + columnNamingStrategy: viewStrategy, + views: buildProjectionViews(root, { dialect, columnNamingStrategy: viewStrategy }), + }); + const governed = scopeExpectedSchema(built, schemaScope); // satisfies GovernedScope + ``` + `scopeExpectedSchema` needs `ExpectedSchemaWithProvenance` — a snapshot plus a + qualified-name → metadata-FQN map — which is why the committed snapshot alone cannot be scoped + and the metadata must be rebuilt here. + +### One engine, one real apply + +`verifyReplay` calls `applyPending` unconditionally (`replay.ts:31`). That is **not** a second +replay: the first `applyPending` (Task 5 step 5.4) recorded every migration in the in-process +ledger, so the second call finds nothing pending and returns immediately. Share one engine, let each +tier keep its own failure message, and do not restructure `verifyReplay` to avoid the call. + +- [ ] **Step 1: Write the failing tests** + +Append to `packages/cli/test/verify-replay.test.ts`: + +```ts +describe("verify --replay-snapshot flag", () => { + test("is parsed", () => { + expect(parseVerifyArgs(["--replay-snapshot"]).replaySnapshot).toBe(true); + }); + + test("counts as an explicit subverb", () => { + expect(parseVerifyArgs(["--replay-snapshot"]).anyExplicit).toBe(true); + }); + + test("does not collide with --lax, which is a different axis", () => { + const f = parseVerifyArgs(["--replay-snapshot", "--lax"]); + expect(f.replaySnapshot).toBe(true); + expect(f.lax).toBe(true); + }); +}); +``` + +**And the behavioural test that keeps the flag from shipping dead** — the previous draft had three +flag-parse tests and nothing that ran the gate, so `--replay-snapshot` would have parsed fine and +done nothing. Using the Task 5 Step 6 project harness: + +```ts +describe("verify --replay-snapshot actually runs the gate", () => { + test("a chain that does not apply fails under --replay-snapshot alone", async () => { + // --replay-snapshot implies --replay's work; a broken chain must fail even when + // --replay was not passed. + const project = await projectWithBrokenChain(); + expect(await verifyCommand(["--replay-snapshot"], project)).toBe(1); + }); + + test("a chain that applies but does not reproduce the snapshot fails", async () => { + const project = await projectWithChainDivergentFromSnapshot(); + expect(await verifyCommand(["--replay-snapshot"], project)).toBe(1); + }); + + test("a chain that applies and reproduces the snapshot passes", async () => { + const project = await projectWithGoodChain(); + expect(await verifyCommand(["--replay-snapshot"], project)).toBe(0); + }); +}); +``` + +Adapt the helper names and the `verifyCommand` call shape to the real harness. + +- [ ] **Step 2: Run it to verify it fails** + +Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts` +Expected: FAIL — `--replay-snapshot` is not a known option (`strict: true` in `parseArgs`). + +- [ ] **Step 3: Implement the flag** + +Same places as Task 5: the `VerifyFlags` field (`replaySnapshot: boolean`), the `parseArgs` option +(`"replay-snapshot": { type: "boolean", default: false }`), the parsed object +(`replaySnapshot: !!values["replay-snapshot"]`), and `anyExplicit`. **Verify that Task 5's +`(flags.replay || flags.replaySnapshot)` condition is present and now reachable** — this is the exact +line whose absence would ship the flag dead. + +- [ ] **Step 4: Implement the tier** + +Extend `runReplayVerify` so that when `flags.replaySnapshot` is set it additionally: + +1. Loads the committed snapshot the way `checkCommittedSnapshot` does (`verify.ts:639-644`): + `readSnapshot(snapshotPath(dir, dialect))` on the already-resolved `dir`, inside a `try`. + **Fail OPEN**: a `null` snapshot, or an unreadable/unparseable file, reports + `meta verify --replay-snapshot: no committed snapshot — nothing to compare` and contributes 0. + A project that has never generated one offline is not in an error state, and a parse failure is + migrate's error to raise with its own message, not a drift verdict. +2. Derives `governed` per the section above. +3. Calls `verifyReplay({ db: engine.db, dialect, migrationsDir: dir, snapshot, ...(governed !== undefined ? { governed } : {}) })`. +4. On `ok === false`, reports the drift and returns **1**, with a message that names baseline + adoption as the first thing to rule out: + `meta verify --replay-snapshot: the replayed chain does not reproduce the committed snapshot. If this project was adopted with 'migrate baseline --from-db', its chain does not build the schema and this tier does not apply — use --replay instead.` + +- [ ] **Step 5: Run the tests** + +Run: `cd server/typescript && bun test packages/cli/test/verify-replay.test.ts` +Expected: PASS. + +- [ ] **Step 6: Run the suites, typecheck, commit** + +Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts` — PASS. +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/cli/src/lib/args.ts \ + server/typescript/packages/cli/src/commands/verify.ts \ + server/typescript/packages/cli/test/verify-replay.test.ts +git commit -m "feat(cli): meta verify --replay-snapshot asserts the chain reproduces the snapshot" +``` + +--- + +## Task 7: Emit-time provenance guard + +**Files:** +- Modify: `server/typescript/packages/migrate-ts/src/types.ts` (`AllowOptions.dropUnmanaged`) +- Modify: `server/typescript/packages/cli/src/lib/args.ts` (`ALLOW_TOKENS`) +- Modify: `server/typescript/packages/cli/src/lib/allow.ts` (`ALLOW_TOKEN_MAP`) +- Modify: `server/typescript/packages/sdk/src/config.ts` (`AllowTokenEnum`) +- Modify: `server/typescript/packages/cli/src/commands/migrate.ts` +- Test: `server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts` (create) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: the `drop-unmanaged` allow token. + +**Context:** Tasks 1–6 make a bad chain survivable and detectable. This stops it being written. The +live path (`migrate.ts:607-620`) diffs metadata against introspection and **never consults the +committed snapshot**, which is why an object no snapshot ever contained gets proposed for a drop. +`drift/classify.ts:6-9` already states the doctrine: objects present in the DB but not the snapshot +"must never be treated as actionable drift or auto-dropped". + +The guard does not false-fire on the brownfield classes, because both of them *add* to the snapshot: +a `baseline --from-db` snapshot contains the foreign table, and a scoped project carries out-of-scope +entries forward into it. The guard fires precisely when nothing ever claimed the object. + +### A new `--allow` token touches FOUR files, and a pin test will catch three of them + +`cli/test/unit/allow-tokens-pinned.test.ts` asserts three invariants, all self-deriving (so it needs +no edit, and it fails loudly if any file is missed): + +1. `ALLOW_TOKENS` (`cli/src/lib/args.ts:173`) ≡ `AllowTokenEnum.options` (`sdk/src/config.ts:22`) — + the sdk enum validates `migrate.allow` in `.metaobjects/config.json`. +2. `ALLOW_TOKEN_MAP` (`cli/src/lib/allow.ts:13`) has exactly one key per token — the map is what + *grants* the permission; a token in the list but not the map validates cleanly and silently grants + nothing. +3. Every mapped value is a **distinct** `AllowOptions` field. + +**Decision: add `dropUnmanaged?: boolean` to `AllowOptions`** (`migrate-ts/src/types.ts:300`) and +wire the token through all four structures. The alternative — keeping `drop-unmanaged` out of +`ALLOW_TOKENS` and validating it separately — means a second token list and a second parse path for +exactly one token, which is precisely the drift the pin test exists to prevent. The cost is that one +`AllowOptions` field is read by the CLI rather than by `diff()`'s status pass; document that at the +field, so it is a stated exception rather than a puzzle. + +- [ ] **Step 1: Write the failing tests** + +```ts +// server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts +import { describe, test, expect } from "bun:test"; +import { ALLOW_TOKENS } from "../src/lib/args.js"; +import { ALLOW_TOKEN_MAP } from "../src/lib/allow.js"; +import { tokensToAllowOptions } from "../src/lib/allow.js"; + +describe("drop-unmanaged allow token", () => { + test("is a recognised allow token", () => { + expect(ALLOW_TOKENS).toContain("drop-unmanaged"); + }); + + test("grants a permission rather than validating into nothing", () => { + expect(ALLOW_TOKEN_MAP["drop-unmanaged"]).toBe("dropUnmanaged"); + expect(tokensToAllowOptions(["drop-unmanaged"]).dropUnmanaged).toBe(true); + }); +}); +``` + +Then the behavioural tests. They must drive the real live `migrate` path with a committed snapshot +that does NOT contain the table being dropped, and assert the run refuses. Three cases: + +1. A drop proposed for a table absent from the committed snapshot ⇒ refused, **exit 2**, message + names the object and `--allow drop-unmanaged`. +2. The same run with `--allow drop-unmanaged` ⇒ proceeds. +3. **Non-false-fire:** a drop for a table that IS in the snapshot ⇒ proceeds without the flag. +4. **Fail open:** the same run with NO snapshot on disk ⇒ proceeds without the flag. + +**The harness is the hard part, and `migrate-scope.test.ts` is not it.** That file drives the +OFFLINE path (`runOfflineGenerate`/`runBaseline`), where the expected side already *is* the snapshot, +so the guard can never fire there. The guard's target is the **live** path (`migrate.ts:607-620`), +which needs a real database. Use the `:memory:` libsql engine from Task 3 — or, if Task 7 runs before +Task 3, `LibsqlDialect({ url: ":memory:" })` directly (verified working) — seed it with the tables the +scenario needs, and point `meta migrate --db` at it. Read +`packages/migrate-ts/test/integration/` for how the existing live-path tests build a database and a +project side by side. + +- [ ] **Step 2: Run to verify it fails** + +Run: `cd server/typescript && bun test packages/cli/test/migrate-drop-unmanaged.test.ts` +Expected: FAIL — the token is not in `ALLOW_TOKENS`. + +- [ ] **Step 3: Add the token — all four files** + +`migrate-ts/src/types.ts`, in `AllowOptions`: + +```ts + /** + * Permits dropping an object the COMMITTED SNAPSHOT never contained — i.e. one + * this toolchain never managed. Without it such a drop is refused at generation + * time, because it produces a migration that cannot replay against a database + * where that object never existed (#313). + * + * The one field here read by the CLI's generation-time provenance guard rather + * than by `diff()`'s status pass: it lives in `AllowOptions` so `--allow` keeps + * ONE token list and ONE grant map (`ALLOW_TOKENS` / `ALLOW_TOKEN_MAP`, pinned by + * `cli/test/unit/allow-tokens-pinned.test.ts`). A second parallel validation path + * for a single token is the drift that pin exists to prevent. + */ + dropUnmanaged?: boolean; +``` + +`cli/src/lib/args.ts`, in `ALLOW_TOKENS`, with a comment in the style of its neighbours: + +```ts + // drop-unmanaged permits dropping an object the COMMITTED SNAPSHOT never + // contained — i.e. one this toolchain never managed. Without it such a drop is + // refused at generation time, because it produces a migration that cannot replay + // against a database where that object never existed (#313). + "drop-unmanaged", +``` + +`cli/src/lib/allow.ts`, in `ALLOW_TOKEN_MAP`: + +```ts + // Read by migrate's generation-time provenance guard, not by diff()'s status pass + // — see AllowOptions.dropUnmanaged for why it still lives in that shape. + "drop-unmanaged": "dropUnmanaged", +``` + +`sdk/src/config.ts`, in `AllowTokenEnum`: + +```ts + "drop-unmanaged", +``` + +- [ ] **Step 4: Implement the guard** + +In `cli/src/commands/migrate.ts`, in the **live** path, immediately after +`changeCounts = summarizeChanges(diffResult.changes);` (`migrate.ts:657`) and **before** the +`emit(...)` block: + +```ts + // #313 — refuse to author a drop for an object the committed snapshot never + // contained. The live path diffs metadata against introspection and never reads + // the snapshot, so an object another tool owns reads as "in the DB, not in the + // model" and is proposed for a drop; the resulting migration then cannot replay + // against a database where the object never existed. `classify.ts:6-9` already + // states the doctrine — this is where it is enforced. + // + // Fails OPEN when there is no snapshot on disk: a project that has never + // generated one is not in an error state, and refusing there would break the + // first `meta migrate` of every greenfield project. +``` + +The implementation: + +1. Load the snapshot the way `migrate.ts:781-782` already spells it: + `readSnapshot(snapshotPath(resolvePath(metaRoot, config.outDir), kysely.dialect))`, inside a + `try`. A thrown read, or a `null` result, **skips the guard entirely**. +2. Build the set of snapshot object names with **`qualifiedDbName`** (`migrate-ts/src/qualified-name.ts`) + over `snapshot.tables` and `snapshot.views`. Use that helper and nothing else — three independent + sets already have to agree on this spelling, and a fourth would silently un-guard objects. +3. Collect every `drop-table` / `drop-view` in `diffResult.changes` whose `qualifiedDbName({ name, schema })` + is absent from that set. (`drop-table` carries `table: string` + `schema?`; `drop-view` carries + `view: string` + `schema?`.) +4. If that set is non-empty and `tokensToAllowOptions(config.allow).dropUnmanaged` is not true, + `log.error` naming each object and `--allow drop-unmanaged`, close the connection, and return 2. + +- [ ] **Step 5: Run the tests** + +Run: `cd server/typescript && bun test packages/cli/test/migrate-drop-unmanaged.test.ts && bun test packages/cli/test/unit/allow-tokens-pinned.test.ts` +Expected: PASS. The pin test passing is the evidence all four files were updated. + +- [ ] **Step 6: Run the suites, typecheck, commit** + +Run: `cd server/typescript && bun test packages/cli && bun test packages/migrate-ts && bun test packages/sdk` — PASS. +Run: `bun run --filter '*' typecheck` — all 18 exit 0. + +```bash +git add server/typescript/packages/migrate-ts/src/types.ts \ + server/typescript/packages/cli/src/lib/args.ts \ + server/typescript/packages/cli/src/lib/allow.ts \ + server/typescript/packages/sdk/src/config.ts \ + server/typescript/packages/cli/src/commands/migrate.ts \ + server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts +git commit -m "feat(cli): refuse to drop an object the committed snapshot never managed" +``` + +--- + +## Task 8: Documentation and CHANGELOG + +**Files:** +- Modify: `docs/features/migrations-and-drift.md` +- Modify: `server/typescript/packages/cli/src/commands/migrate.ts` (help text) +- Modify: `server/typescript/packages/cli/src/commands/verify.ts` (help/subverb note) +- Modify: `CHANGELOG.md` + +**Interfaces:** none. + +- [ ] **Step 1: Correct the overclaim** + +`docs/features/migrations-and-drift.md:58` currently says `apply-pending` "is the way to provision a +fresh or CI database". That is true only of a chain that builds the schema. Scope the sentence, and +point at `meta verify --replay` as the way to know your chain is one of those. + +- [ ] **Step 2: Document both tiers** + +In the same file, document `meta verify --replay` and `--replay-snapshot`: what each asserts, that +they run in-process (PGlite / `:memory:` libsql) and provision nothing, that PGlite is an **optional +peer** a postgres project installs to use the gate, that flyway and d1 are refused, how the dialect +is resolved without `--db`, and — stated plainly — that `--replay-snapshot` does not apply to a +project adopted with `migrate baseline --from-db`, because such a chain does not build the schema. + +- [ ] **Step 3: Document the guard** + +Document `--allow drop-unmanaged`: what triggers the refusal, why (a migration that cannot replay), +that it fails open when there is no committed snapshot, and that the escape hatch exists for a drop +you genuinely intend. + +- [ ] **Step 4: Update the CLI help** + +Add the two subverbs to `verify`'s help and to the one-line note at `verify.ts:127-130` that +advertises the explicit subverbs. Add `drop-unmanaged` to `migrate --help`'s `--allow` token list. +Update `migrate --help`'s `apply-pending` line (`migrate.ts:69-70`) so it no longer promises +fresh-database provisioning unconditionally. + +- [ ] **Step 5: CHANGELOG** + +Add an `## [Unreleased]` entry covering the adopter-visible changes, in this order: + +1. **The new refusal** — a `meta migrate` that proposes dropping an object the committed snapshot + never contained now exits 2 and requires `--allow drop-unmanaged`. This leads because it is the + one change that can fail an existing project's `meta migrate`. +2. Emitted **forward** drops now carry `IF EXISTS` (`drop-table`, `drop-view`, `drop-index` incl. the + constraint-backed arm, `drop-fk`, `drop-check`); downs are unchanged, and D1 inherits the sqlite + change. Note the deliberate exclusions. +3. A chain creating a table **or view** in a non-default schema now emits `CREATE SCHEMA IF NOT EXISTS`. +4. Two new verify subverbs, `--replay` and `--replay-snapshot`, with `@electric-sql/pglite` as a new + **optional peer** of `@metaobjectsdev/migrate-ts` (only needed to replay a postgres chain). + +- [ ] **Step 6: Leak scan and commit** + +The committed hook is the mechanism — do not invent a pattern list. Confirm it is wired, then let it +run on commit: + +```bash +git config core.hooksPath # must print .githooks +git config hooks.denyListPath # must print a path that exists +grep -rnE '/home/[a-z]' docs/features/migrations-and-drift.md CHANGELOG.md && echo "ABSOLUTE HOME PATH" || echo "no home paths" +``` + +If the hook blocks the commit, **genericize** — never `--no-verify`. + +```bash +git add docs/features/migrations-and-drift.md CHANGELOG.md \ + server/typescript/packages/cli/src/commands/migrate.ts \ + server/typescript/packages/cli/src/commands/verify.ts +git commit -m "docs: replay tiers, the drop-unmanaged refusal, and the provisioning promise" +``` + +--- + +## Self-Review + +**Spec coverage.** §3.1 forward drops → Task 1, with every site tabulated and every deliberate +exclusion pinned by a test that must stay green. §3.2 both tiers, engine, refusals, +zero-migrations, exit codes → Tasks 3, 5, 6. §3.2's `excludeFromSnapshot` threading → Task 4. +§3.3 `CREATE SCHEMA` → Task 2, extended to `create-view` because the spec says "the first object". +§3.4 provenance guard → Task 7. §3.5 docs → Task 8. §4 remediation → Task 5 Step 5.6's message text +and Task 8. §5 testing → each task's own steps; §5's "must run through `applyPending`, not `emit()`" +is Task 5 Step 1, which now runs through **both** — `emit()` → `writeMigration()` → `applyPending()` — +so it is RED before Task 1 rather than green regardless. + +**What was verified in code before this revision, not assumed.** The `drop-check` "unreachable" +comment is false (`diff/index.ts:579`, `:592`). `sqlType` is `{ kind: "integer"; bits: 32|64 }`, and +`GovernedScope` names are `.`. `excludeFromSnapshot` returns a `ScopedExpectedSchema`, +so the fix takes `.snapshot`. `discoverMigrations` is module-private, so zero-migrations is detected +from `ApplyPendingResult.pending`. `migrate-ts` has two dependencies and no driver, so PGlite (22 MB) +is an optional peer and `build:binary` needs an `--external`. The PGlite→kysely pool shim in Task 3 +was **executed** against 0.3.16 and 0.5.5 — DDL, schemas, CHECK, transactions, rollback, advisory +locks, and `table "theirs" does not exist`. `LibsqlDialect({ url: ":memory:" })` works and two +instances are isolated. `pg-constraint-backed-index-285.test.ts` needs **two** edits, not one: `:143` +goes red, `:142` goes vacuously green. + +**Two things deliberately left to the implementer, both flagged inline rather than guessed:** the +exact `writeMigration` call shape (Task 5 Step 1), and the live-path `migrate` test harness for the +guard (Task 7 Step 1, which also records that `migrate-scope.test.ts` is the WRONG model because it +drives the offline path). + +**Type consistency.** `openReplayEngine(dialect) → ReplayEngine { db, dispose }` is defined in Task 3 +and used verbatim in Tasks 4, 5, 6. `VerifyReplayArgs.governed?: GovernedScope` is defined in Task 4 +and consumed in Task 6. `replay` / `replaySnapshot` are added in Tasks 5 and 6, both feed +`anyExplicit`, and the `Math.max` condition covering both is written **once**, in Task 5. +`AllowOptions.dropUnmanaged` is added in Task 7 alongside all three token structures the pin test +compares. diff --git a/docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md b/docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md new file mode 100644 index 000000000..7a07bec5d --- /dev/null +++ b/docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md @@ -0,0 +1,213 @@ +# Cross-port metadata `sources` — design + +_Status: DESIGN (decisions made; implementation plan is a separate document)._ +_Date: 2026-08-19._ +_Follows: `2026-08-17-metadata-source-resolution-design.md` (phase 1, TypeScript, merged as PR #311)._ +_Evidence: `2026-08-17-metadata-source-resolution-prior-art.md`._ + +## 1. What this decides + +Phase 1 gave the Node `meta` CLI three keys in the port-neutral `.metaobjects/config.json`: +`sources`, `scope`, and `migrate.scope`. The other four CLIs read none of them. + +This document decides how the **Java, Kotlin, C# and Python** CLIs learn where metadata lives. + +**The decision, in one line: ship `sources` to all four ports now, hold `scope` back, and read a +declared neutral SUBSET of the config file rather than the whole schema.** + +The requirement being served, as stated by the maintainer: + +> the ports need not work exactly the same way, but they must be able to read all the same +> sources, and the `.metaobjects` config should be the default when nothing else is specified. + +## 2. Decision 1 — `sources` and `scope` are separate deliveries + +They were bundled in phase 1 because one TypeScript implementation served both. Cross-port they +are not one feature, and bundling them triples the cost of the half that is nearly free. + +**`sources` is cheap.** Every port's loader already accepts a set of sources; only the CLI +convenience wrapper is single-directory: + +| Port | Set-accepting loader entry | Single-dir wrapper | +|---|---|---| +| Java | `load(List)` — `MetaDataLoader.java:1541`; also `setSourceURIs(List)` at `:994` | `:660` | +| C# | `Load(IReadOnlyList)` — `MetaDataLoader.cs:334` | `FromDirectory` `:93`, `:102` | +| Python | `load(sources: list[MetaDataSource])` — `meta_data_loader.py:102` | `from_directory` `:149` | +| TypeScript | (phase 1) | — | + +`DirectorySource` exists in all four codebases (Kotlin inherits the JVM one). The work is CLI +plumbing plus a config reader, not engine work. + +**`scope` is expensive, and it collides.** Java already ships a filter grammar, and it uses the +same characters for different meanings. From `GeneratorUtil.createRegexFromGlob`: + +```java +case '*': out += ".*"; break; // CROSSES :: — any depth +case '@': out += "[^:]+"; break; // exactly one segment +case ':': out += "\\:"; break; // TODO: This doesn't seem to work on enforcing the ::'s as a separator for * +``` + +Java's `@` is the new grammar's `*`; Java's `*` is the new grammar's `**`. Java additionally has +`!`-prefix exclusion (`:36-39`) and a `.[attr]` predicate suffix (`:72`) that `scope` cannot +express. The file carries a `TODO` conceding the separator handling is wrong. + +The two are also the **same tier**, so they compete rather than layer: +`AbstractMetaDataMojo.java:161-165` merges loader-level and generator-level `` and hands +the union to the generator, and TypeScript's `scope` is documented as an "Output filter applied +across every command" (`config.ts:133`). Both filter output; neither filters the load. + +**Therefore:** `scope` cross-port requires either a grammar migration for existing Java consumers +or two coexisting grammars. That is a decision with adopter impact, and it is not a prerequisite +for "read all the same sources." It is deferred to its own design (§8). + +## 3. Decision 2 — the contract boundary + +This is the part the maintainer's "need not work exactly the same way" licenses, and it needs to +be explicit or five ports will each guess. + +### Identical across ports — this is the contract + +1. **The resolved file SET.** Given the same `sources` and the same tree, every port resolves the + same set of metadata files. +2. **The relative-path base.** A relative `path` resolves against **the directory holding the + declaring `.metaobjects/` folder** — never against ambient cwd. Absolute paths are taken + as-is. (TS: `resolveSpecPath`, `sources.ts:102-104`. Python already obeys this rule for its + own key: `project_config.py:77-79` resolves `metadata` under `config_dir`.) +3. **Which file kinds count as metadata** when a directory is walked recursively. +4. **A declared source that does not exist is `ERR_SOURCE_UNRESOLVED`** — never a silent skip. + Only the *default* may be absent, and then it is `ERR_COLLECTION_NOT_FOUND`. +5. **An unsupported source kind is `ERR_SOURCE_KIND_UNSUPPORTED`.** `resource` and `package` are + declared in the shape but resolve in no port yet. +6. **A config file that exists but is malformed is an error** — it must never degrade to "no + config". TypeScript already does this deliberately (`collection.ts:129-140`). +7. **The default when nothing is declared:** one `path` source named `metaobjects` + (`metadata-files.ts:33`), and it is a default *value*, never a requirement. + +### May differ per port — explicitly NOT the contract + +1. **The ORDER of the resolved files.** It already diverges and always has: Java's + `DirectorySource.expand()` sorts by basename (`DirectorySource.java:105`), C# by full-path + ordinal (`DirectorySource.cs:64`), TypeScript walks depth-first with files-before-subdirs + (`metadata-files.ts:101-121`). Making order a contract would be a behavior change in three + ports for no gain: super-resolution is order-independent (#188) and the loader's overlay + partition discards caller order. **Ports keep their existing walk order.** +2. **How the port is told to override** — a positional argument, a `--config` flag, a pom element. +3. **Whether the port has walk-up discovery** (§5). +4. **Config-file caching, error message wording, and diagnostics formatting.** + +> **Known limit, stated rather than papered over.** An identical file *set* does not guarantee an +> identical loaded *model* — overlay partitioning and super-resolution sit downstream. This design +> gates the set. Model-level equivalence is what the existing metamodel conformance corpus gates, +> and the two are separate claims. + +## 4. Decision 3 — read a neutral SUBSET, not the whole schema + +`.metaobjects/config.json` is parsed by a `.strict()` zod schema that carries **TypeScript-owned +keys**: `pending_in_git`, `confidence_thresholds`, `extract.metaignore`, and `migrate` +(`config.ts:123-142`). + +Requiring four ports to model those — or to reject a valid config containing them — is untenable +and would make every future TS-only key a four-port change. + +**The rule:** + +- The **neutral subset** is `schema_version`, `sources`, and (later) `scope`. It is specified in + its own document and versioned by `schema_version`. +- The four ports **validate the subset strictly** and **ignore unknown top-level keys**. +- **TypeScript remains the only strict validator of the whole file, and the only writer.** + +The cost is real and accepted: a key misspelled *outside* the subset is caught only by the Node +CLI. The alternative — four ports modeling `confidence_thresholds` — is worse. + +## 5. Decision 4 — precedence, and what "default" means + +The maintainer's "the `.metaobjects` config is default if nothing is specified" is read as +**fallback**, not authority. The ladder, per port, first match wins: + +1. **An explicit CLI argument** — C#'s positional ``, Python's positional + `metadata_dir`, `--config`, `--cwd`. Wins outright. +2. **The port's native config surface**, when it names a metadata location — Java/Kotlin's pom + ``/``, Python's `metadata` key in `metaobjects.config.yaml`. +3. **`.metaobjects/config.json`'s `sources`**, when present and non-empty. +4. **The built-in default** — one `path` source named `metaobjects`. + +At every rung: **present but malformed is an error, not a fall-through to the next rung.** + +**Precedence is whole-concern for `sources`.** A native surface either names a metadata location +or it does not. There is no per-entry merging of a pom's `` with the neutral file's — +that would be a five-way merge matrix, and merge matrices are what drift across five +implementations. With `scope` deferred (§2) the per-key question does not arise in this phase; it +must be answered by the `scope` design when it lands. + +**Java's pom `` is not the neutral `sources`.** It lists individual metadata *documents* +(files, `resource:` classpath entries, `model:` URIs) resolved against `sourceDir`; neutral +`sources` is a set of *roots*. They are different primitives that happen to share a name. A pom +declaring either `` or `` occupies rung 2 and the neutral file is not +consulted; neither element changes meaning. + +## 6. Decision 5 — the writer gap is Node-side, not four writers + +The gap §4.6 of the phase-1 design named: a JVM-rooted adopter scaffolded with `agent-docs` has +no `.metaobjects/config.json` at all, so the Node CLI's `migrate` has nothing to discover no +matter which ports can read. + +**It does not follow that four ports need writers.** `migrate` and `verify --db` are Node-only +(ADR-0015), so the adopter who needs that file is by definition already invoking the Node CLI. +The missing piece is a *lighter* Node writer: `meta init` today scaffolds the full TypeScript +project (`metaobjects/`, `codegen/generators/`, `metaobjects.config.ts`, `.gitignore`, +`package.json` edits) and has no flag to write only the config (`init.ts:635-650` — the flags are +`force`, `quiet`, `printOnly`, `refreshDocs`, `d1`). + +**Decision: add a config-only mode to the Node `meta init`.** It writes `.metaobjects/config.json` +and nothing else, so a Maven- or pip-rooted project can declare its sources for the Node CLI +without acquiring a TypeScript scaffold it will not use. Four port writers stay out of scope. + +## 7. Conformance — a new corpus, and an honest note about its reach + +`fixtures/scope-conformance/` has 10 cases and **exactly one runner** +(`server/typescript/packages/sdk/test/scope-conformance.test.ts`). A corpus with one runner pins +nothing cross-port. Since `scope` is deferred, that corpus stays as-is and gains runners with the +`scope` design. + +**This phase adds `fixtures/source-resolution-conformance/`**, gating §3's identical column: +the resolved file **set** (order-insensitively), the relative-path base, the default-when-absent, +and each error condition. Every port ships a runner in the same changeset as its reader. + +> **A limitation to record rather than discover later.** The non-TypeScript lanes do not run on +> pull requests — they run on push-to-`main`, release tags, and manual dispatch +> (`AGENTS.md:91-92`). So this corpus has the same detection latency as the code it gates: it +> cannot catch a divergence *before* merge. It is a regression gate, not a review gate. That is +> the standing reason the four readers ship in one changeset rather than one port at a time. + +## 8. Deferred, with the reason + +- **`scope` cross-port** — blocked on the grammar collision (§2). Needs its own design answering: + migrate Java consumers, or run two grammars? Until then `scope` stays TypeScript-only and no + cross-port behavior may depend on it. +- **`resource` and `package` source kinds** — declared in the shape, resolving nowhere. `resource` + (classpath) is JVM-natural and unreachable from Node; `package` needs the distribution ADR + (prior art P5: no surveyed project publishes one code-free schema artifact to four registries). +- **Named collections** — the phase-1 design's §6 deferral is unchanged. +- **Walk-up discovery in the non-TS ports.** None has it today (Python's `_find_config` is + cwd-only, `cli.py:192-201`; C# has no config surface; Java uses the Maven reactor, which makes + discovery a non-issue for codegen). Ports read the config file at their already-known project + root. Adding a walk is a separate, additive change. + +## 9. What was checked, and what was not + +Verified in the repository at commit `19e421927` while writing this document: every file:line +citation above. The loader entry points, the `GeneratorUtil` grammar, the filter-merge site, the +per-port `DirectorySource` sort orders, the config schema's key list, the malformed-config +behavior, the `meta init` flag set, Python's `metadata` key and its cwd-only config lookup, and +the absence of any Kotlin CLI entry point (`fun main` appears in neither `metadata-ktx` nor +`codegen-kotlin` — Kotlin is Maven-only, so there are **four** CLI surfaces, not five). + +Also verified, after the first draft of this document listed it as an open risk: **all four ports +already agree on which extensions count as metadata** — `.json`, `.yaml`, `.yml`, matched +case-insensitively (`DirectorySource.java:61`, `DirectorySource.cs:29-30`, +`directory_source.py:16`, `metadata-files.ts:52`). §3's identical-item 3 is therefore a property +the ports already hold, not one the plan has to establish. It still gains a corpus case, because +nothing currently pins it. + +Nothing else in §3's identical column is known to diverge today, with the single exception of +file *order*, which §3 places explicitly outside the contract. diff --git a/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md new file mode 100644 index 000000000..20299cfae --- /dev/null +++ b/docs/superpowers/specs/2026-08-19-migrate-chain-replayability-design.md @@ -0,0 +1,259 @@ +# Migration chain replayability — drop safety, a provisioning gate, and the promise the docs already make + +**Issue:** [#313](https://github.com/metaobjectsdev/metaobjects/issues/313) · **Scope:** TypeScript +only (schema migration is TS-owned, [ADR-0015](../../../spec/decisions/ADR-0015-single-shared-migrate-engine.md)) +· **Supersedes nothing.** Completes the `verify --replay` tier specified but never wired in +[2026-05-31 migrate-ts reference-snapshot design](2026-05-31-migrate-ts-reference-snapshot-and-generation-gaps-design.md) §8. + +## 1. The problem, as reported + +`meta migrate` writes a bare `DROP TABLE "x"` into a committed migration when `x` is present in the +live database and absent from metadata — **even when no migration in the chain ever creates `x`**. +In the reported case another tool owned that table. Replaying the chain against an empty database +then dies: + +``` +$ meta migrate apply-pending --db postgresql://…/fresh +meta: migrate apply-pending: apply failed: table "other_owned_table" does not exist +``` + +Nothing warns at generation time. The reporter's chain was broken for roughly three months; the +only working database left was a leftover CI container. This contradicts a promise the toolchain +makes in two places — `meta migrate --help` (`migrate.ts:69-70`) and +[`docs/features/migrations-and-drift.md:58`](../../features/migrations-and-drift.md), which says +`apply-pending` "is the way to provision a fresh or CI database". + +## 2. What the investigation changed about the obvious design + +The obvious design — emit `IF EXISTS`, then add a gate that replays the chain and compares the +result to the committed snapshot — is wrong in a way that only shows up under review. Two findings +reshaped it. + +**The two gates are not one gate.** `migrate-ts/src/verify/replay.ts:31` already implements +`verifyReplay`: replay into a caller-supplied database, introspect, and classify drift **against the +committed snapshot**. It is exported (`index.ts:112`) and has no CLI caller. That is not an +oversight — the 2026-05-31 design §3 ruled it: *"we keep the descriptor snapshot as the generate +reference and retain replay only as the optional `verify --replay` integrity aid (§8)."* Its purpose +is catching **hand-edited structural DDL** that diverges from the snapshot. + +The reporter did not ask for that. Their failure was `table "x" does not exist` — the chain does not +**apply**. That is a strictly weaker assertion, and the difference decides who can use the gate: + +| Project class | Chain applies from empty? | Chain reproduces the snapshot, via `verifyReplay` as built? | +|---|---|---| +| Adopted via `baseline --from-db` (`migrate.ts:870` snapshots the whole introspected DB against an empty chain) | **passes trivially** — nothing to apply | **fails by construction** | +| Declares `migrate.scope` (`carryForwardOutOfScope`, `scope.ts:93`, writes the other owner's tables into the snapshot) | **passes** — the chain only creates in-scope objects | **fails** until `excludeFromSnapshot` is threaded — repairable, and §3.2 threads it | +| Uses `@schema` (`CREATE SCHEMA` is emitted nowhere but the ledger, `ledger.ts:126`) | **fails — a true positive**, fixed by §3.3 | fails | +| The reported bug | **fails — the defect** | fails | + +Two honest qualifications on that table, because an earlier draft overstated it. The scoped row is +about `verifyReplay` **as currently built**: `scopedDiffInputs` (`scope.ts:188-196`) narrows *both* +sides — out-of-scope names are merged into `unmanagedNames`, which suppresses them on the actual +side too — so under the reporter's literal formulation (replay, then diff against metadata through +the normal scoped path) a scoped project would **pass**. It fails only because `verify/replay.ts` +does not thread those inputs, which §3.2 fixes. And the `@schema` row is a true positive in both +columns, not an obstacle. + +So the durable argument for splitting the tiers is narrower than "three unpassable paths", and it is +this: **baseline adoption is unpassable against the snapshot by construction and cannot be +detected** (§3.2), while the reporter's actual failure is an *apply* error that the weaker +assertion catches directly. One tier answers the bug; the other answers a different question worth +asking. + +**Both assertions are worth having, at different strengths.** So this ships one subverb with two +tiers rather than two commands. + +## 3. Design + +### 3.1 `IF EXISTS` on forward drops, and only forward drops + +Change these to `IF EXISTS` — all verified present and bare: + +| Site | Change | +|---|---| +| `emit/postgres.ts:66` | `drop-table` | +| `emit/sqlite.ts:219` | `drop-table` | +| `emit/postgres.ts:375`, `:388` | `renderDropView`, plain and CASCADE | +| `emit/postgres.ts:93-96` | `drop-index` — **both arms**: the plain `DROP INDEX`, and the constraint-backed `ALTER TABLE … DROP CONSTRAINT`, which Postgres spells `DROP CONSTRAINT IF EXISTS` | +| `emit/sqlite.ts:225` | `drop-index` | +| `emit/postgres.ts:98`, `:104` | `drop-fk`, `drop-check` — Postgres only; see the exclusion note below for why this is not a dialect split | + +**`emit/postgres.ts:431` is deliberately NOT in this list.** An earlier draft included it. It is the +illegal-replace fallback inside `renderRestoreView`, which is reached only from `postgres.ts:178` +and `:179` — both inside the **down** renderer. Guarding it would violate the forward-only rule +stated below, in the same change that states it. + +**The rule is per change-kind, not per statement:** every `drop-table`, `drop-view` and `drop-index` +is guarded in **both** dialects. Postgres's constraint-backed index arm is included because it is +how Postgres renders the *same* `drop-index` change whose SQLite rendering is guarded at +`sqlite.ts:225` — guarding one and not the other would leave the change kind half-covered. + +**`drop-fk` and `drop-check` ARE guarded — on Postgres only — and that is not a dialect split.** +An earlier draft excluded them on the reasoning that a Postgres-only guard would make the guarantee +dialect-dependent. That reasoning was backwards. SQLite emits no standalone statement for these +kinds at all: `renderUpNative` throws (`sqlite.ts:225-235`, *"should have been handled by recreate +bundler"*), because SQLite constraints are create-time-only and inline, so the change is folded into +a table recreate. `renderRecreate` builds the replacement table from the **expected** descriptor +(`sqlite.ts:173-183`, `renderCreateTable(tmpDescriptor)`) and never references the dropped +constraint — so **SQLite is already replay-safe here by construction**. Postgres is the only dialect +that can fail on an absent constraint. Guarding it makes the two dialects *agree*. + +**`drop-column` is excluded**, and it is the one genuine dialect limit: SQLite emits it natively +(`sqlite.ts:222`, `ALTER TABLE … DROP COLUMN`) and there is no `DROP COLUMN IF EXISTS` in SQLite. +Guarding Postgres alone here really would make the same declared change behave differently per +dialect, so it stays out and §3.4's guard is what covers it. + +**One known deviation, left alone deliberately.** `sqlite.ts:275` — the `create-view` **down** — +already emits `DROP VIEW IF EXISTS`, while its Postgres twin (`postgres.ts:176`) is bare. That +predates this work and contradicts the forward-only rule below. It is out of scope: it is +view-only, changing it alters rollback behaviour, and no failure has been attributed to it. +Recorded here so a later sweep does not "discover" it as an oversight, and so the rule's one +existing exception is written down rather than remembered. + +**Down statements stay bare** (`postgres.ts:113`, `:176`, `sqlite.ts:256`). `rollbackTo` +(`apply/apply.ts:149`) runs `down.sql` and deletes the ledger row in one transaction; with +`IF EXISTS` a rollback whose object is already gone would no-op and *still* record the rollback as +done. Rollback is the one place the loud failure is load-bearing, and the replay gate never +exercises the down direction. The rule is therefore: **`IF EXISTS` on forward drops; downs +unchanged.** + +**Two forward drops stay bare deliberately** — `sqlite.ts:197` (the recreate-and-copy rebuild) and +`d1-cascade.ts:126`. Both drop a table the same recipe just `INSERT…SELECT`ed from; `IF EXISTS` +there converts a caught corruption into a silent one. This is stated so a later sweep does not +"finish the job". + +**D1 inherits this.** `emit/d1.ts:21` renders through `renderSqlite`, so D1's committed migrations +change too, while `--dialect d1` is refused by the gate in §3.2. Accepted: the emitter fix is +independently correct, and D1 keeps the `apply-pending` refusal it already has. + +### 3.2 `meta verify --replay` — the chain applies from empty + +A new subverb alongside `--templates` / `--db` / `--codegen` (ADR-0021 D2, `verify.ts:116-130`). +Opt-in; a bare `verify` never runs it. + +**`--replay` — applies.** Provision an empty database, run `applyPending` against it, assert it +completes. Nothing is compared. This is the #313 gate. + +**`--replay-snapshot`** additionally asserts the result equals the committed snapshot, via the +existing `verifyReplay`. This is the 2026-05-31 §8 integrity aid, finally wired, and it is where +`excludeFromSnapshot` (`scope.ts:130`) must be threaded so a scoped project can use it — today +`verify/replay.ts` does not thread it, though `verify.ts:659` shows the pattern. + +The second tier is a **separate subverb, not a `--strict` modifier**: `verify` already owns a +`--lax` flag on a different axis (ADR-0023 attribute strictness, `args.ts:244`, `:259`, `:302`), and +a `--strict` beside it would read as that flag's opposite rather than as a replay depth. + +**Engine, per 2026-05-31 §8's tiering** — in-process, so the gate needs no infrastructure and there +is no scratch database to name, collide with, or accidentally drop: + +- **sqlite** → `:memory:` libsql +- **postgres** → PGlite (`@electric-sql/pglite`, WASM Postgres in-process), lazily imported so it + costs nothing for projects that never run the gate +- **CI, optional higher fidelity** → real Postgres when `MIGRATE_TS_PG_URL` is set + +This replaces the sibling-scratch-database approach considered earlier. That approach needed +`CREATEDB`, broke behind connection poolers and on managed Postgres, collided between parallel CI +jobs sharing one server, and — through Postgres's 63-byte identifier truncation — could derive a +scratch name that truncates back to the target database it was about to `DROP`. + +**Refusals**, mirroring `apply-pending` (`migrate.ts:419-426`, `:449-457`): `--migration-format +flyway` and `--dialect d1`. + +**Zero committed migrations** is not a silent pass. `discoverMigrations` returns `[]` for a missing +directory (`apply.ts:316-322`), so the run would otherwise succeed having proved nothing. The gate +reports "no committed migrations — nothing to replay" and, at `--replay-snapshot` against a +non-empty snapshot, fails. + +**`--replay-snapshot` does not support baseline-adopted projects, and says so rather than +detecting it.** An earlier draft had it "skip with a reason". That cannot be implemented: the only +candidate signal is `BASELINE_NAME` / `recordBaseline` (`ledger.ts:205-227`), which has **no +production caller** — it appears only in `ledger.ts` and the package barrel — and even if it were +written, it lands in the *target* database's ledger while this gate runs against a fresh in-process +database that has no ledger at all. So the limitation is documented, not auto-detected: a project +adopted via `baseline --from-db` uses `--replay` and not `--replay-snapshot`, and the failure +message for a snapshot mismatch names baseline adoption as the first thing to rule out. + +**Exit codes** follow `verify`'s convention (`Math.max` at `verify.ts:239`): a chain that fails to +apply, or a `--replay-snapshot` mismatch, is **drift → 1**; an engine that cannot start is +**operational → 2**. + +### 3.3 `CREATE SCHEMA IF NOT EXISTS` for `@schema` projects + +The one true positive the gate surfaces is real and fixable: a chain containing +`CREATE TABLE "reporting"."x"` cannot apply to a virgin database because no migration creates the +schema. The emitter emits `CREATE SCHEMA IF NOT EXISTS "";` ahead of the first object in a +non-default schema. Without this, `@schema` projects get a red gate and no remedy. + +### 3.4 Emit-time provenance guard + +The gate catches a broken chain after it is committed. The guard stops it being written. + +When the diff proposes dropping an object that is **absent from the committed snapshot**, the object +was never managed by this toolchain — the `drop-table` in #313 is exactly this. Refuse at generation +time, naming the object, unless the drop is explicitly allowed by a new `--allow drop-unmanaged` +token. + +The population problem that sinks `--replay-snapshot` for baselined projects does not apply here, +and for a pleasing reason: +both brownfield mechanisms *add* to the snapshot. A baselined project's snapshot contains the +foreign table, so the guard reads it as managed and does not fire; a scoped project's snapshot +carries out-of-scope entries forward for the same reason. The guard fires precisely when nothing +ever claimed the object — which is the reported case. + +`classify.ts:6-9` already states the doctrine this extends: objects present in the DB but not the +snapshot "must never be treated as actionable drift or auto-dropped". The live path +(`migrate.ts:607-620`) compares metadata against introspection and never consults the snapshot, +which is why the doctrine was not enforced where it mattered. + +### 3.5 Documentation + +`docs/features/migrations-and-drift.md:58` and `meta migrate --help` both promise fresh-database +provisioning. §3.2 makes the promise true for projects whose chain builds the schema; the docs must +say that it is *those* projects, and point at `verify --replay` as the way to know you are one. + +## 4. Remediation for a chain that is already broken + +Applied migrations are checksum-immutable (`apply/apply.ts:88-99`): hand-editing a committed +`up.sql` to add `IF EXISTS` is rejected on any database that already applied it. So the reporter — +and anyone the new gate turns red — cannot fix history in place. + +The supported path is a **compensating migration**: author a new migration that creates the missing +object as the chain expects, or that supersedes the bad drop. The gate's failure message must print +this, with the failing statement and the object name. **A gate whose failure has no documented exit +gets suppressed**, and this one would otherwise go red on exactly the population that asked for it. + +## 5. Testing + +- **§3.1** — emit assertions per dialect and per direction, including explicit assertions that the + rebuild-path drops and the down statements remain bare, so the deliberate exclusions are pinned + rather than remembered. +- **§3.2** — the reporter's scenario as a RED-first regression: a chain containing a drop for a + table it never creates, replayed from empty. It must run **through `applyPending`**, not through + `emit()`. Every prior defect in this area (#226/#241, #243, #255, #285, and 0.21.4's + `BEGIN TRANSACTION` finding) shared one shape — SQL proven statement-by-statement and never proven + through the tool that applies it — and `runSqlFileWithLedgerMutation` (`apply.ts:298`) rewrites + statements before execution. +- **Each row of §2's table** as a case: baselined, scoped, `@schema`, and the reported bug. +- **§3.4** — a drop for a snapshot-absent object refuses; the same drop with `--allow drop-unmanaged` + proceeds; a baselined project does not false-fire. +- The Postgres lane is `MIGRATE_TS_PG_URL`-gated and `describe.skip`s when unset + (`apply-pg.test.ts:21-22`); `pg-gate-sentinel.test.ts` exists because that lane silently rotted red + for eight releases. New Postgres cases must run on PGlite by default so they execute everywhere, + with the real-PG lane as the higher-fidelity tier. + +## 6. Non-goals + +- `drop-column` `IF EXISTS` (§3.1's one genuine dialect limit; §3.4's guard covers it instead). +- `sqlite.ts:275`'s pre-existing `IF EXISTS` on a `create-view` down (§3.1). +- Repairing already-applied chains automatically (§4). +- Any cross-port work: `migrate` is TS-owned. +- Making `--replay-snapshot` pass for baselined projects, or auto-detecting them. Neither is + possible with the signals that exist (§3.2); the limitation is documented instead. + +## 7. Verified by + +Every claim above was re-read at `286d50e6e` before writing: the bare-drop sites, the four existing +`IF EXISTS` sites, `verifyReplay` and its absent CLI caller, the 2026-05-31 §3 and §8 rulings, +`baseline --from-db`'s snapshot, `carryForwardOutOfScope`'s call sites, `CREATE SCHEMA`'s two +ledger-only occurrences, `discoverMigrations`'s empty-directory behaviour, `emit/d1.ts:21`, and the +env-gated Postgres lane. diff --git a/fixtures/source-resolution-conformance/README.md b/fixtures/source-resolution-conformance/README.md new file mode 100644 index 000000000..2ff6be920 --- /dev/null +++ b/fixtures/source-resolution-conformance/README.md @@ -0,0 +1,158 @@ +# source-resolution-conformance + +Pins how a consumer's `sources` set in `.metaobjects/config.json` resolves to a +set of metadata files. Every port's CLI must resolve the SAME FILES from the +same declaration — that is the cross-port promise this corpus exists to keep. + +Companion to `scope-conformance/`, which pins the (currently TypeScript-only) +`scope` pattern grammar. The two are independent: `sources` decides which files +are read, `scope` filters what is emitted from them. + +## Shape + +``` +cases.json # { cases: [{ name, tree, symlinks?, config, resolveFrom?, expectFiles?, expectError? }] } +README.md +``` + +- **`tree`** — a map of project-root-relative path → file content. The runner + materializes it in a fresh temporary directory. A `.keep` entry exists only to + force an otherwise-empty directory to be created. +- **`symlinks`** — OPTIONAL, a map of project-root-relative `linkPath` → + project-root-relative `targetPath`. Materialized AFTER `tree` (so the target + already exists), as a directory symlink at `linkPath` pointing at `targetPath`. + Exists to gate that every port's directory walk FOLLOWS a symlinked directory + — whether the `sources` path itself is a symlink + (`a-symlinked-sources-path-resolves-through-it`) or a symlink sits partway + through a walked tree (`a-symlinked-subdirectory-inside-a-walked-tree- + resolves-through-it`) — rather than silently resolving to zero files or + skipping the subtree. The reported path preserves the symlink's OWN name (a + file reached through `link/` is reported as `link/…`, never resolved to + `real/…`) — see "Order is deliberately NOT pinned" below for the parallel + point about `expectFiles` being exact strings, not just "the same underlying + file by any name". Every port's runner must honor this key. +- **`config`** — written verbatim to `.metaobjects/config.json`, under the + directory named by `resolveFrom` (project root when `resolveFrom` is absent). + When `null`, no config file is created at all. +- **`resolveFrom`** — OPTIONAL, a project-root-relative directory path, + default `"."`. Names the directory the resolver is invoked against — i.e. + the directory treated as holding `.metaobjects/`. Exists so a case can prove + a relative `path` source resolves against the *declaring config's* + directory rather than the project root or the process's ambient working + directory: put `resolveFrom` somewhere other than `"."` and a `path` source + containing `../` only lands on the right files if the port under test + resolved it against the right base. Every port's runner must honor this key + — it is part of the case schema from the start rather than retrofitted once + three ports already exist. +- **`expectFiles`** — project-root-relative paths (NOT relative to + `resolveFrom`), compared as an **UNORDERED SET**. See "Order is + deliberately not pinned" below. +- **`expectError`** — either a STRING error code the resolution must fail with + exactly, or the literal `true` meaning "must raise, but which error/code is + deliberately not pinned across ports" (see "Also deliberately NOT pinned: + the malformed-config error code" below for why the latter form exists). + Exactly one of `expectFiles` / `expectError` is present per case. + +## Semantics pinned here + +- **Default.** `sources` absent or empty ⇒ exactly one `path` source, the literal + `metaobjects`. It is a default VALUE, never a requirement. +- **Replacement, not merge.** A declared `sources` replaces the default entirely — + the default directory is not implicitly appended. +- **Relative base.** A relative `path` resolves against the directory HOLDING the + `.metaobjects/` folder, never against the process working directory. See + `a-parent-relative-path-resolves-against-the-declaring-configs-directory`, + which uses `resolveFrom` to invoke resolution from a subdirectory while the + config's own `path` source climbs back out with `../` — the case only + passes when a port resolves relative to the config's directory, not to + wherever the process happened to be started. +- **Recursion.** A directory `path` is walked recursively; a file `path` resolves + to that one file. +- **Extensions.** `.json`, `.yaml`, `.yml`, matched case-insensitively. Nothing else. + See `metadata-extensions-are-matched-case-insensitively`, which mixes + `.JSON`/`.YAML`/`.Yml` spellings alongside a normal lowercase file AND a + same-family unsupported extension in uppercase (`.TXT`) — a port that only + lowercases `.json` (missing `.yaml`/`.yml`), or that folds case and then + matches too loosely (accepting any extension once folded), diverges from + this case either way. +- **Union with de-duplication.** Overlapping sources yield each file exactly once. +- **A declared path that does not exist is `ERR_SOURCE_UNRESOLVED`** — never a + silent skip. Only the DEFAULT may be absent, and then it is + `ERR_COLLECTION_NOT_FOUND`. +- **`resource` and `package` kinds are declared but resolve nowhere yet:** + `ERR_SOURCE_KIND_UNSUPPORTED`. +- **Kind validation precedes path resolution, and that precedence is + order-independent.** Every spec's kind is checked across the WHOLE + declared list before any spec's path is touched on disk — so a multi-spec + list containing both an unsupported kind and a path that does not exist + always fails `ERR_SOURCE_KIND_UNSUPPORTED`, in EITHER declaration order, + never `ERR_SOURCE_UNRESOLVED`. See + `unsupported-kind-precedes-unresolved-path-when-path-is-declared-first` + and its `-declared-second` sibling — a port that interleaves the two + checks one spec at a time (kind-check-then-stat, per spec, in a single + loop) reports whichever error comes first in declaration order instead, + diverging on exactly one of the two cases depending on which order it + happens to process first. +- **Unknown top-level config keys are IGNORED.** The file carries + TypeScript-owned keys no other port models. `schema_version` and `sources` are + the neutral subset; each port validates those strictly and ignores the rest. + +## Order is deliberately NOT pinned + +`expectFiles` is a set. The ports' directory walks already differ and always +have — Java sorts by basename (`DirectorySource.java:105`), C# by full-path +ordinal (`DirectorySource.cs:64`), Python by basename +(`directory_source.py:40-48`), TypeScript walks depth-first with files before +subdirectories (`metadata-files.ts:101-121`). Making order a contract would be a +behavior change in three ports for no benefit: super-resolution is +order-independent (#188) and the loader's overlay partition discards caller +order anyway. + +A port MAY have a stable internal order — several do, and their own generated +output depends on it. It just is not a cross-port promise. + +## Also deliberately NOT pinned: the malformed-config error code + +`sources-must-be-an-array-not-an-object` pins that a `.metaobjects/config.json` +that exists but is malformed (here: `sources` declared as a bare object instead +of an array) MUST raise rather than silently degrade to "no config" — the +regression this case exists to catch is a wrong-typed `sources` reading as +absent, falling back to the default `metaobjects/` directory with no +diagnostic at all, even when a stale file sits there. It uses `"expectError": +true` rather than a string code, because which code it raises with is left to +each port. The reference implementation is why: `collection.ts:129-140` has no +try/catch around config loading and lets the raw zod/JSON error propagate, so +TypeScript throws a `ZodError` carrying no MetaObjects error code at all. +Pinning a shared code across ports would mean changing the reference, which +this corpus does not do. Python raises `ERR_COLLECTION_NOT_FOUND`; C# and Java +both raise `ERR_BAD_ATTR_VALUE` — three distinct outcomes across four ports, +which is exactly why this case checks only that resolution raises, never with +which error, same as file order above. The same `true` form covers five more +cases beyond this one: an unsupported `schema_version` +(`an-unsupported-schema-version-is-an-error`) and four malformed `sources` +entry shapes (`sources-null-is-an-error-not-the-default`, +`an-empty-path-is-an-error`, `a-sources-entry-with-two-keys-is-an-error`, +`a-non-string-source-value-is-an-error`). Genuinely malformed JSON SYNTAX (an +unparseable `.metaobjects/config.json`) is still not in the corpus — the case +schema's `config` field is always a valid JSON value that the runner +re-serializes via `JSON.stringify`/equivalent, so expressing broken syntax +would need a schema extension carrying raw file text instead of a config +object. Nothing about the `true`-sentinel mechanism is specific to any of +these shapes. + +## Behavioral contract + +Each port's runner reads `cases.json`, and for every case: materializes `tree` +in a fresh temp directory, then `symlinks` (when present), writes `config` +when non-null under the directory named by `resolveFrom` (default the project +root), resolves sources against that directory, then asserts either that the +resolved file set equals `expectFiles` (as a set, project-root-relative, path +separators normalized to `/`) — AND that its size matches the raw resolved +count, since a Set/HashSet comparison alone cannot see a duplicate emission +collapse invisibly into one set element — or that resolution failed with +`expectError`. + +## Reference implementation + +`server/typescript/packages/sdk/src/sources.ts` (`resolveSources`) and +`server/typescript/packages/sdk/src/collection.ts` (`resolveCollection`). diff --git a/fixtures/source-resolution-conformance/cases.json b/fixtures/source-resolution-conformance/cases.json new file mode 100644 index 000000000..4ed9faafc --- /dev/null +++ b/fixtures/source-resolution-conformance/cases.json @@ -0,0 +1,269 @@ +{ + "cases": [ + { + "name": "no-config-uses-default-directory", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": null, + "expectFiles": ["metaobjects/meta.users.json"] + }, + { + "name": "empty-sources-uses-default-directory", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [] }, + "expectFiles": ["metaobjects/meta.users.json"] + }, + { + "name": "declared-path-replaces-the-default-entirely", + "tree": { + "metaobjects/ignored.json": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.orders.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.orders.json"] + }, + { + "name": "directory-is-walked-recursively", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/nested/meta.b.yaml": "metadata.root:\n children: []\n", + "model/nested/deep/meta.c.yml": "metadata.root:\n children: []\n" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [ + "model/meta.a.json", + "model/nested/meta.b.yaml", + "model/nested/deep/meta.c.yml" + ] + }, + { + "name": "non-metadata-extensions-are-ignored", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/README.md": "not metadata", + "model/notes.txt": "not metadata", + "model/script.ts": "export {}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.a.json"] + }, + { + "name": "metadata-extensions-are-matched-case-insensitively", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.b.JSON": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.c.YAML": "metadata.root:\n children: []\n", + "model/meta.d.Yml": "metadata.root:\n children: []\n", + "model/notes.TXT": "not metadata" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [ + "model/meta.a.json", + "model/meta.b.JSON", + "model/meta.c.YAML", + "model/meta.d.Yml" + ] + }, + { + "name": "a-single-file-path-resolves-to-that-file", + "tree": { + "vendor/meta.catalog.json": "{\"metadata.root\":{\"children\":[]}}", + "vendor/meta.other.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "vendor/meta.catalog.json" }] }, + "expectFiles": ["vendor/meta.catalog.json"] + }, + { + "name": "two-sources-union", + "tree": { + "a/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "b/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "a" }, { "path": "b" }] }, + "expectFiles": ["a/meta.a.json", "b/meta.b.json"] + }, + { + "name": "overlapping-sources-yield-each-file-once", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "model/nested/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "model" }, { "path": "model/nested" }] + }, + "expectFiles": ["model/meta.a.json", "model/nested/meta.b.json"] + }, + { + "name": "source-order-does-not-change-the-resolved-set", + "tree": { + "a/meta.a.json": "{\"metadata.root\":{\"children\":[]}}", + "b/meta.b.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "b" }, { "path": "a" }] }, + "expectFiles": ["a/meta.a.json", "b/meta.b.json"] + }, + { + "name": "a-parent-relative-path-resolves-against-the-declaring-configs-directory", + "tree": { + "shared/meta.shared.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "../shared" }] }, + "resolveFrom": "app", + "expectFiles": ["shared/meta.shared.json"] + }, + { + "name": "a-declared-path-that-does-not-exist-is-an-error", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "nope" }] }, + "expectError": "ERR_SOURCE_UNRESOLVED" + }, + { + "name": "an-empty-directory-source-resolves-to-no-files", + "tree": { + "model/.keep": "" + }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": [] + }, + { + "name": "resource-kind-is-unsupported", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "resource": "com/acme/model" }] }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "package-kind-is-unsupported", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "package": "acme-model" }] }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "unsupported-kind-precedes-unresolved-path-when-path-is-declared-first", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "nope" }, { "resource": "com/acme/model" }] + }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "unsupported-kind-precedes-unresolved-path-when-path-is-declared-second", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "resource": "com/acme/model" }, { "path": "nope" }] + }, + "expectError": "ERR_SOURCE_KIND_UNSUPPORTED" + }, + { + "name": "no-config-and-no-default-directory-is-an-error", + "tree": { + "src/.keep": "" + }, + "config": null, + "expectError": "ERR_COLLECTION_NOT_FOUND" + }, + { + "name": "unknown-top-level-keys-are-ignored", + "tree": { + "model/meta.a.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "model" }], + "pending_in_git": true, + "confidence_thresholds": { "pending_promote": 0.8 }, + "extract": { "metaignore": ".metaignore" }, + "migrate": { "dialect": "postgres" } + }, + "expectFiles": ["model/meta.a.json"] + }, + { + "name": "sources-must-be-an-array-not-an-object", + "tree": { + "metaobjects/meta.stale.json": "{\"metadata.root\":{\"children\":[]}}", + "custom/meta.real.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": { "path": "custom" } }, + "expectError": true + }, + { + "name": "sources-null-is-an-error-not-the-default", + "tree": { + "metaobjects/meta.stale.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": null }, + "expectError": true + }, + { + "name": "an-empty-path-is-an-error", + "tree": { + "metaobjects/meta.users.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": "" }] }, + "expectError": true + }, + { + "name": "a-sources-entry-with-two-keys-is-an-error", + "tree": { + "metaobjects/meta.stale.json": "{\"metadata.root\":{\"children\":[]}}", + "custom/meta.real.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { + "schema_version": 1, + "sources": [{ "path": "custom", "resource": "com/acme/model" }] + }, + "expectError": true + }, + { + "name": "an-unsupported-schema-version-is-an-error", + "tree": { + "metaobjects/meta.stale.json": "{\"metadata.root\":{\"children\":[]}}", + "model/meta.real.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 2, "sources": [{ "path": "model" }] }, + "expectError": true + }, + { + "name": "a-non-string-source-value-is-an-error", + "tree": { + "123/meta.real.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "config": { "schema_version": 1, "sources": [{ "path": 123 }] }, + "expectError": true + }, + { + "name": "a-symlinked-sources-path-resolves-through-it", + "tree": { + "real/meta.a.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "symlinks": { "link": "real" }, + "config": { "schema_version": 1, "sources": [{ "path": "link" }] }, + "expectFiles": ["link/meta.a.json"] + }, + { + "name": "a-symlinked-subdirectory-inside-a-walked-tree-resolves-through-it", + "tree": { + "model/meta.top.json": "{\"metadata.root\":{\"children\":[]}}", + "external/meta.linked.json": "{\"metadata.root\":{\"children\":[]}}" + }, + "symlinks": { "model/linked": "external" }, + "config": { "schema_version": 1, "sources": [{ "path": "model" }] }, + "expectFiles": ["model/meta.top.json", "model/linked/meta.linked.json"] + } + ] +} diff --git a/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs b/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs new file mode 100644 index 000000000..e56667c1b --- /dev/null +++ b/server/csharp/MetaObjects.Cli.Tests/MetadataDirFallbackTests.cs @@ -0,0 +1,196 @@ +using System.Diagnostics; +using Xunit; + +namespace MetaObjects.Cli.Tests; + +///

+/// Proves the <metadataDir>-optional fallback (Program.cs's +/// ResolveMetadataDirOrExit) is actually WIRED into dotnet meta gen, +/// not merely defined and unit-tested in isolation. A resolver function that +/// nothing calls is exactly the gap the Python port shipped first (see the +/// SourceResolutionConformanceTests header + task-4-report.md) — the only way to +/// catch that class of bug is to drive the real command entry point (the built +/// CLI assembly, invoked as a subprocess) rather than calling a helper method +/// directly, since a top-level-statement local function compiles to a name this +/// test assembly cannot reference at all. +/// +public sealed class MetadataDirFallbackTests : IDisposable +{ + private readonly string _tmp = Path.Combine(Path.GetTempPath(), "meta-cli-fallback-" + Guid.NewGuid().ToString("N")); + + private const string Metadata = """ + { "metadata.root": { "package": "acme", "children": [ + { "object.entity": { "name": "Subscriber", "children": [ + { "source.rdb": { "@table": "subscribers" } }, + { "field.long": { "name": "id" } }, + { "field.string": { "name": "email", "@required": true } }, + { "identity.primary": { "@fields": "id" } } + ]}} + ]}} + """; + + public void Dispose() { try { Directory.Delete(_tmp, recursive: true); } catch { } } + + [Fact] + public void Gen_with_no_positional_metadataDir_resolves_the_declared_source_and_generates() + { + var modelDir = Path.Combine(_tmp, "model"); + Directory.CreateDirectory(modelDir); + File.WriteAllText(Path.Combine(modelDir, "meta.acme.json"), Metadata); + var cfgDir = Path.Combine(_tmp, ".metaobjects"); + Directory.CreateDirectory(cfgDir); + File.WriteAllText( + Path.Combine(cfgDir, "config.json"), + """{ "schema_version": 1, "sources": [ { "path": "model" } ] }"""); + + var outDir = Path.Combine(_tmp, "generated"); + var (exitCode, stdout, stderr) = RunCli(_tmp, "gen", "--out", outDir, "--namespace", "Acme.Generated"); + + Assert.True(exitCode == 0, $"exit={exitCode}\nstdout={stdout}\nstderr={stderr}"); + Assert.True(File.Exists(Path.Combine(outDir, "Subscriber.g.cs")), stdout + stderr); + } + + [Fact] + public void Gen_with_no_positional_metadataDir_and_nothing_to_resolve_reports_the_ladders_own_error() + { + // Before the ladder was wired, an omitted positional always produced the + // generic "usage: ..." 2-exit, regardless of what (if anything) the cwd + // contained. Once wired, an empty project reaches SourceResolver and + // fails with ITS OWN diagnostic — proof the omitted case is actually + // being routed through resolution rather than still failing the old way. + Directory.CreateDirectory(_tmp); + + var (exitCode, _, stderr) = RunCli(_tmp, "gen", "--out", Path.Combine(_tmp, "generated"), "--namespace", "X"); + + Assert.Equal(2, exitCode); + Assert.Contains("ERR_COLLECTION_NOT_FOUND", stderr); + } + + [Fact] + public void Gen_with_no_outDir_and_nothing_to_resolve_prints_usage_not_the_ladders_error() + { + // Usage-first: a missing --out is a plain CLI-usage mistake, independent of + // whether metadata can be found. Before the fix, ResolveMetadataDirOrExit ran + // FIRST and this empty project (no --out either) exited 2 with the ladder's + // own ERR_COLLECTION_NOT_FOUND instead of the actionable "usage: ..." line — + // confusing on the common first-run case where both are missing at once. + Directory.CreateDirectory(_tmp); + + var (exitCode, _, stderr) = RunCli(_tmp, "gen"); + + Assert.Equal(2, exitCode); + Assert.Contains("usage: dotnet meta gen", stderr); + Assert.DoesNotContain("ERR_COLLECTION_NOT_FOUND", stderr); + } + + [Fact] + public void Gen_with_no_positional_metadataDir_and_multiple_declared_sources_refuses_rather_than_picking_one() + { + var aDir = Path.Combine(_tmp, "a"); + var bDir = Path.Combine(_tmp, "b"); + Directory.CreateDirectory(aDir); + Directory.CreateDirectory(bDir); + File.WriteAllText(Path.Combine(aDir, "meta.a.json"), """{ "metadata.root": { "children": [] } }"""); + File.WriteAllText(Path.Combine(bDir, "meta.b.json"), """{ "metadata.root": { "children": [] } }"""); + var cfgDir = Path.Combine(_tmp, ".metaobjects"); + Directory.CreateDirectory(cfgDir); + File.WriteAllText( + Path.Combine(cfgDir, "config.json"), + """{ "schema_version": 1, "sources": [ { "path": "a" }, { "path": "b" } ] }"""); + + var outDir = Path.Combine(_tmp, "generated"); + var (exitCode, _, stderr) = RunCli(_tmp, "gen", "--out", outDir, "--namespace", "X"); + + Assert.Equal(2, exitCode); + Assert.Contains("2 metadata sources", stderr); + Assert.False(Directory.Exists(outDir), "must not silently generate from just one of several declared sources"); + } + + [Fact] + public void Gen_with_no_positional_metadataDir_and_a_single_FILE_source_refuses_clearly() + { + // Before this fix, a single declared `path` source resolving to a FILE + // (rather than a directory) was handed straight to MetaDataLoader.FromDirectory, + // which fails deep inside DirectorySource with an opaque, uncoded ERR_UNKNOWN — + // never naming the actual limit (this CLI's loader takes only a directory). + var vendorDir = Path.Combine(_tmp, "vendor"); + Directory.CreateDirectory(vendorDir); + File.WriteAllText(Path.Combine(vendorDir, "meta.catalog.json"), """{ "metadata.root": { "children": [] } }"""); + var cfgDir = Path.Combine(_tmp, ".metaobjects"); + Directory.CreateDirectory(cfgDir); + File.WriteAllText( + Path.Combine(cfgDir, "config.json"), + """{ "schema_version": 1, "sources": [ { "path": "vendor/meta.catalog.json" } ] }"""); + + var outDir = Path.Combine(_tmp, "generated"); + var (exitCode, _, stderr) = RunCli(_tmp, "gen", "--out", outDir, "--namespace", "X"); + + Assert.Equal(2, exitCode); + Assert.Contains("is a FILE", stderr); + Assert.DoesNotContain("ERR_UNKNOWN", stderr); + Assert.False(Directory.Exists(outDir)); + } + + [Fact] + public void Gen_with_an_explicit_positional_metadataDir_is_unaffected() + { + // The explicit-argument path must stay byte-identical: no .metaobjects/ + // config.json in play at all, yet generation still succeeds because the + // ladder is never consulted when the caller already named a directory. + var modelDir = Path.Combine(_tmp, "model"); + Directory.CreateDirectory(modelDir); + File.WriteAllText(Path.Combine(modelDir, "meta.acme.json"), Metadata); + + var outDir = Path.Combine(_tmp, "generated"); + var (exitCode, stdout, stderr) = RunCli(_tmp, "gen", modelDir, "--out", outDir, "--namespace", "Acme.Generated"); + + Assert.True(exitCode == 0, $"exit={exitCode}\nstdout={stdout}\nstderr={stderr}"); + Assert.True(File.Exists(Path.Combine(outDir, "Subscriber.g.cs")), stdout + stderr); + } + + /// Runs the actual built `dotnet meta` assembly as a subprocess, cwd + /// pinned to , so the test exercises Program.cs's + /// real Main/argument-parsing rather than any method reachable in-process. + private static (int ExitCode, string Stdout, string Stderr) RunCli(string workingDir, params string[] args) + { + var psi = new ProcessStartInfo("dotnet") + { + WorkingDirectory = workingDir, + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + }; + psi.ArgumentList.Add(ResolveCliDll()); + foreach (var a in args) psi.ArgumentList.Add(a); + + using var proc = Process.Start(psi) ?? throw new InvalidOperationException("failed to start dotnet"); + var stdout = proc.StandardOutput.ReadToEnd(); + var stderr = proc.StandardError.ReadToEnd(); + proc.WaitForExit(); + return (proc.ExitCode, stdout, stderr); + } + + /// Locates the MetaObjects.Cli build output next to this test + /// assembly's own build output — both projects share the same Configuration + /// and TargetFramework (net8.0), and MetaObjects.Cli.Tests already builds + /// MetaObjects.Cli as a project reference, so the dll is guaranteed present + /// by the time `dotnet test` starts running tests. + private static string ResolveCliDll() + { + var testsProjectDir = new DirectoryInfo(AppContext.BaseDirectory); + while (testsProjectDir is not null && + !File.Exists(Path.Combine(testsProjectDir.FullName, "MetaObjects.Cli.Tests.csproj"))) + testsProjectDir = testsProjectDir.Parent; + if (testsProjectDir is null) + throw new InvalidOperationException( + "could not locate MetaObjects.Cli.Tests.csproj by walking up from " + AppContext.BaseDirectory); + + var relSuffix = Path.GetRelativePath(testsProjectDir.FullName, AppContext.BaseDirectory); + var dll = Path.Combine(testsProjectDir.Parent!.FullName, "MetaObjects.Cli", relSuffix, "MetaObjects.Cli.dll"); + if (!File.Exists(dll)) + throw new FileNotFoundException( + $"expected the MetaObjects.Cli build output at {dll} (built automatically as a project " + + "reference of MetaObjects.Cli.Tests)", dll); + return dll; + } +} diff --git a/server/csharp/MetaObjects.Cli/Program.cs b/server/csharp/MetaObjects.Cli/Program.cs index bad143a68..fa6a457e8 100644 --- a/server/csharp/MetaObjects.Cli/Program.cs +++ b/server/csharp/MetaObjects.Cli/Program.cs @@ -73,13 +73,23 @@ static int RunGen(string[] rest) return 0; } - if (metadataDir is null || outDir is null) + // Usage-first: a missing --out is a plain CLI-usage error, unconditional on + // whether metadata can be found — it must win over ResolveMetadataDirOrExit + // below, which can itself terminate the process with an unrelated + // ERR_COLLECTION_NOT_FOUND. Checking outDir after resolution would show that + // confusing error instead of this actionable usage line on the (common) + // first-run case where BOTH are missing. + if (outDir is null) { Console.Error.WriteLine("usage: dotnet meta gen --out [--namespace ] [--generators ] [--template-root ] [--template-spec ] [--emit-abstract-shapes]"); Console.Error.WriteLine(" dotnet meta gen --list"); return 2; } + // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir + // falls back to the port-neutral .metaobjects/config.json ladder. + metadataDir = ResolveMetadataDirOrExit(metadataDir); + // Advisory: nudge a re-scaffold if the copied-in agent context predates this build. // Never throws, never changes the exit code (a missing/corrupt manifest is ignored). AgentContextStalenessCheck.WarnIfStale(Directory.GetCurrentDirectory()); @@ -121,12 +131,18 @@ static int RunDocs(string[] rest) else if (!rest[i].StartsWith('-')) metadataDir ??= rest[i]; } - if (metadataDir is null || outDir is null) + // Usage-first — see the identical comment in RunGen above; a missing --out + // must win over ResolveMetadataDirOrExit's own possible ERR_COLLECTION_NOT_FOUND. + if (outDir is null) { Console.Error.WriteLine("usage: dotnet meta docs --out [--namespace ] [--project ] [--model-base-url ]"); return 2; } + // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir + // falls back to the port-neutral .metaobjects/config.json ladder. + metadataDir = ResolveMetadataDirOrExit(metadataDir); + // Default the project label to the input directory's leaf name (cosmetic — surfaces // in the AGENT-API header). Trailing-separator-safe. project ??= new DirectoryInfo(Path.TrimEndingDirectorySeparator(Path.GetFullPath(metadataDir))).Name; @@ -149,6 +165,86 @@ static int RunDocs(string[] rest) return 0; } +// The metadata-location ladder's rungs 3-4 (source-resolution design doc §3): +// rung 1 is the explicit positional argument the caller already tried; rung 2 +// (a port-native config surface) doesn't exist in C#; rungs 3 (a declared +// `sources` in .metaobjects/config.json) and 4 (the default "metaobjects" +// directory) live in MetaObjects.Config.SourceResolver.ResolveCollection, +// which this wraps. Called from all three metadataDir-taking commands (gen, +// docs, verify) so an omitted positional argument is never a hard requirement +// wherever a project's config can name the location instead. +// +// Never returns null: either hands back a real directory, or prints a +// diagnostic and terminates the process — callers may treat the result as +// always-present and keep their existing (now-unreachable-when-omitted) +// null checks for the OTHER positional/option they still require. +static string ResolveMetadataDirOrExit(string? metadataDir) +{ + if (metadataDir is not null) return metadataDir; + + var cwd = Directory.GetCurrentDirectory(); + try + { + var cfg = MetaObjects.Config.NeutralConfig.Read(cwd); + var specs = cfg?.Sources ?? Array.Empty>(); + + if (specs.Count == 0) + { + // No declared sources — validate + apply the DEFAULT directory through + // the same ladder the shared conformance corpus gates (raises + // ERR_COLLECTION_NOT_FOUND when the default is also absent). + _ = MetaObjects.Config.SourceResolver.ResolveCollection(cwd); + return Path.Combine(cwd, MetaObjects.Config.NeutralConfig.DefaultMetadataDir); + } + + if (specs.Count > 1) + { + // MetaDataLoader.FromDirectory takes ONE directory — it cannot express a + // multi-source SET. Fail loudly rather than silently loading just one of + // the declared sources; MetaDataLoader.Load(IReadOnlyList) + // (MetaDataLoader.cs:334) is the documented follow-up that lifts this. + Console.Error.WriteLine( + $"error: {cwd}: .metaobjects/config.json declares {specs.Count} metadata sources, but " + + "this CLI's loader accepts only one directory at a time. Pass explicitly, " + + "or reduce \"sources\" to a single entry."); + Environment.Exit(2); + throw new InvalidOperationException("unreachable"); + } + + // Exactly one declared source. Resolve + validate it through the same + // kind/existence checks ResolveSources applies (ERR_SOURCE_KIND_UNSUPPORTED / + // ERR_SOURCE_UNRESOLVED), then hand the loader that spec's OWN root — never + // the default directory name, which this project may not even have. + MetaObjects.Config.SourceResolver.ResolveSources(cwd, specs); + var rawPath = specs[0]["path"]; // guaranteed present: ResolveSources above + // would already have thrown otherwise. + var resolved = Path.IsPathRooted(rawPath) ? rawPath : Path.GetFullPath(Path.Combine(cwd, rawPath)); + + if (!Directory.Exists(resolved)) + { + // ResolveSources above already proved `resolved` exists, so this means + // it is a FILE. MetaDataLoader.FromDirectory below takes a directory — + // handing it a file path used to fail deep inside DirectorySource with + // an opaque ERR_UNKNOWN instead of naming the actual limit. Refuse + // clearly here instead, the same way the multi-source branch above does. + Console.Error.WriteLine( + $"error: {cwd}: .metaobjects/config.json's single \"sources\" entry (\"{rawPath}\") is a FILE, " + + "but this CLI's loader only accepts a directory source. Pass explicitly, or point " + + "\"sources\" at the file's containing directory."); + Environment.Exit(2); + throw new InvalidOperationException("unreachable"); + } + + return resolved; + } + catch (MetaObjects.MetaModelException e) + { + Console.Error.WriteLine($"error: {e.Code}: {e.Message}"); + Environment.Exit(2); + throw; + } +} + static int Unknown(string cmd) { Console.Error.WriteLine($"dotnet meta: unknown command \"{cmd}\""); @@ -208,11 +304,9 @@ static int RunVerify(string[] rest) else templatesRoot ??= a; } - if (metadataDir is null) - { - Console.Error.WriteLine("usage: dotnet meta verify [--templates ] [--codegen --out [--namespace ]] [--db]"); - return 2; - } + // Rung 1 (explicit positional) is honored as-is; an omitted metadataDir + // falls back to the port-neutral .metaobjects/config.json ladder. + metadataDir = ResolveMetadataDirOrExit(metadataDir); // The templates gate needs a root. Bare verify (defaults to templates) and an // explicit --templates both require it; surface a clear usage error if absent. diff --git a/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs index 2a6467697..0dee418dd 100644 --- a/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs +++ b/server/csharp/MetaObjects.Conformance.Tests/DirectorySourceTests.cs @@ -34,6 +34,63 @@ public void Expand_ReturnsFileSourcesSortedByOrdinalName() } } + [Fact] + public void Expand_ExcludePending_IsOffByDefault() + { + // Loader-level default is OFF (matches TS's loader-level DirectorySource, + // which has no _pending concept at all) — only the CLI-facing + // SourceResolver turns it on. An app embedding `new DirectorySource(dir)` + // directly must see every file, _pending/ included. + string dir = Path.Combine(Path.GetTempPath(), "ds_" + Path.GetRandomFileName()); + Directory.CreateDirectory(dir); + try + { + File.WriteAllText(Path.Combine(dir, "meta.live.json"), "{}"); + Directory.CreateDirectory(Path.Combine(dir, "_pending")); + File.WriteAllText(Path.Combine(dir, "_pending", "meta.draft.json"), "{}"); + + var src = new DirectorySource(dir); + var expanded = src.Expand().ToList(); + + Assert.Equal(2, expanded.Count); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + + [Fact] + public void Expand_Excludes_PendingDir_AtAnyDepth_WhenOptedIn() + { + // Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft + // entity under _pending/ must be invisible to codegen, not merely a file + // that happens to be NAMED "_pending". SourceResolver (the CLI-facing + // caller) opts in via ExcludePending = true; this test exercises the + // option directly. + string dir = Path.Combine(Path.GetTempPath(), "ds_" + Path.GetRandomFileName()); + Directory.CreateDirectory(dir); + try + { + File.WriteAllText(Path.Combine(dir, "meta.live.json"), "{}"); + Directory.CreateDirectory(Path.Combine(dir, "_pending")); + File.WriteAllText(Path.Combine(dir, "_pending", "meta.draft.json"), "{}"); + // Nested: _pending/ excluded at ANY depth, not just top-level. + Directory.CreateDirectory(Path.Combine(dir, "nested", "_pending")); + File.WriteAllText(Path.Combine(dir, "nested", "_pending", "meta.deep-draft.json"), "{}"); + + var src = new DirectorySource(dir, new DirectorySource.Options { ExcludePending = true }); + var expanded = src.Expand().ToList(); + + Assert.Single(expanded); + Assert.Equal("meta.live.json", expanded[0].Id); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + [Fact] public void Expand_HonorsExcludeGlobs() { diff --git a/server/csharp/MetaObjects.Conformance.Tests/NeutralConfigTests.cs b/server/csharp/MetaObjects.Conformance.Tests/NeutralConfigTests.cs new file mode 100644 index 000000000..81823c55e --- /dev/null +++ b/server/csharp/MetaObjects.Conformance.Tests/NeutralConfigTests.cs @@ -0,0 +1,77 @@ +using System.IO; +using MetaObjects.Config; +using Xunit; + +namespace MetaObjects.Conformance.Tests; + +/// +/// Focused unit coverage for on shapes that are +/// NOT gated by the shared source-resolution-conformance corpus — either +/// because the reference TypeScript implementation deliberately behaves +/// differently (whitespace-only path) or to pin a specific prior defect +/// directly against this port's own code (the schema_version float parse). +/// +public sealed class NeutralConfigTests +{ + private static string WriteConfig(string payloadJson) + { + var dir = Path.Combine(Path.GetTempPath(), "neutral-config-" + System.Guid.NewGuid().ToString("N")); + var cfgDir = Path.Combine(dir, ".metaobjects"); + Directory.CreateDirectory(cfgDir); + File.WriteAllText(Path.Combine(cfgDir, "config.json"), payloadJson); + return dir; + } + + [Fact] + public void WhitespaceOnlyPath_Raises() + { + // Deliberately NOT gated by the shared cross-port corpus: the TS + // reference (`config.ts`'s `z.string().min(1)`) rejects only a + // fully-empty path, not a whitespace-only one, and the reference is + // out of scope to change here. This port is stricter on this one + // edge case by design. + var dir = WriteConfig("""{ "schema_version": 1, "sources": [ { "path": " " } ] }"""); + try + { + Assert.ThrowsAny(() => NeutralConfig.Read(dir)); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + + [Fact] + public void SchemaVersionAsFloatLiteral_IsAcceptedLikeTheOtherThreePorts() + { + // Regression pin: `schema_version: 1.0` used to throw a raw, uncoded + // FormatException from GetInt32() — the other three ports all accept a + // float-looking `1.0` as equal to the supported version `1`. + var dir = WriteConfig("""{ "schema_version": 1.0, "sources": [ { "path": "model" } ] }"""); + try + { + var cfg = NeutralConfig.Read(dir); + Assert.NotNull(cfg); + Assert.Single(cfg!.Sources); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } + + [Fact] + public void SchemaVersionNonIntegral_RaisesCodedError_NotFormatException() + { + var dir = WriteConfig("""{ "schema_version": 1.5, "sources": [] }"""); + try + { + var ex = Assert.ThrowsAny(() => NeutralConfig.Read(dir)); + Assert.Equal(ErrorCode.ERR_BAD_ATTR_VALUE, ex.Code); + } + finally + { + Directory.Delete(dir, recursive: true); + } + } +} diff --git a/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs new file mode 100644 index 000000000..c79347340 --- /dev/null +++ b/server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs @@ -0,0 +1,162 @@ +// Runs the shared source-resolution corpus against this port. Reads the single +// committed fixtures/source-resolution-conformance/cases.json — no per-port +// fixture. The corpus is the contract; see server/typescript/packages/sdk/src/ +// sources.ts + collection.ts for the authoritative behavior each case pins. +using System.Text.Json; +using MetaObjects; +using MetaObjects.Config; +using Xunit; + +namespace MetaObjects.Conformance.Tests; + +public class SourceResolutionConformanceTests +{ + private sealed record Case( + string Name, + Dictionary Tree, + JsonElement? Config, + // Project-root-relative directory the resolver is invoked FROM. Defaults + // to "." — every case leaves it there EXCEPT + // "a-parent-relative-path-resolves-against-the-declaring-configs-directory", + // so for all the others the config lives at the project root and + // "relative to project root" vs "relative to the invocation directory" + // coincide. That one case is the one place those two bases diverge, and + // both the config's own location AND the expectFiles comparison base + // below must honor it correctly for that case to mean anything. (Do not + // restate this as "N of M cases" — the corpus grows and a hardcoded + // count silently goes stale; the structural description above does not.) + string ResolveFrom, + string[]? ExpectFiles, + // A JSON string pins the exact error code raised; JSON `true` pins only + // that resolution RAISES — the malformed-config error code is + // deliberately not pinned cross-port (see the corpus README). + JsonElement? ExpectError, + // Optional: linkPath -> targetPath, both project-root-relative, materialized + // AFTER `Tree` (I1 — a symlinked source root, or a symlinked subdirectory + // inside a walked tree). + Dictionary Symlinks); + + public static TheoryData CaseNames() + { + var data = new TheoryData(); + foreach (var c in LoadCases()) data.Add(c.Name); + return data; + } + + private static string CorpusPath() + { + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !Directory.Exists(Path.Combine(dir.FullName, "fixtures"))) + dir = dir.Parent; + Assert.NotNull(dir); + return Path.Combine(dir!.FullName, "fixtures", "source-resolution-conformance", "cases.json"); + } + + private static List LoadCases() + { + using var doc = JsonDocument.Parse(File.ReadAllText(CorpusPath())); + var cases = new List(); + foreach (var el in doc.RootElement.GetProperty("cases").EnumerateArray()) + { + var tree = new Dictionary(); + foreach (var p in el.GetProperty("tree").EnumerateObject()) + tree[p.Name] = p.Value.GetString() ?? ""; + + var cfgEl = el.GetProperty("config"); + JsonElement? cfg = cfgEl.ValueKind == JsonValueKind.Null ? null : cfgEl.Clone(); + + var resolveFrom = el.TryGetProperty("resolveFrom", out var rf) ? rf.GetString()! : "."; + + string[]? expectFiles = el.TryGetProperty("expectFiles", out var ef) + ? ef.EnumerateArray().Select(x => x.GetString()!).ToArray() + : null; + JsonElement? expectError = el.TryGetProperty("expectError", out var ee) ? ee.Clone() : null; + + var symlinks = new Dictionary(); + if (el.TryGetProperty("symlinks", out var sl)) + { + foreach (var p in sl.EnumerateObject()) symlinks[p.Name] = p.Value.GetString()!; + } + + cases.Add(new Case(el.GetProperty("name").GetString()!, tree, cfg, resolveFrom, expectFiles, expectError, symlinks)); + } + return cases; + } + + [Theory] + [MemberData(nameof(CaseNames))] + public void ResolvesTheSameFileSet(string name) + { + var c = LoadCases().Single(x => x.Name == name); + // `root` is the PROJECT ROOT — the base every `tree` path and every + // `expectFiles` entry is written relative to, regardless of `resolveFrom`. + var root = Path.Combine(Path.GetTempPath(), "mo-src-conf-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(root); + try + { + foreach (var (rel, content) in c.Tree) + { + var abs = Path.Combine(root, rel.Replace('/', Path.DirectorySeparatorChar)); + Directory.CreateDirectory(Path.GetDirectoryName(abs)!); + File.WriteAllText(abs, content); + } + // Materialized AFTER tree — see the Case record's Symlinks doc. + foreach (var (linkRel, targetRel) in c.Symlinks) + { + var linkAbs = Path.Combine(root, linkRel.Replace('/', Path.DirectorySeparatorChar)); + var targetAbs = Path.Combine(root, targetRel.Replace('/', Path.DirectorySeparatorChar)); + Directory.CreateDirectory(Path.GetDirectoryName(linkAbs)!); + Directory.CreateSymbolicLink(linkAbs, targetAbs); + } + + // The invocation directory: project root joined with `resolveFrom`. + // The config MUST be materialized here, not at the project root — a + // config placed at the project root while resolving FROM a + // subdirectory would go undetected by ResolveCollection there and + // fail loudly with ERR_COLLECTION_NOT_FOUND, which is what makes this + // half of the mistake self-catching. Getting it right on purpose (not + // by luck) is what this comment is pinning. + var invokeDir = Path.GetFullPath(Path.Combine(root, c.ResolveFrom)); + Directory.CreateDirectory(invokeDir); + if (c.Config is not null) + { + var d = Path.Combine(invokeDir, ".metaobjects"); + Directory.CreateDirectory(d); + File.WriteAllText(Path.Combine(d, "config.json"), c.Config.Value.GetRawText()); + } + + if (c.ExpectError is not null) + { + var ex = Assert.ThrowsAny(() => SourceResolver.ResolveCollection(invokeDir)); + // A string pins the exact code; `true` only pins that it raises — + // see the ExpectError field doc above. + if (c.ExpectError.Value.ValueKind == JsonValueKind.String) + { + Assert.Equal(c.ExpectError.Value.GetString(), ex.Code.ToString()); + } + return; + } + + // Compared against the PROJECT ROOT explicitly — never against + // `invokeDir`. For every case but the one that sets `resolveFrom` the + // two coincide (resolveFrom "."), so a comparison base bug here would + // pass every other case and fail only that one — which is exactly why + // that case exists. + var raw = SourceResolver.ResolveCollection(invokeDir); + var got = raw + .Select(f => Path.GetRelativePath(root, f).Replace(Path.DirectorySeparatorChar, '/')) + .ToHashSet(); + Assert.Equal(c.ExpectFiles!.ToHashSet(), got); + // A HashSet comparison alone cannot see a duplicate emission (two entries + // for the same file collapse invisibly into one set element) — + // `overlapping-sources-yield-each-file-once` specifically exercises + // ResolveCollection's own de-duplication, so the RAW list count must be + // asserted too, before it is thrown away by the ToHashSet conversion. + Assert.Equal(c.ExpectFiles!.Length, raw.Count); + } + finally + { + Directory.Delete(root, recursive: true); + } + } +} diff --git a/server/csharp/MetaObjects/Config/NeutralConfig.cs b/server/csharp/MetaObjects/Config/NeutralConfig.cs new file mode 100644 index 000000000..4f73255ac --- /dev/null +++ b/server/csharp/MetaObjects/Config/NeutralConfig.cs @@ -0,0 +1,122 @@ +// Port-neutral `.metaobjects/config.json` reading. +// +// Reads only the NEUTRAL SUBSET (`schema_version`, `sources`). The file also +// carries TypeScript-owned keys (`pending_in_git`, `confidence_thresholds`, +// `extract`, `migrate`, `scope`); those are IGNORED rather than modeled, so a +// new TS-only key never becomes a four-port change. `scope` in particular is +// entirely out of scope for this reader — see +// docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md §4. +using System.Text.Json; + +namespace MetaObjects.Config; + +public sealed class NeutralConfig +{ + /// The DEFAULT value of `sources` when the key is absent or empty — never a + /// requirement, and never assumed to exist by any other code path. + public const string DefaultMetadataDir = "metaobjects"; + + private const string MetaObjectsDir = ".metaobjects"; + private const string ConfigFile = "config.json"; + private const int SupportedSchemaVersion = 1; + + /// Each entry is a raw source spec (e.g. `{"path": "model"}`, + /// `{"resource": "com/acme/model"}`) — kind interpretation belongs to + /// SourceResolver, not here. + public IReadOnlyList> Sources { get; } + + private NeutralConfig(IReadOnlyList> sources) => Sources = sources; + + /// Returns null when `/.metaobjects/config.json` does not exist. + /// A file that EXISTS but is malformed THROWS — swallowing it would make a + /// typo'd config behave identically to no config at all, silently loading + /// from a possibly-stale default directory instead. The exact error code + /// used here is deliberately NOT part of the cross-port contract (only the + /// raise-don't-degrade behavior is), so callers should match on message, + /// not code, for this path. + public static NeutralConfig? Read(string configDir) + { + var path = Path.Combine(configDir, MetaObjectsDir, ConfigFile); + if (!File.Exists(path)) return null; + + JsonDocument doc; + try + { + doc = JsonDocument.Parse(File.ReadAllText(path)); + } + catch (Exception e) when (e is JsonException or IOException) + { + throw new MetaModelException( + $"{path} exists but could not be read as JSON: {e.Message}", + ErrorCode.ERR_MALFORMED_JSON); + } + + using (doc) + { + var root = doc.RootElement; + if (root.ValueKind != JsonValueKind.Object) + throw new MetaModelException($"{path} must contain a JSON object", ErrorCode.ERR_TOP_LEVEL_NOT_OBJECT); + + if (!root.TryGetProperty("schema_version", out var v) || + v.ValueKind != JsonValueKind.Number || + // Compare as a double, not GetInt32(): a float-looking literal like + // `1.0` is valid JSON and the other three ports all accept it as + // equal to 1 — GetInt32() used to throw a raw, uncoded + // FormatException on it instead. TryGetDouble never throws and a + // genuinely non-integral value (e.g. `1.5`) still correctly fails + // the equality check below. + !v.TryGetDouble(out var schemaVersion) || + schemaVersion != SupportedSchemaVersion) + { + throw new MetaModelException( + $"{path}: unsupported schema_version (expected {SupportedSchemaVersion})", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + + var specs = new List>(); + if (root.TryGetProperty("sources", out var srcs)) + { + // A present `sources` key that is not an array must RAISE, not + // silently read as "absent" — the latter would fall back to the + // default directory with no diagnostic, exactly the "typo'd config + // behaves like no config" failure this class exists to prevent (see + // the file header). This is the GENERAL rule: a present-but-JSON- + // null `sources` is just as wrong-typed as a present-but-object + // `sources` — it is not a special case. + if (srcs.ValueKind != JsonValueKind.Array) + throw new MetaModelException($"{path}: \"sources\" must be an array", ErrorCode.ERR_BAD_ATTR_VALUE); + + foreach (var s in srcs.EnumerateArray()) + { + if (s.ValueKind != JsonValueKind.Object) + throw new MetaModelException($"{path}: each \"sources\" entry must be an object", ErrorCode.ERR_BAD_ATTR_VALUE); + + var d = new Dictionary(); + foreach (var p in s.EnumerateObject()) + { + // Every source-spec value (`path`/`resource`/`package`) is a + // string — silently stringifying a non-string (the prior + // behavior) would let {"path": 123} load a directory + // literally named "123" rather than failing loudly on the + // typo'd config. + if (p.Value.ValueKind != JsonValueKind.String) + throw new MetaModelException($"{path}: \"sources\" entry \"{p.Name}\" must be a string", ErrorCode.ERR_BAD_ATTR_VALUE); + var value = p.Value.GetString()!; + if (string.IsNullOrWhiteSpace(value)) + throw new MetaModelException($"{path}: \"sources\" entry \"{p.Name}\" must not be empty", ErrorCode.ERR_BAD_ATTR_VALUE); + d[p.Name] = value; + } + + if (d.Count != 1) + throw new MetaModelException($"{path}: each \"sources\" entry must have exactly one key", ErrorCode.ERR_BAD_ATTR_VALUE); + + specs.Add(d); + } + } + + // Unknown top-level keys (including `scope`/`migrate`) are IGNORED by + // design — see the file header. + return new NeutralConfig(specs); + } + } +} diff --git a/server/csharp/MetaObjects/Config/SourceResolver.cs b/server/csharp/MetaObjects/Config/SourceResolver.cs new file mode 100644 index 000000000..d4e6b0669 --- /dev/null +++ b/server/csharp/MetaObjects/Config/SourceResolver.cs @@ -0,0 +1,115 @@ +// Turns a declared source SET (`.metaobjects/config.json`'s `sources`, or the +// single-entry default when it is absent/empty) into a de-duplicated list of +// metadata file paths. +// +// Behavioral contract: server/typescript/packages/sdk/src/sources.ts + +// collection.ts (the cross-port authority). File ORDER is deliberately NOT a +// cross-port contract (each port keeps its own natural order) — only the +// resolved SET and the error behavior are gated by the shared corpus at +// fixtures/source-resolution-conformance/cases.json. +using MetaObjects.Loader; + +namespace MetaObjects.Config; + +public static class SourceResolver +{ + /// Resolve a declared source SET to a de-duplicated list of metadata files. + /// A relative `path` resolves against `configDir` — the directory HOLDING the + /// `.metaobjects/` folder — never against the process working directory. + /// + /// Validation runs in two passes, mirroring `sources.ts`'s `orderedPathSpecs`: + /// EVERY spec's kind is checked first, in declared order, before any spec is + /// resolved against the filesystem. Interleaving the two (validate-then-resolve + /// spec by spec) would make which error fires depend on which unsupported spec + /// or missing path happens to sit first — the corpus pins that an unsupported + /// KIND anywhere in the list wins over an unresolved PATH regardless of which + /// is declared first (`unsupported-kind-precedes-unresolved-path-when-path-is- + /// declared-first`/`-second`). + public static IReadOnlyList ResolveSources( + string configDir, + IReadOnlyList> specs) + { + // Pass 1 — kind validation across the WHOLE set, no filesystem I/O yet. + var pathSpecs = new List(specs.Count); + foreach (var spec in specs) + { + if (spec.TryGetValue("path", out var rawPath)) + { + pathSpecs.Add(rawPath); + } + else + { + var kind = spec.Keys.FirstOrDefault() ?? ""; + throw new MetaModelException( + $"source kind \"{kind}\" is not supported by this toolchain yet; use a \"path\" source", + ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED); + } + } + + // Pass 2 — resolve each validated path spec against the filesystem. + var seen = new List(); + var known = new HashSet(StringComparer.Ordinal); + + foreach (var rawPath in pathSpecs) + { + var target = Path.IsPathRooted(rawPath) ? rawPath : Path.GetFullPath(Path.Combine(configDir, rawPath)); + + var isDir = Directory.Exists(target); + if (!isDir && !File.Exists(target)) + throw new MetaModelException( + $"source path \"{rawPath}\" does not exist (resolved to {target}, relative to {configDir})", + ErrorCode.ERR_SOURCE_UNRESOLVED); + + // Directory expansion — extension filter + ordinal sort — is + // DirectorySource's, the SAME code the loader itself uses to turn a + // directory into metadata files (Loader/DirectorySource.cs). Reimplementing + // the filter/sort here would be a second, driftable definition of "which + // files count as metadata" for exactly the reason DirectorySource's own + // header calls out: order within one directory spec is this port's own + // full-path ordinal sort, deliberately NOT a cross-port contract (see the + // file header above), but it MUST still be the loader's own order. + // ExcludePending = true: this IS the CLI-facing resolver — `_pending/` is + // the TypeScript CLI's pending/promote-workflow concept, not a loader + // concept, so the loader-level default (off) is overridden here, the one + // place this port's CLI turns it on. + var found = isDir + ? new DirectorySource(target, new DirectorySource.Options { ExcludePending = true }) + .Expand().Select(f => f.FilePath) + : new[] { target }.AsEnumerable(); + + foreach (var f in found) + { + var full = Path.GetFullPath(f); + if (known.Add(full)) seen.Add(full); + } + } + + return seen; + } + + /// The full ladder: declared `sources`, else the default directory. + /// Only the DEFAULT may be silently absent — a declared source that does not + /// resolve is `ERR_SOURCE_UNRESOLVED`, a louder failure than "nothing declared". + public static IReadOnlyList ResolveCollection(string root) + { + root = Path.GetFullPath(root); + var cfg = NeutralConfig.Read(root); + var specs = cfg?.Sources ?? Array.Empty>(); + + if (specs.Count == 0) + { + var defaultDir = Path.Combine(root, NeutralConfig.DefaultMetadataDir); + if (!Directory.Exists(defaultDir)) + throw new MetaModelException( + $"no metadata sources declared in {root} and no default \"{NeutralConfig.DefaultMetadataDir}\" " + + "directory found. Declare \"sources\" in .metaobjects/config.json, or run 'meta init' to scaffold.", + ErrorCode.ERR_COLLECTION_NOT_FOUND); + specs = new IReadOnlyDictionary[] + { + new Dictionary { ["path"] = NeutralConfig.DefaultMetadataDir }, + }; + } + + return ResolveSources(root, specs); + } +} diff --git a/server/csharp/MetaObjects/Loader/DirectorySource.cs b/server/csharp/MetaObjects/Loader/DirectorySource.cs index a893069a5..df06a62a5 100644 --- a/server/csharp/MetaObjects/Loader/DirectorySource.cs +++ b/server/csharp/MetaObjects/Loader/DirectorySource.cs @@ -24,11 +24,28 @@ public sealed class Options /// Recurse into subdirectories. Default: true. public bool Recurse { get; init; } = true; + + /// + /// Exclude _pending/ at any depth. Default: false — this is a + /// LOADER-level primitive, and _pending/ is a CLI/pending-promote-workflow + /// concept (TypeScript's metadata-files.ts, not its loader-level + /// DirectorySource, which has no _pending concept at all). + /// — the CLI-facing caller — + /// turns this ON explicitly rather than the exclusion being baked into every + /// embedder of this class: an app calling new DirectorySource(dir) + /// directly gets every file back, matching the reference loader. + /// + public bool ExcludePending { get; init; } = false; } private static readonly HashSet _supportedExtensions = new(StringComparer.OrdinalIgnoreCase) { ".json", ".yaml", ".yml" }; + /// Directory excluded at every level of — drafts that are + /// deliberately not part of the loaded model. Mirrors TypeScript's + /// `PENDING_DIR` in `metadata-files.ts`. + private const string PendingDir = "_pending"; + /// The directory being scanned. public string Directory { get; } @@ -55,6 +72,15 @@ public IEnumerable Expand() IEnumerable files = System.IO.Directory.EnumerateFiles(Directory, "*", search) .Where(p => _supportedExtensions.Contains(Path.GetExtension(p))); + if (Opts.ExcludePending) + { + // Excludes _pending/ at ANY depth — every ancestor path component + // between `Directory` and the file is checked, not merely the file's + // own name, so the whole subtree is skipped. Off by default — see + // Options.ExcludePending. + files = files.Where(p => !IsUnderPendingDir(Directory, p)); + } + if (Opts.Exclude is { Count: > 0 } excludes) { files = files.Where(p => !excludes.Any(e => MatchesGlob(Path.GetFileName(p), e))); @@ -65,6 +91,18 @@ public IEnumerable Expand() .Select(p => new FileSource(p)); } + /// True when any ancestor path component between and + /// (i.e. excluding the file's own name) is + /// exactly . + private static bool IsUnderPendingDir(string root, string filePath) + { + var relative = Path.GetRelativePath(root, filePath); + var dir = Path.GetDirectoryName(relative); + if (string.IsNullOrEmpty(dir)) return false; + return dir.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + .Any(part => part == PendingDir); + } + private static bool MatchesGlob(string name, string pattern) { // Minimal glob: literal match, or single leading '*' / trailing '*' wildcard. diff --git a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java index 1ca010650..a2969660a 100644 --- a/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java +++ b/server/java/maven-plugin/src/main/java/com/metaobjects/mojo/AbstractMetaDataMojo.java @@ -260,8 +260,17 @@ protected MetaDataLoader createLoader(ClassLoader projectClassLoader) { sourceDir = loaderConfig.getSourceDir(); } + // Precedence ladder (spec §5): when the pom names neither nor + // , fall back to the port-neutral .metaobjects/config.json (else the + // built-in default directory) instead of silently loading nothing. + List sources = loaderConfig.getSources(); + List neutralSources = resolveNeutralSourcesIfPomIsSilent(loaderConfig); + if (!neutralSources.isEmpty()) { + sources = neutralSources; + } + MavenLoaderConfiguration.configure(configurable, sourceDir, projectClassLoader, - loaderConfig.getSources(), loaderArgs(strict)); + sources, loaderArgs(strict)); MetaDataLoader loader = configurable.getLoader(); @@ -468,4 +477,55 @@ protected File getSourceDir() { } return sourceDir; } + + /** + * The module basedir metadata-source resolution is anchored to — the same + * directory that would hold a project's {@code .metaobjects/} folder. Falls back + * to the process working directory when {@code project} is unset (e.g. a Mojo + * driven directly in a unit test, matching {@link #warnIfAgentContextStale()}'s + * same fallback). + */ + protected File getProjectBaseDir() { + return project != null ? project.getBasedir() : new File(System.getProperty("user.dir")); + } + + /** + * The precedence ladder for where metadata lives (spec §5). First match wins. + * + *

1. The pom — {@code } or {@code }. If + * EITHER is present the pom owns the whole concern and the neutral file is not + * consulted; precedence is whole-concern, not a per-entry merge. + *
2. {@code sources} in the port-neutral {@code .metaobjects/config.json}, + * read from the module basedir. + *
3. The built-in default directory. + * + *

A neutral file that EXISTS but is malformed throws rather than falling + * through. {@code } is untouched by this ladder — {@code scope} stays + * out of scope for this mechanism (Global Constraints). + * + * @return the resolved metadata file paths, or an empty list when the pom names + * either {@code } or {@code } (i.e. the neutral file is + * not consulted at all) + */ + protected List resolveNeutralSourcesIfPomIsSilent(LoaderParam loaderConfig) { + boolean pomNamesLocation = + (loaderConfig.getSourceDir() != null && !loaderConfig.getSourceDir().isBlank()) + || (loaderConfig.getSources() != null && !loaderConfig.getSources().isEmpty()); + if (pomNamesLocation) return List.of(); + + // Prefixed "model:file:" explicitly rather than handed back as a bare + // Path::toString(): MetaDataLoader.processSources decides how to wrap a bare + // source string by checking `s.indexOf(':') < 0`, and an absolute path is not + // guaranteed colon-free — a Windows path (`C:\...`) fails that ambiguous + // sniff and is handed to URIHelper.toURI() unwrapped, which dies in + // validateUriType(). These are always fully-resolved absolute filesystem + // paths (SourceResolver.resolveCollection), so there is nothing to sniff: + // say "file" outright, the same shape processSources itself already builds + // for its own `new File(s).exists()` branch. + return com.metaobjects.config.SourceResolver + .resolveCollection(getProjectBaseDir().toPath()) + .stream() + .map(p -> "model:file:" + p.toString().replace(java.io.File.separatorChar, '/')) + .toList(); + } } diff --git a/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java new file mode 100644 index 000000000..1f1c5f6e9 --- /dev/null +++ b/server/java/maven-plugin/src/test/java/com/metaobjects/mojo/NeutralConfigMojoFallbackTest.java @@ -0,0 +1,173 @@ +package com.metaobjects.mojo; + +import com.metaobjects.MetaDataException; +import com.metaobjects.loader.MetaDataLoader; +import org.apache.maven.project.MavenProject; +import org.junit.Test; +import org.mockito.Mockito; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; +import java.util.Comparator; +import java.util.stream.Stream; + +import static org.junit.Assert.assertEquals; + +/** + * Proves the port-neutral {@code .metaobjects/config.json} fallback (spec §5, ADR + * cross-port metadata-source-resolution phase 1) is actually WIRED into + * {@link AbstractMetaDataMojo#createLoader}, not merely defined and unit-tested in + * isolation — the same gap that shipped in an earlier port's task before it was + * caught in review. Every test here leaves the pom's {@code } silent on both + * {@code sourceDir} and {@code sources}, so a regression that removes the + * {@code resolveNeutralSourcesIfPomIsSilent(...)} call site (or short-circuits it) + * makes {@code createLoader} load ZERO metadata objects instead of the one declared + * under the neutral config's declared source — turning every assertion below false. + */ +public class NeutralConfigMojoFallbackTest { + + private static final String WIDGET_JSON = """ + { + "metadata.root": { + "package": "neutral::fallback", + "children": [ + { "object.entity": { "name": "Widget", "children": [ + { "field.long": { "name": "id" } } + ] } } + ] + } + } + """; + + /** A pom-silent loader: no {@code }, no {@code }. */ + private MetaDataGeneratorMojo mojoWithSilentPom(Path basedir) { + MavenProject mavenProject = Mockito.mock(MavenProject.class); + Mockito.when(mavenProject.getBasedir()).thenReturn(basedir.toFile()); + + MetaDataGeneratorMojo mojo = new MetaDataGeneratorMojo(); + mojo.project = mavenProject; + + LoaderParam loader = LoaderParam.builder("neutral-fallback-test") + .withClassname("com.metaobjects.loader.MetaDataLoader") + .build(); + mojo.setLoader(loader); + mojo.setGenerators(Collections.emptyList()); + mojo.setGlobals(Collections.emptyMap()); + return mojo; + } + + @Test + public void createLoaderFallsBackToTheNeutralConfigsDeclaredSourceWhenPomIsSilent() throws IOException { + Path root = Files.createTempDirectory("mo-mojo-neutral-").toAbsolutePath().normalize(); + try { + Path metaDir = root.resolve("custom-metadata"); + Files.createDirectories(metaDir); + Files.write(metaDir.resolve("meta.widget.json"), WIDGET_JSON.getBytes(StandardCharsets.UTF_8)); + + Path dotMo = root.resolve(".metaobjects"); + Files.createDirectories(dotMo); + Files.write(dotMo.resolve("config.json"), + "{\"schema_version\":1,\"sources\":[{\"path\":\"custom-metadata\"}]}" + .getBytes(StandardCharsets.UTF_8)); + + MetaDataGeneratorMojo mojo = mojoWithSilentPom(root); + MetaDataLoader loaded = mojo.createLoader(mojo.createProjectClassLoader()); + + // If resolveNeutralSourcesIfPomIsSilent were never called (or its result + // discarded), the loader would have been configured with zero sources and + // loaded nothing — this assertion is exactly what removing the wiring + // breaks. + assertEquals("expected the ONE object declared under the neutral config's " + + "declared \"custom-metadata\" source — proves the fallback ran, " + + "not just the default directory", + 1, loaded.getMetaObjects().size()); + assertEquals("Widget", loaded.getMetaObjects().get(0).getShortName()); + } finally { + deleteRecursive(root); + } + } + + @Test + public void createLoaderFallsBackToTheBuiltInDefaultDirectoryWhenNoNeutralConfigExists() throws IOException { + Path root = Files.createTempDirectory("mo-mojo-neutral-default-").toAbsolutePath().normalize(); + try { + // No .metaobjects/config.json at all — only the built-in default directory. + Path defaultDir = root.resolve("metaobjects"); + Files.createDirectories(defaultDir); + Files.write(defaultDir.resolve("meta.widget.json"), WIDGET_JSON.getBytes(StandardCharsets.UTF_8)); + + MetaDataGeneratorMojo mojo = mojoWithSilentPom(root); + MetaDataLoader loaded = mojo.createLoader(mojo.createProjectClassLoader()); + + assertEquals(1, loaded.getMetaObjects().size()); + assertEquals("Widget", loaded.getMetaObjects().get(0).getShortName()); + } finally { + deleteRecursive(root); + } + } + + @Test + public void createLoaderHandlesAColonInTheResolvedAbsolutePath() throws IOException { + // Regression: resolveNeutralSourcesIfPomIsSilent used to hand + // MetaDataLoader.processSources a bare Path::toString() — ambiguous + // whenever the absolute path contains a colon, since processSources + // sniffs `s.indexOf(':') < 0` to decide whether to wrap the string as a + // "model:file:" source. A Windows drive letter (`C:\...`) is the common + // case; reproduced here without a Windows machine via a colon in a plain + // Unix directory NAME (legal on ext4/most POSIX filesystems), which is + // just as ambiguous to that sniff. Before the fix this died with an + // uncoded IllegalArgumentException out of URIHelper.validateUriType. + Path root = Files.createTempDirectory("mo-mojo-neutral-colon-").toAbsolutePath().normalize(); + try { + Path weirdRoot = root.resolve("cache:v1"); + Files.createDirectories(weirdRoot); + Path metaDir = weirdRoot.resolve("custom-metadata"); + Files.createDirectories(metaDir); + Files.write(metaDir.resolve("meta.widget.json"), WIDGET_JSON.getBytes(StandardCharsets.UTF_8)); + + Path dotMo = weirdRoot.resolve(".metaobjects"); + Files.createDirectories(dotMo); + Files.write(dotMo.resolve("config.json"), + "{\"schema_version\":1,\"sources\":[{\"path\":\"custom-metadata\"}]}" + .getBytes(StandardCharsets.UTF_8)); + + MetaDataGeneratorMojo mojo = mojoWithSilentPom(weirdRoot); + MetaDataLoader loaded = mojo.createLoader(mojo.createProjectClassLoader()); + + assertEquals(1, loaded.getMetaObjects().size()); + assertEquals("Widget", loaded.getMetaObjects().get(0).getShortName()); + } finally { + deleteRecursive(root); + } + } + + @Test(expected = MetaDataException.class) + public void createLoaderRaisesWhenPomIsSilentAndNoCollectionExists() throws IOException { + // Neither a neutral config nor a default "metaobjects/" directory — the final + // rung of the ladder must raise (ERR_COLLECTION_NOT_FOUND), not silently load + // zero objects. + Path root = Files.createTempDirectory("mo-mojo-neutral-empty-").toAbsolutePath().normalize(); + try { + MetaDataGeneratorMojo mojo = mojoWithSilentPom(root); + mojo.createLoader(mojo.createProjectClassLoader()); + } finally { + deleteRecursive(root); + } + } + + private static void deleteRecursive(Path dir) throws IOException { + if (!Files.exists(dir)) return; + try (Stream walk = Files.walk(dir)) { + walk.sorted(Comparator.reverseOrder()).forEach(p -> { + try { + Files.delete(p); + } catch (IOException ignored) { + // best-effort temp-dir cleanup + } + }); + } + } +} diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java new file mode 100644 index 000000000..391cb0d0c --- /dev/null +++ b/server/java/metadata/src/main/java/com/metaobjects/config/NeutralConfig.java @@ -0,0 +1,179 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.google.gson.JsonElement; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import com.google.gson.JsonSyntaxException; +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * The port-neutral subset of {@code .metaobjects/config.json}. + * + *

Reads only {@code schema_version} and {@code sources}. The file also carries + * TypeScript-owned keys ({@code pending_in_git}, {@code confidence_thresholds}, + * {@code extract}, {@code migrate}, {@code scope}); those are IGNORED rather than + * modeled, so a new TS-only key never becomes a four-port change. {@code scope} in + * particular is entirely out of scope for this reader — see + * {@code docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md} §4. + */ +public final class NeutralConfig { + + /** + * The DEFAULT value of {@code sources} when the key is absent or empty — never a + * requirement, and never assumed to exist by any other code path. + */ + public static final String DEFAULT_METADATA_DIR = "metaobjects"; + + private static final String METAOBJECTS_DIR = ".metaobjects"; + private static final String CONFIG_FILE = "config.json"; + private static final int SUPPORTED_SCHEMA_VERSION = 1; + + private final List> sources; + + private NeutralConfig(List> sources) { + this.sources = List.copyOf(sources); + } + + /** + * Each entry is a raw source spec (e.g. {@code {"path": "model"}}, + * {@code {"resource": "com/acme/model"}}) — kind interpretation belongs to + * {@link SourceResolver}, not here. + */ + public List> getSources() { + return sources; + } + + /** + * Returns empty when {@code /.metaobjects/config.json} does not exist. + * A file that EXISTS but is malformed throws — swallowing it would make a typo'd + * config behave identically to no config at all, silently loading from a possibly + * stale default directory instead. The exact {@link ErrorCode} used for the + * malformed path is deliberately NOT part of the cross-port contract (only the + * raise-don't-degrade behavior is) — callers should match on the raise, not the + * code, for this path. + */ + public static Optional read(Path configDir) { + Path path = configDir.resolve(METAOBJECTS_DIR).resolve(CONFIG_FILE); + if (!Files.isRegularFile(path)) return Optional.empty(); + + String content; + try { + content = new String(Files.readAllBytes(path), StandardCharsets.UTF_8); + } catch (IOException e) { + throw new MetaDataException( + path + " exists but could not be read: " + e.getMessage(), + ErrorCode.ERR_MALFORMED_JSON); + } + + JsonElement parsed; + try { + parsed = JsonParser.parseString(content); + } catch (JsonSyntaxException e) { + throw new MetaDataException( + path + " exists but could not be parsed as JSON: " + e.getMessage(), + ErrorCode.ERR_MALFORMED_JSON); + } + + if (!parsed.isJsonObject()) { + throw new MetaDataException( + path + " must contain a JSON object", + ErrorCode.ERR_TOP_LEVEL_NOT_OBJECT); + } + JsonObject root = parsed.getAsJsonObject(); + + JsonElement version = root.get("schema_version"); + // Compared as a double, not getAsInt(): Gson's getAsInt() on a non-integral + // BigDecimal TRUNCATES rather than raising, so a typo'd `schema_version: 1.5` + // silently read as 1 and passed. getAsDouble() still accepts an + // integral-valued float like `1.0` (equal to 1, and valid JSON — every other + // port accepts it too, C#'s NeutralConfig.cs for the identical reason) while + // correctly rejecting a genuinely non-integral value. + if (version == null || !version.isJsonPrimitive() || !version.getAsJsonPrimitive().isNumber() + || version.getAsDouble() != SUPPORTED_SCHEMA_VERSION) { + throw new MetaDataException( + path + ": unsupported schema_version (expected " + SUPPORTED_SCHEMA_VERSION + ")", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + + List> specs = new ArrayList<>(); + JsonElement srcs = root.get("sources"); + // A present `sources` key that is not an array must RAISE, not silently + // read as "absent" — the latter would fall back to the default directory + // with no diagnostic, exactly the "typo'd config behaves like no config" + // failure this class exists to prevent (see the class javadoc). This is + // the GENERAL rule: a present-but-JSON-null `sources` is just as wrong- + // typed as a present-but-object `sources` — it is not a special case. + if (srcs != null && !srcs.isJsonArray()) { + throw new MetaDataException( + path + ": \"sources\" must be an array", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + if (srcs != null) { + for (JsonElement el : srcs.getAsJsonArray()) { + if (!el.isJsonObject()) { + throw new MetaDataException( + path + ": each \"sources\" entry must be an object", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + JsonObject entry = el.getAsJsonObject(); + if (entry.size() != 1) { + throw new MetaDataException( + path + ": each \"sources\" entry must have exactly one key", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + Map spec = new LinkedHashMap<>(); + for (Map.Entry e : entry.entrySet()) { + JsonElement v = e.getValue(); + if (!v.isJsonPrimitive() || !v.getAsJsonPrimitive().isString()) { + // Every source-spec value (`path`/`resource`/`package`) is a + // string — a bare number/boolean/object/array silently + // stringified (the prior behavior) would let + // {"path": 123} load a directory literally named "123" + // rather than failing loudly on the typo'd config. + throw new MetaDataException( + path + ": \"sources\" entry \"" + e.getKey() + "\" must be a string", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + String value = v.getAsString(); + if (value.strip().isEmpty()) { + throw new MetaDataException( + path + ": \"sources\" entry \"" + e.getKey() + "\" must not be empty", + ErrorCode.ERR_BAD_ATTR_VALUE); + } + spec.put(e.getKey(), value); + } + specs.add(spec); + } + } + + // Unknown top-level keys (including `scope`/`migrate`) are IGNORED by + // design — see the class javadoc. + return Optional.of(new NeutralConfig(specs)); + } +} diff --git a/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java new file mode 100644 index 000000000..74b7e7f3a --- /dev/null +++ b/server/java/metadata/src/main/java/com/metaobjects/config/SourceResolver.java @@ -0,0 +1,135 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; +import com.metaobjects.loader.DirectorySource; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; + +/** + * Turns a declared source SET ({@code .metaobjects/config.json}'s {@code sources}, or + * the single-entry default when it is absent/empty) into a de-duplicated list of + * metadata file paths. + * + *

Behavioral contract: {@code server/typescript/packages/sdk/src/sources.ts} + + * {@code collection.ts} (the cross-port authority). File ORDER is deliberately NOT a + * cross-port contract (each port keeps its own natural order — this port's is a + * basename sort within a directory) — only the resolved SET and the error behavior + * are gated by the shared corpus at + * {@code fixtures/source-resolution-conformance/cases.json}. + */ +public final class SourceResolver { + + private SourceResolver() {} + + /** + * Resolve a declared source SET to a de-duplicated list of metadata files. + * + *

A relative {@code path} resolves against {@code configDir} — the directory + * HOLDING the {@code .metaobjects/} folder — never against the process working + * directory. + * + *

Validation runs in two passes: EVERY spec's kind is checked first, in + * declared order, before any spec is resolved against the filesystem. + * Interleaving the two (validate-then-resolve spec by spec) would make which + * error fires depend on which unsupported spec or missing path happens to sit + * first — the corpus pins that an unsupported KIND anywhere in the list wins over + * an unresolved PATH regardless of which is declared first + * ({@code unsupported-kind-precedes-unresolved-path-when-path-is-declared-first}/ + * {@code -second}). + */ + public static List resolveSources(Path configDir, List> specs) { + // Pass 1 — kind validation across the WHOLE set, no filesystem I/O yet. + List pathSpecs = new ArrayList<>(specs.size()); + for (Map spec : specs) { + String rawPath = spec.get("path"); + if (rawPath == null) { + String kind = spec.keySet().stream().findFirst().orElse(""); + throw new MetaDataException( + "source kind \"" + kind + "\" is not supported by this toolchain yet; use a \"path\" source", + ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED); + } + pathSpecs.add(rawPath); + } + + // Pass 2 — resolve each validated path spec against the filesystem. + LinkedHashSet seen = new LinkedHashSet<>(); + for (String rawPath : pathSpecs) { + Path raw = Path.of(rawPath); + Path target = raw.isAbsolute() ? raw : configDir.resolve(raw).normalize(); + + boolean isDir = Files.isDirectory(target); + if (!isDir && !Files.isRegularFile(target)) { + throw new MetaDataException( + "source path \"" + rawPath + "\" does not exist (resolved to " + target + + ", relative to " + configDir + ")", + ErrorCode.ERR_SOURCE_UNRESOLVED); + } + + if (isDir) { + // Directory expansion — extension filter and basename sort (this + // port's own order, deliberately NOT a cross-port contract — see the + // class javadoc) — is DirectorySource's, the SAME code the loader + // itself uses to turn a directory into metadata files. Reimplementing + // the walk here would be a second, driftable definition of "which + // files count as metadata". `_pending/`-at-any-depth exclusion is OFF + // by default at the loader level (a runtime app embedding + // DirectorySource directly gets every file) — this CLI-facing + // resolver turns it ON, since `_pending/` is the TypeScript CLI's + // pending/promote-workflow concept, not a loader concept. + new DirectorySource(target, new DirectorySource.Options().setExcludePending(true)).expand() + .forEach(fs -> seen.add(fs.getPath().toAbsolutePath().normalize())); + } else { + seen.add(target.toAbsolutePath().normalize()); + } + } + + return new ArrayList<>(seen); + } + + /** + * The full ladder: declared {@code sources}, else the default directory. Only the + * DEFAULT may be silently absent — a declared source that does not resolve is + * {@code ERR_SOURCE_UNRESOLVED}, a louder failure than "nothing declared". + */ + public static List resolveCollection(Path root) { + Path base = root.toAbsolutePath().normalize(); + List> specs = NeutralConfig.read(base) + .map(NeutralConfig::getSources) + .orElse(List.of()); + + if (specs.isEmpty()) { + Path defaultDir = base.resolve(NeutralConfig.DEFAULT_METADATA_DIR); + if (!Files.isDirectory(defaultDir)) { + throw new MetaDataException( + "no metadata sources declared in " + base + " and no default \"" + + NeutralConfig.DEFAULT_METADATA_DIR + "\" directory found. Declare \"sources\" in " + + ".metaobjects/config.json, or run 'meta init' to scaffold.", + ErrorCode.ERR_COLLECTION_NOT_FOUND); + } + specs = List.of(Map.of("path", NeutralConfig.DEFAULT_METADATA_DIR)); + } + + return resolveSources(base, specs); + } +} diff --git a/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java b/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java index f6268674c..42b90d3cb 100644 --- a/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java +++ b/server/java/metadata/src/main/java/com/metaobjects/loader/DirectorySource.java @@ -2,6 +2,8 @@ import java.io.IOException; import java.io.UncheckedIOException; +import java.nio.file.FileSystemLoopException; +import java.nio.file.FileVisitOption; import java.nio.file.Files; import java.nio.file.Path; import java.util.Comparator; @@ -37,6 +39,7 @@ public final class DirectorySource { public static final class Options { private List exclude = List.of(); private boolean recurse = true; + private boolean excludePending = false; /** * Sets filenames to exclude from expansion (exact filename match). @@ -54,12 +57,35 @@ public Options setRecurse(boolean recurse) { return this; } + /** + * Sets whether {@code _pending/} (at any depth) is excluded from expansion. + * Default: {@code false} — this is a LOADER-level primitive, and + * {@code _pending/} is a CLI/pending-workflow concept (TypeScript's + * {@code metadata-files.ts}, not its loader-level {@code DirectorySource}). + * {@link com.metaobjects.config.SourceResolver}, the CLI-facing caller, turns + * this ON explicitly rather than the exclusion being baked into every + * embedder of this class (a runtime app calling {@code new DirectorySource(dir)} + * directly gets every file back, matching the reference loader). + */ + public Options setExcludePending(boolean excludePending) { + this.excludePending = excludePending; + return this; + } + public List getExclude() { return exclude; } public boolean isRecurse() { return recurse; } + public boolean isExcludePending() { return excludePending; } } private static final Set EXTENSIONS = Set.of(".json", ".yaml", ".yml"); + /** + * Directory excluded at every level of {@link #expand()} — drafts that are + * deliberately not part of the loaded model. Mirrors TypeScript's + * {@code PENDING_DIR} in {@code metadata-files.ts}. + */ + private static final String PENDING_DIR = "_pending"; + private final Path directory; private final Options opts; @@ -90,18 +116,34 @@ public Options getOptions() { * Expands this directory into a sorted stream of {@link FileSource}. * Sorted by full path (ordinal) for deterministic ordering across runs. * + *

Follows symlinked directories — including when {@code directory} itself + * is a symlink — matching TypeScript ({@code stat}, not {@code lstat}) and C# + * ({@code EnumerateFiles(..., AllDirectories)}), both of which have always + * followed symlinks this way. A symlink CYCLE is a loud error rather than a + * hang: {@code Files.walk}'s own traversal detects it and throws + * {@link FileSystemLoopException} (naming the looping path), which the JDK + * wraps in {@link UncheckedIOException} and surfaces from whichever stream + * operation is consuming this method's lazily-returned {@link Stream} — not + * from this method itself, since the walk has not actually run yet when + * {@code expand()} returns.

+ * * @return stream of file sources; caller should close if iteration is partial * @throws UncheckedIOException if the directory cannot be listed */ public Stream expand() { try { Stream walk = opts.isRecurse() - ? Files.walk(directory) + ? Files.walk(directory, FileVisitOption.FOLLOW_LINKS) : Files.list(directory); return walk .filter(Files::isRegularFile) .filter(p -> hasSupportedExtension(p.getFileName().toString())) .filter(p -> !opts.getExclude().contains(p.getFileName().toString())) + // Excludes _pending/ at ANY depth — every ancestor path component + // between `directory` and `p` is checked, not merely `p`'s own + // basename, so the whole subtree is skipped. Off by default — see + // Options.setExcludePending. + .filter(p -> !opts.isExcludePending() || !isUnderPendingDir(directory, p)) .sorted(Comparator.comparing(p -> p.getFileName().toString())) .map(FileSource::new); } catch (IOException e) { @@ -116,6 +158,19 @@ public List expandToList() { return expand().map(fs -> (MetaDataSource) fs).collect(Collectors.toList()); } + /** + * True when any ancestor path component between {@code base} and {@code file} + * (i.e. excluding {@code file}'s own name) is exactly {@link #PENDING_DIR}. + */ + private static boolean isUnderPendingDir(Path base, Path file) { + Path rel = base.relativize(file).getParent(); + if (rel == null) return false; + for (Path part : rel) { + if (part.toString().equals(PENDING_DIR)) return true; + } + return false; + } + private static boolean hasSupportedExtension(String name) { String lower = name.toLowerCase(Locale.ROOT); for (String ext : EXTENSIONS) { diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java new file mode 100644 index 000000000..7dd719138 --- /dev/null +++ b/server/java/metadata/src/test/java/com/metaobjects/config/NeutralConfigTest.java @@ -0,0 +1,81 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.metaobjects.ErrorCode; +import com.metaobjects.MetaDataException; +import org.junit.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertThrows; + +/** + * Focused unit coverage for {@link NeutralConfig#read} on shapes NOT gated by the + * shared {@code source-resolution-conformance} corpus — either because the + * reference TypeScript implementation deliberately behaves differently + * (whitespace-only path) or to pin a prior defect class directly. + */ +public class NeutralConfigTest { + + private static Path writeConfig(String payloadJson) throws IOException { + Path dir = Files.createTempDirectory("neutral-config-"); + Path cfgDir = Files.createDirectories(dir.resolve(".metaobjects")); + Files.write(cfgDir.resolve("config.json"), payloadJson.getBytes(StandardCharsets.UTF_8)); + return dir; + } + + @Test + public void whitespaceOnlyPathRaises() throws IOException { + // Deliberately NOT gated by the shared cross-port corpus: the TS + // reference (`config.ts`'s `z.string().min(1)`) rejects only a + // fully-empty path, not a whitespace-only one, and the reference is out + // of scope to change here. This port is stricter on this one edge case + // by design. + Path dir = writeConfig("{ \"schema_version\": 1, \"sources\": [ { \"path\": \" \" } ] }"); + assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); + } + + @Test + public void nonStringSourceValueRaises() throws IOException { + Path dir = writeConfig("{ \"schema_version\": 1, \"sources\": [ { \"path\": 123 } ] }"); + MetaDataException ex = assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); + assertEquals(ErrorCode.ERR_BAD_ATTR_VALUE, ex.getCode().orElseThrow()); + } + + @Test + public void schemaVersionAsFloatLiteralIsAcceptedLikeTheOtherThreePorts() throws IOException { + // `schema_version: 1.0` is valid JSON, equal to 1, and every other port + // accepts it (C#'s NeutralConfigTests.cs pins the identical case). + Path dir = writeConfig("{ \"schema_version\": 1.0, \"sources\": [ { \"path\": \"model\" } ] }"); + NeutralConfig cfg = NeutralConfig.read(dir).orElseThrow(); + assertEquals(1, cfg.getSources().size()); + } + + @Test + public void schemaVersionNonIntegralRaisesRatherThanTruncating() throws IOException { + // Regression: Gson's JsonPrimitive#getAsInt() TRUNCATES a non-integral + // BigDecimal instead of raising, so `schema_version: 1.5` used to read as + // 1 and pass silently. Comparing as a double (NeutralConfig.java) catches it. + Path dir = writeConfig("{ \"schema_version\": 1.5, \"sources\": [] }"); + MetaDataException ex = assertThrows(MetaDataException.class, () -> NeutralConfig.read(dir)); + assertEquals(ErrorCode.ERR_BAD_ATTR_VALUE, ex.getCode().orElseThrow()); + } +} diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java new file mode 100644 index 000000000..d24765fbf --- /dev/null +++ b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java @@ -0,0 +1,244 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.google.gson.JsonArray; +import com.google.gson.JsonElement; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import com.metaobjects.MetaDataException; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.junit.runners.Parameterized.Parameter; +import org.junit.runners.Parameterized.Parameters; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Comparator; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.fail; + +/** + * Runs the shared source-resolution corpus against this port. Reads the single + * committed {@code fixtures/source-resolution-conformance/cases.json} — no per-port + * fixture. The corpus is the contract; see + * {@code server/typescript/packages/sdk/src/sources.ts} + {@code collection.ts} for + * the authoritative behavior each case pins. + */ +@RunWith(Parameterized.class) +public class SourceResolutionConformanceTest { + + /** + * One row per corpus case. + * + * @param resolveFrom Project-root-relative directory the resolver is invoked + * FROM. Defaults to {@code "."} — every case leaves it there EXCEPT + * {@code a-parent-relative-path-resolves-against-the-declaring-configs- + * directory}, so for all the others the config lives at the project root + * and "relative to project root" vs "relative to the invocation directory" + * coincide. That one case is the one place those two bases diverge, and + * both the config's own location AND the {@code expectFiles} comparison + * base below must honor it correctly for that case to mean anything. (Do + * not restate this as "N of M cases" — the corpus grows and a hardcoded + * count silently goes stale; the structural description above does not.) + */ + /** + * {@code expectError}: a JSON string pins the exact error code raised; JSON + * {@code true} pins only that resolution RAISES — the malformed-config error + * code is deliberately not pinned cross-port (see the corpus README). + */ + private record Case(String name, Map tree, JsonObject config, + String resolveFrom, List expectFiles, JsonElement expectError, + Map symlinks) {} + + /** Package-private (not private): shared with {@link SourceResolutionCorpusNotEmptyTest}, + * which needs to locate the same committed corpus file without a second definition + * of "walk up to find fixtures/". */ + static Path corpus() { + Path dir = Paths.get("").toAbsolutePath(); + while (dir != null && !Files.isDirectory(dir.resolve("fixtures"))) dir = dir.getParent(); + assertNotNull("could not locate the repository fixtures/ directory", dir); + return dir.resolve("fixtures/source-resolution-conformance/cases.json"); + } + + @Parameters(name = "{0}") + public static Collection cases() throws IOException { + String content = new String(Files.readAllBytes(corpus()), StandardCharsets.UTF_8); + JsonObject root = JsonParser.parseString(content).getAsJsonObject(); + JsonArray arr = root.getAsJsonArray("cases"); + + List rows = new ArrayList<>(); + for (JsonElement el : arr) { + JsonObject c = el.getAsJsonObject(); + String name = c.get("name").getAsString(); + + Map tree = new LinkedHashMap<>(); + for (Map.Entry e : c.getAsJsonObject("tree").entrySet()) { + tree.put(e.getKey(), e.getValue().getAsString()); + } + + // A LITERALLY ABSENT "config" key is a malformed corpus case, not the same + // thing as an explicit `"config": null` (which means "no config file" — + // see e.g. no-config-uses-default-directory). Python's dict indexing and + // C#'s JsonElement.GetProperty both throw on the former; Gson's `get()` + // returns Java null for BOTH, so without this check a future case that + // simply forgot the key would silently read as "no config" here while the + // other three runners crash loudly on the same corpus file. + if (!c.has("config")) { + throw new IllegalStateException( + "corpus case \"" + name + "\" has no \"config\" key (use JSON null for " + + "\"no config file\", not an absent key)"); + } + JsonElement cfgEl = c.get("config"); + JsonObject config = cfgEl.isJsonNull() ? null : cfgEl.getAsJsonObject(); + + String resolveFrom = c.has("resolveFrom") ? c.get("resolveFrom").getAsString() : "."; + + List expectFiles = null; + if (c.has("expectFiles")) { + expectFiles = new ArrayList<>(); + for (JsonElement f : c.getAsJsonArray("expectFiles")) { + expectFiles.add(f.getAsString()); + } + } + + JsonElement expectError = c.has("expectError") ? c.get("expectError") : null; + + // Optional: linkPath -> targetPath, both project-root-relative, materialized + // AFTER `tree` (I1 — a symlinked source root, or a symlinked subdirectory + // inside a walked tree). + Map symlinks = new LinkedHashMap<>(); + if (c.has("symlinks")) { + for (Map.Entry e : c.getAsJsonObject("symlinks").entrySet()) { + symlinks.put(e.getKey(), e.getValue().getAsString()); + } + } + + rows.add(new Object[]{name, new Case(name, tree, config, resolveFrom, expectFiles, expectError, symlinks)}); + } + return rows; + } + + /** First arg drives the JUnit display name. */ + @Parameter(0) + public String caseName; + + @Parameter(1) + public Case testCase; + + @Test + public void resolvesTheSameFileSet() throws IOException { + // `root` is the PROJECT ROOT — the base every `tree` path and every + // `expectFiles` entry is written/compared relative to, regardless of + // `resolveFrom`. Normalized the same way SourceResolver.resolveCollection + // normalizes its own `root` argument, so relativizing against it later lines + // up exactly with what the resolver returns. + Path root = Files.createTempDirectory("mo-src-resolution-").toAbsolutePath().normalize(); + try { + for (Map.Entry entry : testCase.tree().entrySet()) { + Path abs = root.resolve(entry.getKey()); + Files.createDirectories(abs.getParent()); + Files.write(abs, entry.getValue().getBytes(StandardCharsets.UTF_8)); + } + // Materialized AFTER tree — see the Case record's symlinks doc. + for (Map.Entry link : testCase.symlinks().entrySet()) { + Path linkPath = root.resolve(link.getKey()); + Files.createDirectories(linkPath.getParent()); + Files.createSymbolicLink(linkPath, root.resolve(link.getValue())); + } + + // The invocation directory: project root joined with `resolveFrom`. The + // config MUST be materialized here, not at the project root — a config + // placed at the project root while resolving FROM a subdirectory would go + // undetected by resolveCollection there and fail loudly with + // ERR_COLLECTION_NOT_FOUND, which is what makes this half of the mistake + // self-catching. Getting it right on purpose (not by luck) is what this + // comment is pinning. + Path invokeDir = root.resolve(testCase.resolveFrom()).normalize(); + Files.createDirectories(invokeDir); + if (testCase.config() != null) { + Path dotMo = invokeDir.resolve(".metaobjects"); + Files.createDirectories(dotMo); + Files.write(dotMo.resolve("config.json"), + testCase.config().toString().getBytes(StandardCharsets.UTF_8)); + } + + if (testCase.expectError() != null) { + try { + SourceResolver.resolveCollection(invokeDir); + fail("expected " + testCase.expectError() + " for case " + testCase.name()); + } catch (MetaDataException e) { + JsonElement expected = testCase.expectError(); + // A string pins the exact code; `true` only pins that it raises — + // see the Case record's javadoc above. + if (expected.isJsonPrimitive() && expected.getAsJsonPrimitive().isString()) { + assertEquals(expected.getAsString(), e.getCode().orElseThrow().name()); + } + } + return; + } + + // Compared against the PROJECT ROOT explicitly — never against + // `invokeDir`. For every case but the one that sets `resolveFrom` the two + // coincide (resolveFrom "."), so a comparison-base bug here would pass + // every other case and fail only that one — which is exactly why that + // case exists. + List raw = SourceResolver.resolveCollection(invokeDir); + Set got = raw.stream() + .map(p -> root.relativize(p).toString().replace('\\', '/')) + .collect(Collectors.toSet()); + + Set want = new HashSet<>(testCase.expectFiles()); + assertEquals(want, got); + // The Set comparison above cannot see a duplicate emission (two entries + // for the same file collapse invisibly into one Set element) — + // `overlapping-sources-yield-each-file-once` specifically exercises + // resolveCollection's own de-duplication, so the RAW list length must be + // asserted too, before it is thrown away by the Set conversion. + assertEquals(want.size(), raw.size()); + } finally { + deleteRecursive(root); + } + } + + private static void deleteRecursive(Path dir) throws IOException { + if (!Files.exists(dir)) return; + try (Stream walk = Files.walk(dir)) { + walk.sorted(Comparator.reverseOrder()).forEach(p -> { + try { + Files.delete(p); + } catch (IOException ignored) { + // best-effort temp-dir cleanup + } + }); + } + } +} diff --git a/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionCorpusNotEmptyTest.java b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionCorpusNotEmptyTest.java new file mode 100644 index 000000000..334c83eca --- /dev/null +++ b/server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionCorpusNotEmptyTest.java @@ -0,0 +1,51 @@ +/* + * Copyright 2003 Doug Mealing LLC dba Meta Objects + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.metaobjects.config; + +import com.google.gson.JsonArray; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import org.junit.Test; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; + +import static org.junit.Assert.assertTrue; + +/** + * Deliberately NOT part of {@link SourceResolutionConformanceTest}'s + * {@code @RunWith(Parameterized.class)} run: JUnit4's {@code Parameterized} runner + * executes its {@code @Test} methods once PER PARAMETER ROW, so if {@code cases()} + * ever returned zero rows — a bad path, a JSON-parsing bug, an accidental corpus + * truncation — every {@code @Test} in that class (including a guard living inside + * it) would run zero times and Maven would report the class GREEN, having checked + * nothing. This is a plain, separate JUnit4 test class so it always runs exactly + * once regardless of what the corpus contains. Mirrors the TS and Python runners' + * identically-purposed guards ({@code source-resolution-conformance.test.ts}, + * {@code test_source_resolution_conformance.py::test_corpus_is_non_empty}). + */ +public class SourceResolutionCorpusNotEmptyTest { + + @Test + public void corpusIsNonEmpty() throws IOException { + String content = new String( + Files.readAllBytes(SourceResolutionConformanceTest.corpus()), StandardCharsets.UTF_8); + JsonObject root = JsonParser.parseString(content).getAsJsonObject(); + JsonArray cases = root.getAsJsonArray("cases"); + assertTrue("a silent zero-case corpus is a failed gate, not a pass", cases.size() > 0); + } +} diff --git a/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java b/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java index 5fa5314d6..469fc81c8 100644 --- a/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java +++ b/server/java/metadata/src/test/java/com/metaobjects/loader/DirectorySourceTest.java @@ -2,6 +2,7 @@ import org.junit.Test; import java.io.IOException; +import java.io.UncheckedIOException; import java.nio.file.Files; import java.nio.file.Path; import java.util.List; @@ -62,6 +63,111 @@ public class DirectorySourceTest { } } + @Test public void excludePendingIsOffByDefault() throws IOException { + // Loader-level default is OFF (matches TS's loader-level DirectorySource, + // which has no _pending concept at all) — only the CLI-facing SourceResolver + // turns it on. A runtime app embedding `new DirectorySource(dir)` directly + // must see every file, _pending/ included. + Path dir = Files.createTempDirectory("ds-"); + try { + Files.writeString(dir.resolve("meta.live.json"), "{}"); + Path pending = Files.createDirectory(dir.resolve("_pending")); + Files.writeString(pending.resolve("meta.draft.json"), "{}"); + + DirectorySource src = new DirectorySource(dir); + List expanded = src.expand().collect(Collectors.toList()); + + assertEquals(2, expanded.size()); + } finally { + deleteRecursively(dir); + } + } + + @Test public void excludesPendingDirAtAnyDepthWhenOptedIn() throws IOException { + // Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft + // entity under _pending/ must be invisible to codegen, not merely a file + // that happens to be NAMED "_pending". SourceResolver (the CLI-facing + // caller) opts in via setExcludePending(true); this test exercises the + // option directly. + Path dir = Files.createTempDirectory("ds-"); + try { + Files.writeString(dir.resolve("meta.live.json"), "{}"); + Path pending = Files.createDirectory(dir.resolve("_pending")); + Files.writeString(pending.resolve("meta.draft.json"), "{}"); + // Nested: _pending/ excluded at ANY depth, not just top-level. + Path nestedPending = Files.createDirectories(dir.resolve("nested").resolve("_pending")); + Files.writeString(nestedPending.resolve("meta.deep-draft.json"), "{}"); + + DirectorySource src = new DirectorySource(dir, + new DirectorySource.Options().setExcludePending(true)); + List expanded = src.expand().collect(Collectors.toList()); + + assertEquals(1, expanded.size()); + assertEquals("meta.live.json", expanded.get(0).getId()); + } finally { + deleteRecursively(dir); + } + } + + @Test public void followsASymlinkedRoot() throws IOException { + // I1: the SOURCE path itself is a symlink to a directory. Files.isDirectory + // follows symlinks (the existence guard passes), so the walk must too, or + // the root resolves to zero files, silently. + Path real = Files.createTempDirectory("ds-real-"); + Path parent = Files.createTempDirectory("ds-link-parent-"); + Path link = parent.resolve("link"); + try { + Files.writeString(real.resolve("meta.a.json"), "{}"); + Files.createSymbolicLink(link, real); + + List expanded = new DirectorySource(link).expand().collect(Collectors.toList()); + + assertEquals(1, expanded.size()); + assertEquals("meta.a.json", expanded.get(0).getId()); + } finally { + deleteRecursively(parent); + deleteRecursively(real); + } + } + + @Test public void followsASymlinkedSubdirectory() throws IOException { + // I1, second arm: a symlinked SUBDIRECTORY inside a walked tree. + Path dir = Files.createTempDirectory("ds-"); + Path external = Files.createTempDirectory("ds-external-"); + try { + Files.writeString(dir.resolve("meta.top.json"), "{}"); + Files.writeString(external.resolve("meta.linked.json"), "{}"); + Files.createSymbolicLink(dir.resolve("linked"), external); + + List expanded = new DirectorySource(dir).expand().collect(Collectors.toList()); + + assertEquals(2, expanded.size()); + } finally { + deleteRecursively(dir); + deleteRecursively(external); + } + } + + @Test public void aSymlinkCycleFailsLoudlyRatherThanHanging() throws IOException { + Path dir = Files.createTempDirectory("ds-"); + try { + Files.writeString(dir.resolve("meta.top.json"), "{}"); + // A directory symlinked to its own ancestor — a cycle. + Files.createSymbolicLink(dir.resolve("loop"), dir); + + try { + new DirectorySource(dir).expand().collect(Collectors.toList()); + fail("expected a symlink-loop failure, not a completed walk"); + } catch (UncheckedIOException expected) { + // Files.walk(FOLLOW_LINKS) detects the cycle and throws + // FileSystemLoopException, naming the looping path. + assertNotNull(expected.getCause()); + } + } finally { + deleteRecursively(dir); + } + } + @Test public void nonRecursiveSkipsSubdirectories() throws IOException { Path dir = Files.createTempDirectory("ds-"); try { @@ -79,7 +185,11 @@ public class DirectorySourceTest { } private static void deleteRecursively(Path p) throws IOException { - if (Files.isDirectory(p)) { + // NOFOLLOW_LINKS: a symlink (incl. one deliberately cyclic, as in + // aSymlinkCycleFailsLoudlyRatherThanHanging above) is a leaf to delete + // outright, never a directory to recurse INTO — Files.isDirectory's default + // symlink-following would walk right back into the loop during cleanup. + if (Files.isDirectory(p, java.nio.file.LinkOption.NOFOLLOW_LINKS)) { try (var s = Files.list(p)) { for (Path c : s.collect(Collectors.toList())) deleteRecursively(c); } diff --git a/server/python/src/metaobjects/cli.py b/server/python/src/metaobjects/cli.py index 02ea2418f..1a337d0eb 100644 --- a/server/python/src/metaobjects/cli.py +++ b/server/python/src/metaobjects/cli.py @@ -47,6 +47,7 @@ from pathlib import Path from metaobjects import MetaDataLoader +from metaobjects.errors import ParseError from metaobjects.agent_context import ( AGENT_CONTEXT_MANIFEST_PATH, agent_context_staleness, @@ -274,6 +275,35 @@ def _load_root( return result.root, [] +def _load_root_from_paths( + paths: list[str], + strict: bool = False, + providers: list[object] | None = None, +) -> tuple[MetaData | None, list[str]]: + """Load metadata from an explicit file list rather than a single directory. + + The source-resolution ladder's ``.metaobjects/config.json`` rung + (:func:`resolve_metadata_location`) can resolve to several directories or + individual files, which a single ``from_directory`` call cannot express — + so this loads each resolved file as its own ``file://`` source via + :meth:`MetaDataLoader.from_uris`. Mirrors :func:`_load_root`'s ``strict``/ + ``providers`` contract exactly. + """ + uris = [Path(p).resolve().as_uri() for p in paths] + if providers: + from metaobjects.core_types import core_providers + + result = MetaDataLoader.from_uris( + uris, providers=[*core_providers, *providers], strict=strict + ) + else: + result = MetaDataLoader.from_uris(uris, strict=strict) + if result.errors: + msgs = [f"{e.code}: {e.message}" for e in result.errors] + return None, msgs + return result.root, [] + + def _strict_load_hint() -> str: """Actionable next-steps when strict verify rejects an undeclared @attr.""" return ( @@ -407,6 +437,65 @@ def gen_state_dir_for(metadata_dir: str) -> str: return str(Path(metadata_dir).resolve().parent / ".metaobjects" / ".gen-state") +def resolve_metadata_location( + config: ProjectConfig | None, + root: Path, +) -> list[str]: + """The precedence ladder for where metadata lives, rungs 2-4. First match wins. + + 2. This port's native surface — ``metadata`` in ``metaobjects.config.yaml``. + 3. ``sources`` in the port-neutral ``.metaobjects/config.json``. + 4. The built-in default directory. + + Rung 1 — an explicit CLI argument (the positional ``metadata_dir``) — is + NOT this function's concern: every command handler that accepts one already + loads it directly via ``_load_root(args.metadata_dir)`` (``MetaDataLoader + .from_directory``, a single-directory load with no resolve-then-rejoin step + of its own) BEFORE this function would even be reachable, so this function is + only ever called with rung 1 already having been tried and found absent. An + earlier revision accepted an ``explicit`` parameter and duplicated rung 1 + here via `resolve_sources` — reachable from no command handler, so it could + silently drift from the real rung-1 path (e.g. picking up + `source_resolver.py`'s CLI-facing `_pending` exclusion while + `_load_root`'s loader-level walk does not) with nothing to notice. Removed + rather than wired: replumbing every metadataDir-taking command onto + `resolve_sources`'s `from_uris`-based load instead of `from_directory` is a + materially different, higher-risk change than this ladder needs. + + A file that EXISTS at any rung but is malformed raises rather than falling + through to the next rung. See + `docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §5. + """ + from metaobjects.config.source_resolver import resolve_collection, resolve_sources + + if config is not None: + # `config.metadata_dir()` is already resolved to an absolute path + # (`ProjectConfig._resolve_under`), so the base passed here is + # likewise irrelevant. + return [ + str(p) for p in resolve_sources(root, [{"path": config.metadata_dir()}]) + ] + + # Rungs 3 and 4 both live in `resolve_collection`. + return [str(p) for p in resolve_collection(root)] + + +def _resolve_metadata_location_or_print_error( + config: ProjectConfig | None, root: Path +) -> list[str] | None: + """``resolve_metadata_location`` for the no-explicit-``metadata_dir`` CLI + paths (``docs``, and the ``gen``/``verify --codegen`` neutral fallbacks), + translating a raised ``ParseError`` into this CLI's print-and-return-1 + convention instead of letting it propagate. Returns ``None`` on failure — + the caller has nothing further to print and should return 1. + """ + try: + return resolve_metadata_location(config=config, root=root) + except ParseError as exc: + print(f"error: could not resolve metadata location: {exc}", file=sys.stderr) + return None + + #: The default api-surface subdir (the cross-port contract's ``api/python``). _DOCS_DEFAULT_API_SUBDIR = "api/python" @@ -433,7 +522,33 @@ def _cmd_docs(args: argparse.Namespace) -> int: providers, providers_ok = _providers_from_args(args) if not providers_ok: return 1 - root, errors = _load_root(args.metadata_dir, providers=providers) + + if args.metadata_dir is None: + # Mirrors `gen`'s no-positional routing (#267): rung 2 (`metadata` in + # `metaobjects.config.yaml`, if one is found), else rungs 3-4 via + # `resolve_metadata_location` (the neutral `.metaobjects/config.json` + # `sources` key this feature adds, else the built-in default + # directory). Before this, `metadata_dir` was a REQUIRED positional + # here, so this ladder was unreachable from `docs` even though `gen` + # and `verify --codegen` could both already reach it. + root_dir = Path.cwd() + config: ProjectConfig | None = None + config_path = _find_config(args) + if config_path is not None: + try: + config = load_project_config(config_path) + except ConfigError as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + paths = _resolve_metadata_location_or_print_error(config, root_dir) + if paths is None: + return 1 + root, errors = _load_root_from_paths(paths, providers=providers) + project_default = root_dir.name + else: + root, errors = _load_root(args.metadata_dir, providers=providers) + project_default = Path(args.metadata_dir).resolve().name + if root is None: print("error: failed to load metadata:", file=sys.stderr) for msg in errors: @@ -441,7 +556,7 @@ def _cmd_docs(args: argparse.Namespace) -> int: return 1 api_subdir = args.api_subdir or _DOCS_DEFAULT_API_SUBDIR - project = args.project or Path(args.metadata_dir).resolve().name + project = args.project or project_default model_base_url = getattr(args, "model_base_url", None) model = PythonApiModelBuilder().build(root, project) @@ -652,22 +767,71 @@ def _run_gen_targets( return all_written, errors +def _cmd_gen_neutral_fallback(args: argparse.Namespace) -> int: + """``gen`` with no positional ``metadata_dir`` AND no ``metaobjects.config.yaml``. + + Descends the source-resolution ladder's remaining rungs — ``sources`` in + ``.metaobjects/config.json``, else the built-in default directory — via + :func:`resolve_metadata_location`. There is no declarative target registry + at this rung (that is what a ``metaobjects.config.yaml`` provides), so + ``--out`` is required exactly as it is in explicit-```` flag + mode; this function otherwise mirrors ``_cmd_gen``'s flag-mode body, minus + the ``--template-spec`` pass (out of scope for this rung). + """ + if args.out is None: + print( + "error: gen requires and --out " + "(or a metaobjects.config.yaml / --list).", + file=sys.stderr, + ) + return 2 + + root_dir = Path.cwd() + paths = _resolve_metadata_location_or_print_error(None, root_dir) + if paths is None: + return 1 + + generators: list[Generator] | None = None + if args.generators: + generators, gen_errors = _resolve_generators(args.generators) + if gen_errors: + print("error: invalid --generators selection:", file=sys.stderr) + for msg in gen_errors: + print(f" {msg}", file=sys.stderr) + return 1 + + entities = _parse_entities(getattr(args, "entities", None)) + providers, providers_ok = _providers_from_args(args) + if not providers_ok: + return 1 + + root, load_errors = _load_root_from_paths(paths, providers=providers) + if root is None: + print("error: failed to load metadata:", file=sys.stderr) + for msg in load_errors: + print(f" {msg}", file=sys.stderr) + return 1 + + gen_state = str(root_dir.resolve() / ".metaobjects" / ".gen-state") + written = _run_suite(root, args.out, generators, entities, gen_state_dir=gen_state) + for path in written: + print(path) + print(f"metaobjects gen: wrote {len(written)} file(s) to {args.out}") + return 0 + + def _cmd_gen_config(args: argparse.Namespace) -> int: """``gen`` with no positional ``metadata_dir`` → declarative-config mode (#267). Load ``metaobjects.config.yaml``, load metadata ONCE, and run every target (or ``--target``) into its own ``outDir`` with a cross-target duplicate-path guard. Providers resolve relative to the config file (no ``PYTHONPATH=``). + No ``metaobjects.config.yaml`` at all → :func:`_cmd_gen_neutral_fallback` + (the ladder's remaining rungs), not an error. """ config_path = _find_config(args) if config_path is None: - print( - "error: no given and no metaobjects.config.yaml found. " - "Either pass --out (flag mode) or create a " - "metaobjects.config.yaml (or pass --config ).", - file=sys.stderr, - ) - return 2 + return _cmd_gen_neutral_fallback(args) try: config = load_project_config(config_path) except ConfigError as exc: @@ -721,6 +885,38 @@ def _relative_set(root: Path) -> dict[str, str]: return files +def _diff_report(expected: dict[str, str], committed: dict[str, str]) -> int: + """Compare a regenerated file map against the committed one; print the + standard ``verify --codegen`` drift report. Returns 0 (in sync) or 1. + + Extracted so the explicit-```` flag mode + (:func:`_verify_codegen`) and the ``.metaobjects/config.json`` fallback + rung (:func:`_verify_codegen_neutral_fallback`) report drift identically. + """ + changed = sorted( + k for k in expected if k in committed and expected[k] != committed[k] + ) + missing = sorted(k for k in expected if k not in committed) # not yet committed + extra = sorted(k for k in committed if k not in expected) # stale committed file + + if not changed and not missing and not extra: + print(f"metaobjects verify: in sync ({len(expected)} file(s)).") + return 0 + + print("error: generated code is out of sync with metadata.", file=sys.stderr) + for k in changed: + print(f" drifted: {k}", file=sys.stderr) + for k in missing: + print(f" missing: {k}", file=sys.stderr) + for k in extra: + print(f" extra: {k}", file=sys.stderr) + print( + "regenerate (metaobjects gen) and commit the result.", + file=sys.stderr, + ) + return 1 + + def _verify_codegen(args: argparse.Namespace) -> int: """``verify --codegen`` — regenerate to a temp dir + diff vs committed ``--out``. @@ -764,28 +960,52 @@ def _verify_codegen(args: argparse.Namespace) -> int: expected = _relative_set(Path(tmp)) committed = _relative_set(Path(args.out)) - changed = sorted( - k for k in expected if k in committed and expected[k] != committed[k] - ) - missing = sorted(k for k in expected if k not in committed) # not yet committed - extra = sorted(k for k in committed if k not in expected) # stale committed file + return _diff_report(expected, committed) - if not changed and not missing and not extra: - print(f"metaobjects verify: in sync ({len(expected)} file(s)).") - return 0 - print("error: generated code is out of sync with metadata.", file=sys.stderr) - for k in changed: - print(f" drifted: {k}", file=sys.stderr) - for k in missing: - print(f" missing: {k}", file=sys.stderr) - for k in extra: - print(f" extra: {k}", file=sys.stderr) - print( - "regenerate (metaobjects gen) and commit the result.", - file=sys.stderr, - ) - return 1 +def _verify_codegen_neutral_fallback(args: argparse.Namespace) -> int: + """``verify --codegen`` with no positional ``metadata_dir`` AND no + ``metaobjects.config.yaml``. + + Descends the source-resolution ladder's remaining rungs via + :func:`resolve_metadata_location`, exactly like + :func:`_cmd_gen_neutral_fallback`. There is no declarative target registry + at this rung, so ``--out`` (the committed dir to diff against) is required + exactly as it is in explicit-```` flag mode. + """ + if args.out is None: + print( + "error: verify --codegen requires --out (the committed output dir).", + file=sys.stderr, + ) + return 2 + + root_dir = Path.cwd() + paths = _resolve_metadata_location_or_print_error(None, root_dir) + if paths is None: + return 1 + + strict = not getattr(args, "lax", False) + providers, providers_ok = _providers_from_args(args) + if not providers_ok: + return 1 + + with tempfile.TemporaryDirectory() as tmp: + entities = _parse_entities(getattr(args, "entities", None)) + root, load_errors = _load_root_from_paths(paths, strict=strict, providers=providers) + if root is None: + print("error: failed to load metadata:", file=sys.stderr) + for msg in load_errors: + print(f" {msg}", file=sys.stderr) + if strict and any("ERR_UNKNOWN_ATTR" in m for m in load_errors): + print(_strict_load_hint(), file=sys.stderr) + return 1 + + _run_suite(root, tmp, None, entities, gen_state_dir=None) + expected = _relative_set(Path(tmp)) + committed = _relative_set(Path(args.out)) + + return _diff_report(expected, committed) def _temp_slot_for(temp_root: Path, real_outdir: str, config_dir: Path) -> str: @@ -820,15 +1040,12 @@ def _verify_codegen_config(args: argparse.Namespace) -> int: drift (mirrors the TS ``computeCodegenDrift`` unit = unique outDir). ``--target`` widens to the outDir-sharing closure (an outDir is verified as a unit). Strict-by-default (ADR-0023) unless ``--lax``. + No ``metaobjects.config.yaml`` at all → :func:`_verify_codegen_neutral_fallback` + (the ladder's remaining rungs), not an error. """ config_path = _find_config(args) if config_path is None: - print( - "error: verify --codegen with no requires a " - "metaobjects.config.yaml (or --config ).", - file=sys.stderr, - ) - return 2 + return _verify_codegen_neutral_fallback(args) try: config = load_project_config(config_path) except ConfigError as exc: @@ -1240,7 +1457,16 @@ def _build_parser() -> argparse.ArgumentParser: "surface of the cross-port SDK-docs contract) under --out" ), ) - docs.add_argument("metadata_dir", help="directory of metadata JSON/YAML files") + # Optional, like `gen`'s: an omitted positional falls through the same + # location ladder (`metaobjects.config.yaml`'s `metadata` key, else the + # neutral `.metaobjects/config.json` `sources`, else the default directory) + # instead of hard-requiring the caller to name a directory. + docs.add_argument( + "metadata_dir", + nargs="?", + default=None, + help="directory of metadata JSON/YAML files (omit to use the config ladder)", + ) docs.add_argument( "--out", required=True, diff --git a/server/python/src/metaobjects/config/__init__.py b/server/python/src/metaobjects/config/__init__.py new file mode 100644 index 000000000..c0881621e --- /dev/null +++ b/server/python/src/metaobjects/config/__init__.py @@ -0,0 +1,17 @@ +"""Port-neutral `.metaobjects/config.json` reading and source resolution. + +Reads only the NEUTRAL SUBSET (`schema_version`, `sources`). The file also +carries TypeScript-owned keys; those are ignored rather than modeled, so a new +TS-only key never becomes a four-port change. See +`docs/superpowers/specs/2026-08-19-cross-port-metadata-sources-design.md` §4. +""" +from .neutral_config import DEFAULT_METADATA_DIR, NeutralConfig, read_neutral_config +from .source_resolver import resolve_collection, resolve_sources + +__all__ = [ + "DEFAULT_METADATA_DIR", + "NeutralConfig", + "read_neutral_config", + "resolve_collection", + "resolve_sources", +] diff --git a/server/python/src/metaobjects/config/neutral_config.py b/server/python/src/metaobjects/config/neutral_config.py new file mode 100644 index 000000000..b1075ba0e --- /dev/null +++ b/server/python/src/metaobjects/config/neutral_config.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path + +from metaobjects.errors import ErrorCode, ParseError + +#: The DEFAULT value of `sources` when the key is absent or empty — never a +#: requirement, and never assumed to exist by any other code path. +DEFAULT_METADATA_DIR = "metaobjects" + +_METAOBJECTS_DIR = ".metaobjects" +_CONFIG_FILE = "config.json" + + +@dataclass(frozen=True) +class NeutralConfig: + """The port-neutral subset of `.metaobjects/config.json`.""" + + #: Raw source specs, each a single-key mapping (`path` / `resource` / `package`). + sources: list[dict[str, str]] + + +def read_neutral_config(config_dir: Path) -> NeutralConfig | None: + """Read the neutral subset from ``config_dir/.metaobjects/config.json``. + + Returns ``None`` when the file does not exist. A file that EXISTS but is + malformed raises — swallowing it would make a typo'd config behave + identically to no config at all, silently resolving a possibly-stale + default directory with no diagnostic. + """ + path = config_dir / _METAOBJECTS_DIR / _CONFIG_FILE + if not path.is_file(): + return None + + try: + raw = json.loads(path.read_text()) + except (OSError, ValueError) as e: + raise ParseError( + f"{path} exists but could not be read as JSON: {e}", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) from e + + if not isinstance(raw, dict): + raise ParseError( + f"{path} must contain a JSON object", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + version = raw.get("schema_version") + if version != 1: + raise ParseError( + f"{path}: unsupported schema_version {version!r} (expected 1)", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + # `.get("sources", [])` only applies the [] default when the key is ABSENT — + # a present `sources: null` returns None here, which correctly fails the + # `isinstance(sources, list)` check below rather than silently reading as + # "absent" and falling back to the default directory with no diagnostic. + sources = raw.get("sources", []) + if not isinstance(sources, list) or not all( + isinstance(s, dict) + and len(s) == 1 + # Every source-spec value (`path`/`resource`/`package`) must be a + # non-empty (after stripping whitespace) string — a bare number/ + # boolean/null would otherwise reach `Path()` downstream and raise an + # uncaught TypeError instead of this coded error, and an empty or + # whitespace-only `path` would resolve to the config-holding directory + # itself rather than failing loudly on the typo'd config. + and all(isinstance(v, str) and v.strip() for v in s.values()) + for s in sources + ): + raise ParseError( + f"{path}: 'sources' must be an array of single-key objects, each " + "value a non-empty string", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + + # Unknown top-level keys are IGNORED by design — see the module docstring. + return NeutralConfig(sources=[dict(s) for s in sources]) diff --git a/server/python/src/metaobjects/config/source_resolver.py b/server/python/src/metaobjects/config/source_resolver.py new file mode 100644 index 000000000..4f5ea94af --- /dev/null +++ b/server/python/src/metaobjects/config/source_resolver.py @@ -0,0 +1,116 @@ +from __future__ import annotations + +import os +from pathlib import Path + +from metaobjects.errors import ErrorCode, ParseError +from metaobjects.loader.sources import DirectorySource + +from .neutral_config import DEFAULT_METADATA_DIR, read_neutral_config + + +def _list_metadata_files(directory: Path) -> list[Path]: + """Recursively list metadata files under ``directory``. + + Delegates to the loader's own `DirectorySource` — the SAME code the loader + uses to turn a directory into metadata files — rather than re-walking with + a second, driftable definition of "which files count as metadata" (extension + set). Order is this port's own and is deliberately NOT a cross-port contract + — see the corpus README. + + `exclude_pending=True`: this IS the CLI-facing resolver — `_pending/` is the + TypeScript CLI's pending/promote-workflow concept, not a loader concept, so + the loader-level `DirectorySource` default (off) is overridden here, the one + place this port's CLI turns it on. + """ + return [fs.path for fs in DirectorySource(directory, exclude_pending=True).expand()] + + +def _normalize(p: Path) -> Path: + """Absolute + lexically normalized (``.``/``..`` collapsed), WITHOUT + resolving symlinks. + + `Path.resolve()` does both jobs at once — and following symlinks here is + the wrong half: it would silently rewrite a source declared as a symlink + (e.g. `sources: [{"path": "link"}]` where `link -> real`) to its target's + real name, diverging from the other three ports, none of which collapse a + walked path's symlinked directory components (Java's + `toAbsolutePath().normalize()`, C#'s `Path.GetFullPath()`, TypeScript's + `path.resolve()` are all lexical-only, like this). `os.path.abspath` is + exactly that: anchor to cwd if relative, then `normpath` — no filesystem + symlink lookups. + """ + return Path(os.path.abspath(p)) + + +def _validate_kinds(specs: list[dict[str, str]]) -> None: + """Validate every spec's kind before ANY filesystem access. + + Mirrors `sources.ts`'s `orderedPathSpecs` (`.map(toPathSpec)` runs over the + whole list before `resolveSources` performs a single `stat()`): a kind + check interleaved with resolution, one spec at a time, would make which + error code comes back depend on declaration order — `{"path": "nope"}, + {"resource": "x"}` and its reverse must both report + `ERR_SOURCE_KIND_UNSUPPORTED`, never `ERR_SOURCE_UNRESOLVED` on one + ordering and the kind error on the other. + """ + for spec in specs: + if "path" not in spec: + kind = next(iter(spec), "") + raise ParseError( + f'source kind "{kind}" is not supported by this toolchain yet; use a "path" source', + code=ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED, + ) + + +def resolve_sources(config_dir: Path, specs: list[dict[str, str]]) -> list[Path]: + """Resolve a declared source SET to a de-duplicated list of metadata files. + + A relative ``path`` resolves against ``config_dir`` — the directory HOLDING + the ``.metaobjects/`` folder — never against the process working directory. + """ + # Whole-list kind validation FIRST — see `_validate_kinds`. + _validate_kinds(specs) + + seen: dict[Path, None] = {} + + for spec in specs: + raw = Path(spec["path"]) + target = raw if raw.is_absolute() else (config_dir / raw) + + if not target.exists(): + raise ParseError( + f'source path "{spec["path"]}" does not exist ' + f"(resolved to {target}, relative to {config_dir})", + code=ErrorCode.ERR_SOURCE_UNRESOLVED, + ) + + found = _list_metadata_files(target) if target.is_dir() else [target] + for f in found: + seen.setdefault(_normalize(f), None) + + return list(seen) + + +def resolve_collection(root: Path) -> list[Path]: + """The full ladder: declared `sources`, else the default directory. + + Only the DEFAULT may be absent — a declared source that does not resolve is + `ERR_SOURCE_UNRESOLVED`, a louder failure. + """ + root = root.resolve() + cfg = read_neutral_config(root) + specs = cfg.sources if cfg is not None and cfg.sources else [] + + if not specs: + default_dir = root / DEFAULT_METADATA_DIR + if not default_dir.is_dir(): + raise ParseError( + f'no metadata sources declared in {root} and no default ' + f'"{DEFAULT_METADATA_DIR}" directory found. Declare "sources" in ' + f".metaobjects/config.json, or run 'meta init' to scaffold.", + code=ErrorCode.ERR_COLLECTION_NOT_FOUND, + ) + specs = [{"path": DEFAULT_METADATA_DIR}] + + return resolve_sources(root, specs) diff --git a/server/python/src/metaobjects/loader/sources/__init__.py b/server/python/src/metaobjects/loader/sources/__init__.py index 977a2d647..2f5a9fb14 100644 --- a/server/python/src/metaobjects/loader/sources/__init__.py +++ b/server/python/src/metaobjects/loader/sources/__init__.py @@ -9,7 +9,7 @@ """ from __future__ import annotations -from .directory_source import DirectorySource +from .directory_source import DirectorySource, SymlinkLoopError from .file_source import FileSource from .meta_data_source import InMemoryStringSource, MetaDataFormat, MetaDataSource from .uri_source import UriSource @@ -20,5 +20,6 @@ "InMemoryStringSource", "FileSource", "DirectorySource", + "SymlinkLoopError", "UriSource", ] diff --git a/server/python/src/metaobjects/loader/sources/directory_source.py b/server/python/src/metaobjects/loader/sources/directory_source.py index afc609856..0c6ee81a0 100644 --- a/server/python/src/metaobjects/loader/sources/directory_source.py +++ b/server/python/src/metaobjects/loader/sources/directory_source.py @@ -15,6 +15,23 @@ _SUPPORTED_SUFFIXES = (".json", ".yaml", ".yml") +#: Directory excluded at every level of a recursive expand() — drafts that are +#: deliberately not part of the loaded model. Mirrors TypeScript's +#: `PENDING_DIR` in `metadata-files.ts`. OFF by default — see +#: `DirectorySource.__init__`'s `exclude_pending` doc. +_PENDING_DIR = "_pending" + + +class SymlinkLoopError(OSError): + """Raised when expand() finds a directory symlink that revisits an ancestor. + + A clear error beats a hang: expand() follows symlinked directories (I1 — + matching TypeScript's `stat`-not-`lstat` walk and C#'s + `EnumerateFiles(..., AllDirectories)`, both of which have always followed + them), so an accidental or malicious symlink cycle must be caught explicitly + rather than recursing forever. + """ + class DirectorySource: """Expands a directory into a sorted, filtered list of FileSource objects.""" @@ -24,27 +41,73 @@ def __init__( directory: Path | str, exclude: Iterable[str] | None = None, recurse: bool = True, + exclude_pending: bool = False, ) -> None: + """``exclude_pending`` (I2) — exclude ``_pending/`` at any depth. + + Default ``False``: this is a LOADER-level primitive, and ``_pending/`` is + a CLI/pending-promote-workflow concept (TypeScript's `metadata-files.ts`, + not its loader-level `DirectorySource`, which has no `_pending` concept + at all). `source_resolver.py` — the CLI-facing caller — turns this ON + explicitly rather than baking the exclusion into every embedder of this + class; an app calling ``DirectorySource(dir)`` directly gets every file + back, matching the reference loader. + """ self._directory = Path(directory) self._exclude = set(exclude or ()) self._recurse = recurse + self._exclude_pending = exclude_pending @property def directory(self) -> Path: return self._directory def expand(self) -> Iterator[FileSource]: - candidates = ( - self._directory.rglob("*") if self._recurse else self._directory.iterdir() - ) - files = sorted( - ( - p - for p in candidates - if p.is_file() - and p.suffix.lower() in _SUPPORTED_SUFFIXES - and p.name not in self._exclude - ), - key=lambda p: p.name, - ) + files = sorted(self._collect(self._directory, frozenset()), key=lambda p: p.name) yield from (FileSource(p) for p in files) + + def _collect(self, directory: Path, ancestors: frozenset[Path]) -> list[Path]: + """Recursively collect matching files under ``directory``, following + symlinked subdirectories (I1) — `iterdir()`/`is_dir()` naturally follow a + symlink, unlike `rglob("*")`'s `**` traversal, which does not descend + into one. + + Paths are built by lexical join (`directory / name`) throughout, exactly + like a plain non-symlink-aware walk would build them — a symlinked + directory's OWN name survives in the reported path; only the WALK follows + the link, matching Java/C#/TypeScript, none of which collapse a source's + directory name to its symlink target's real name. + + ``ancestors`` is a set of REAL (`Path.resolve()`) directory paths already + on the current walk branch, used only to detect a symlink cycle — never + to rewrite a reported path. + """ + real = directory.resolve() + if real in ancestors: + raise SymlinkLoopError( + f"symlink loop detected while expanding {self._directory}: " + f"{directory} revisits {real}" + ) + ancestors = ancestors | {real} + + out: list[Path] = [] + for entry in directory.iterdir(): + if entry.is_dir(): + if self._recurse: + out.extend(self._collect(entry, ancestors)) + elif ( + entry.is_file() + and entry.suffix.lower() in _SUPPORTED_SUFFIXES + and entry.name not in self._exclude + # Excludes _pending/ at ANY depth — a directory NAME check on every + # ancestor component between `self._directory` and `entry`, not + # merely a basename filter on `entry` itself, so the whole subtree + # is skipped (a draft entity must be invisible to codegen, not + # just a file that happens to be named "_pending"). + and not ( + self._exclude_pending + and _PENDING_DIR in entry.relative_to(self._directory).parts[:-1] + ) + ): + out.append(entry) + return out diff --git a/server/python/tests/codegen/test_cli_config_gen.py b/server/python/tests/codegen/test_cli_config_gen.py index fbba3572a..26f9f8078 100644 --- a/server/python/tests/codegen/test_cli_config_gen.py +++ b/server/python/tests/codegen/test_cli_config_gen.py @@ -222,3 +222,45 @@ def test_gen_flag_path_ignores_config_when_present(tmp_path: Path) -> None: rc = main(["gen", str(meta), "--out", str(out)]) assert rc == 0 assert (out / "Program.py").exists() + + +def test_gen_no_args_no_yaml_falls_back_to_neutral_config( + tmp_path: Path, monkeypatch +) -> None: + """No positional AND no metaobjects.config.yaml anywhere: + `gen` must descend to the `.metaobjects/config.json` `sources` rung + (source-resolution ladder rung 3) rather than erroring, per fix round 1. + + The metadata lives under `model/`, NOT the built-in default `metaobjects/` + directory — so this only passes if the declared `sources` path is actually + consulted, not a coincidental default-directory hit. + """ + model = tmp_path / "model" + model.mkdir() + (model / "meta.fitness.json").write_text(FITNESS.read_text()) + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + + monkeypatch.chdir(tmp_path) + rc = main(["gen", "--out", "gen/models", "--generators", "entity"]) + assert rc == 0 + assert (tmp_path / "gen/models/Program.py").exists() + + +def test_gen_no_args_no_config_at_all_falls_back_to_default_directory( + tmp_path: Path, monkeypatch +) -> None: + """No positional , no metaobjects.config.yaml, AND no + `.metaobjects/config.json`: `gen` descends all the way to the ladder's + built-in default directory (`metaobjects/`) rather than erroring.""" + meta = tmp_path / "metaobjects" + meta.mkdir() + (meta / "meta.fitness.json").write_text(FITNESS.read_text()) + + monkeypatch.chdir(tmp_path) + rc = main(["gen", "--out", "gen/models", "--generators", "entity"]) + assert rc == 0 + assert (tmp_path / "gen/models/Program.py").exists() diff --git a/server/python/tests/codegen/test_cli_config_verify.py b/server/python/tests/codegen/test_cli_config_verify.py index 6b7acb817..b427ba627 100644 --- a/server/python/tests/codegen/test_cli_config_verify.py +++ b/server/python/tests/codegen/test_cli_config_verify.py @@ -152,3 +152,38 @@ def test_verify_target_scoping_widens_to_shared_outdir(tmp_path: Path, capsys) - rc = main(["verify", "--codegen", "--config", str(cfg), "--target", "a"]) assert rc == 1 assert "Week.py" in capsys.readouterr().err + + +def test_verify_codegen_no_args_no_yaml_falls_back_to_neutral_config( + tmp_path: Path, monkeypatch +) -> None: + """No positional AND no metaobjects.config.yaml anywhere: + `verify --codegen` must descend to the `.metaobjects/config.json` `sources` + rung rather than erroring, per fix round 1 — and must agree with `gen` + taking the SAME rung (a fresh gen is in sync; drift is still caught). + + The metadata lives under `model/`, NOT the built-in default `metaobjects/` + directory, so this only passes if the declared `sources` path is actually + consulted by BOTH commands. + """ + model = tmp_path / "model" + model.mkdir() + (model / "meta.fitness.json").write_text(FITNESS.read_text()) + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + '{"schema_version": 1, "sources": [{"path": "model"}]}' + ) + + monkeypatch.chdir(tmp_path) + # No --generators here: `verify --codegen` has no such flag (it always + # regenerates the full default suite), so `gen` must run the full suite + # too or the diff reports the un-emitted generators as spurious drift — + # same constraint the flag-mode docstring notes for --entities. + assert main(["gen", "--out", "gen/models"]) == 0 + # Fresh gen -> no drift, via the same fallback rung. + assert main(["verify", "--codegen", "--out", "gen/models"]) == 0 + # Drift the committed output; the fallback rung must still catch it. + program = tmp_path / "gen/models/Program.py" + program.write_text(program.read_text() + "\n# drift\n") + assert main(["verify", "--codegen", "--out", "gen/models"]) == 1 diff --git a/server/python/tests/config/__init__.py b/server/python/tests/config/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/server/python/tests/config/test_neutral_config.py b/server/python/tests/config/test_neutral_config.py new file mode 100644 index 000000000..2f9f58e64 --- /dev/null +++ b/server/python/tests/config/test_neutral_config.py @@ -0,0 +1,92 @@ +import json +import pytest +from pathlib import Path + +from metaobjects.config.neutral_config import read_neutral_config +from metaobjects.errors import ErrorCode, ParseError + + +def _write_config(root: Path, payload: object) -> None: + d = root / ".metaobjects" + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(payload)) + + +def test_absent_config_returns_none(tmp_path: Path) -> None: + assert read_neutral_config(tmp_path) is None + + +def test_reads_sources(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": "model"}]}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "model"}] + + +def test_unknown_top_level_keys_are_ignored(tmp_path: Path) -> None: + # The file carries TypeScript-owned keys this port must not model. + _write_config( + tmp_path, + { + "schema_version": 1, + "sources": [{"path": "model"}], + "pending_in_git": True, + "confidence_thresholds": {"pending_promote": 0.8}, + "extract": {"metaignore": ".metaignore"}, + "migrate": {"dialect": "postgres"}, + }, + ) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [{"path": "model"}] + + +def test_absent_sources_key_yields_empty_list(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 1}) + cfg = read_neutral_config(tmp_path) + assert cfg is not None + assert cfg.sources == [] + + +def test_malformed_json_raises_not_none(tmp_path: Path) -> None: + d = tmp_path / ".metaobjects" + d.mkdir(parents=True) + (d / "config.json").write_text("{ not json") + # A file that EXISTS but cannot be read must never look like no config at all. + with pytest.raises(ParseError) as e: + read_neutral_config(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND + + +def test_wrong_schema_version_raises(tmp_path: Path) -> None: + _write_config(tmp_path, {"schema_version": 2, "sources": []}) + with pytest.raises(ParseError) as e: + read_neutral_config(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND + + +def test_null_sources_raises(tmp_path: Path) -> None: + # A present `sources: null` is present-but-wrong-typed, same as a bare + # object — it must not silently read as "absent" (see the shared + # source-resolution-conformance corpus for the cross-port pin of this). + _write_config(tmp_path, {"schema_version": 1, "sources": None}) + with pytest.raises(ParseError): + read_neutral_config(tmp_path) + + +def test_whitespace_only_path_raises(tmp_path: Path) -> None: + # Deliberately NOT gated by the shared cross-port corpus: the TS reference + # (`config.ts`'s `z.string().min(1)`) rejects only a fully-empty path, not + # a whitespace-only one, and the reference is out of scope to change here. + # This port is stricter on this one edge case by design. + _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": " "}]}) + with pytest.raises(ParseError): + read_neutral_config(tmp_path) + + +def test_non_string_source_value_raises(tmp_path: Path) -> None: + # A bare number must fail here, at the config-read boundary, rather than + # reaching `Path()` downstream and raising an uncaught TypeError. + _write_config(tmp_path, {"schema_version": 1, "sources": [{"path": 123}]}) + with pytest.raises(ParseError): + read_neutral_config(tmp_path) diff --git a/server/python/tests/config/test_source_resolver.py b/server/python/tests/config/test_source_resolver.py new file mode 100644 index 000000000..00bab4bf8 --- /dev/null +++ b/server/python/tests/config/test_source_resolver.py @@ -0,0 +1,91 @@ +import json +import pytest +from pathlib import Path + +from metaobjects.config.source_resolver import resolve_collection, resolve_sources +from metaobjects.errors import ErrorCode, ParseError + + +def _rel(root: Path, files: list[Path]) -> set[str]: + return {p.relative_to(root).as_posix() for p in files} + + +def test_directory_is_walked_recursively(tmp_path: Path) -> None: + (tmp_path / "model" / "nested").mkdir(parents=True) + (tmp_path / "model" / "a.json").write_text("{}") + (tmp_path / "model" / "nested" / "b.yaml").write_text("{}") + (tmp_path / "model" / "README.md").write_text("x") + got = resolve_sources(tmp_path, [{"path": "model"}]) + assert _rel(tmp_path, got) == {"model/a.json", "model/nested/b.yaml"} + + +def test_single_file_spec(tmp_path: Path) -> None: + (tmp_path / "vendor").mkdir() + (tmp_path / "vendor" / "one.json").write_text("{}") + (tmp_path / "vendor" / "two.json").write_text("{}") + got = resolve_sources(tmp_path, [{"path": "vendor/one.json"}]) + assert _rel(tmp_path, got) == {"vendor/one.json"} + + +def test_overlapping_sources_dedupe(tmp_path: Path) -> None: + (tmp_path / "model" / "nested").mkdir(parents=True) + (tmp_path / "model" / "a.json").write_text("{}") + (tmp_path / "model" / "nested" / "b.json").write_text("{}") + got = resolve_sources(tmp_path, [{"path": "model"}, {"path": "model/nested"}]) + assert _rel(tmp_path, got) == {"model/a.json", "model/nested/b.json"} + assert len(got) == 2 + + +def test_missing_path_raises_unresolved(tmp_path: Path) -> None: + with pytest.raises(ParseError) as e: + resolve_sources(tmp_path, [{"path": "nope"}]) + assert e.value.code == ErrorCode.ERR_SOURCE_UNRESOLVED + + +def test_resource_kind_unsupported(tmp_path: Path) -> None: + with pytest.raises(ParseError) as e: + resolve_sources(tmp_path, [{"resource": "com/acme"}]) + assert e.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED + + +def test_kind_error_precedes_unresolved_path_regardless_of_order(tmp_path: Path) -> None: + # Kind validation runs across the WHOLE spec list before any filesystem + # access, so an unsupported-kind spec always wins over an unresolved-path + # spec — in EITHER declaration order. Interleaving the two checks one spec + # at a time (kind-check-then-stat, per spec) would make the reported error + # code depend on which spec happens to come first; pinned against the + # TypeScript reference (`sources.ts` `orderedPathSpecs`), verified + # empirically to raise ERR_SOURCE_KIND_UNSUPPORTED in both orders. + with pytest.raises(ParseError) as e_missing_first: + resolve_sources(tmp_path, [{"path": "nope"}, {"resource": "x"}]) + assert e_missing_first.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED + + with pytest.raises(ParseError) as e_resource_first: + resolve_sources(tmp_path, [{"resource": "x"}, {"path": "nope"}]) + assert e_resource_first.value.code == ErrorCode.ERR_SOURCE_KIND_UNSUPPORTED + + +def test_collection_falls_back_to_default_dir(tmp_path: Path) -> None: + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "a.json").write_text("{}") + got = resolve_collection(tmp_path) + assert _rel(tmp_path, got) == {"metaobjects/a.json"} + + +def test_collection_with_no_config_and_no_default_raises(tmp_path: Path) -> None: + with pytest.raises(ParseError) as e: + resolve_collection(tmp_path) + assert e.value.code == ErrorCode.ERR_COLLECTION_NOT_FOUND + + +def test_declared_sources_replace_the_default(tmp_path: Path) -> None: + (tmp_path / ".metaobjects").mkdir() + (tmp_path / ".metaobjects" / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + (tmp_path / "metaobjects").mkdir() + (tmp_path / "metaobjects" / "ignored.json").write_text("{}") + (tmp_path / "model").mkdir() + (tmp_path / "model" / "used.json").write_text("{}") + got = resolve_collection(tmp_path) + assert _rel(tmp_path, got) == {"model/used.json"} diff --git a/server/python/tests/conformance/test_source_resolution_conformance.py b/server/python/tests/conformance/test_source_resolution_conformance.py new file mode 100644 index 000000000..a929d0784 --- /dev/null +++ b/server/python/tests/conformance/test_source_resolution_conformance.py @@ -0,0 +1,136 @@ +"""Runs the shared source-resolution corpus against this port. + +Reads `fixtures/source-resolution-conformance/cases.json` — the single +committed source of truth. There is no per-port fixture. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from metaobjects.config.source_resolver import resolve_collection +from metaobjects.errors import ParseError + +_CORPUS = ( + Path(__file__).resolve().parents[4] + / "fixtures" + / "source-resolution-conformance" + / "cases.json" +) + +_CASES = json.loads(_CORPUS.read_text())["cases"] + + +def test_corpus_is_non_empty() -> None: + """A silent zero-case run is a failed gate, not a pass. + + `@pytest.mark.parametrize` over an empty list simply collects zero tests — + pytest reports that as a SKIP, not a failure, so a corpus that quietly lost + its cases (e.g. a bad path, a JSON-parsing bug) would report green here + with nothing actually checked. Mirrors the TS runner's identically-named + guard (`source-resolution-conformance.test.ts`). + """ + assert len(_CASES) > 0 + + +def _materialize(case: dict, root: Path) -> Path: + """Materialize ``tree`` under ``root``, then ``symlinks`` (I1 — a symlinked + source root, or a symlinked subdirectory inside a walked tree), then + ``config`` (when present) under the directory named by ``resolveFrom`` + (project root when absent). Returns the directory resolution must be + invoked against. + """ + for rel, content in case["tree"].items(): + p = root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(content) + + # Materialized AFTER tree — both link/target are project-root-relative. + for link_rel, target_rel in case.get("symlinks", {}).items(): + link = root / link_rel + link.parent.mkdir(parents=True, exist_ok=True) + link.symlink_to(root / target_rel, target_is_directory=True) + + resolve_from = root / case.get("resolveFrom", ".") + + if case["config"] is not None: + d = resolve_from / ".metaobjects" + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(case["config"], indent=2)) + + return resolve_from + + +@pytest.mark.parametrize("case", _CASES, ids=[c["name"] for c in _CASES]) +def test_source_resolution_conformance(case: dict, tmp_path: Path) -> None: + resolve_from = _materialize(case, tmp_path) + + if "expectError" in case: + with pytest.raises(ParseError) as e: + resolve_collection(resolve_from) + # A string pins the exact code; `True` only pins that resolution + # RAISES — the malformed-config error code is deliberately not + # pinned cross-port (see the corpus README). + if isinstance(case["expectError"], str): + assert e.value.code.value == case["expectError"] + return + + # `expectFiles` is project-root-relative even when `resolveFrom` points + # elsewhere — resolve against `tmp_path`, not `resolve_from`. + root = tmp_path.resolve() + raw = resolve_collection(resolve_from) + got = {p.relative_to(root).as_posix() for p in raw} + assert got == set(case["expectFiles"]) + # A set comparison alone cannot see a duplicate emission (two entries for the + # same file collapse invisibly into one set element) — + # `overlapping-sources-yield-each-file-once` specifically exercises + # resolve_collection's own de-duplication, so the RAW list length must be + # asserted too, before it is thrown away by the set conversion. + assert len(raw) == len(case["expectFiles"]) + + +def test_cli_falls_back_to_neutral_config(tmp_path: Path, monkeypatch) -> None: + """No positional metadata_dir and no YAML `metadata` key => neutral config wins.""" + (tmp_path / "model").mkdir() + (tmp_path / "model" / "meta.a.json").write_text('{"metadata.root":{"children":[]}}') + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + + from metaobjects.cli import resolve_metadata_location + + monkeypatch.chdir(tmp_path) + got = resolve_metadata_location(config=None, root=tmp_path) + assert {Path(p).relative_to(tmp_path.resolve()).as_posix() for p in got} == { + "model/meta.a.json" + } + + +def test_docs_with_no_positional_falls_back_to_neutral_config( + tmp_path: Path, monkeypatch +) -> None: + """`docs` used to declare `metadata_dir` as a REQUIRED positional, so the + neutral `.metaobjects/config.json` `sources` rung was unreachable from it + even though `gen` and `verify --codegen` could already reach it. A bare + `metaobjects docs --out ` must resolve metadata via the same ladder + `gen` uses. + """ + (tmp_path / "model").mkdir() + (tmp_path / "model" / "meta.a.json").write_text('{"metadata.root":{"children":[]}}') + d = tmp_path / ".metaobjects" + d.mkdir() + (d / "config.json").write_text( + json.dumps({"schema_version": 1, "sources": [{"path": "model"}]}) + ) + + from metaobjects.cli import main + + monkeypatch.chdir(tmp_path) + out = tmp_path / "out" + rc = main(["docs", "--out", str(out)]) + assert rc == 0 + assert (out / "api" / "python" / "README.md").exists() diff --git a/server/python/tests/unit/test_sources.py b/server/python/tests/unit/test_sources.py index 905dfb48e..254312334 100644 --- a/server/python/tests/unit/test_sources.py +++ b/server/python/tests/unit/test_sources.py @@ -98,6 +98,78 @@ def test_directory_source_non_recursive(tmp_path: Path) -> None: assert ids == ["top.json"] +def test_directory_source_exclude_pending_is_off_by_default(tmp_path: Path) -> None: + # Loader-level default is OFF (matches TS's loader-level DirectorySource, + # which has no _pending concept at all) — only the CLI-facing + # source_resolver.py turns it on. An app embedding DirectorySource(dir) + # directly must see every file, _pending/ included. + (tmp_path / "meta.live.json").write_text("{}", encoding="utf-8") + pending = tmp_path / "_pending" + pending.mkdir() + (pending / "meta.draft.json").write_text("{}", encoding="utf-8") + + ids = sorted(s.id for s in DirectorySource(tmp_path).expand()) + assert ids == ["meta.draft.json", "meta.live.json"] + + +def test_directory_source_excludes_pending_dir_at_any_depth_when_opted_in(tmp_path: Path) -> None: + # Mirrors TypeScript's PENDING_DIR exclusion (metadata-files.ts) — a draft + # entity under _pending/ must be invisible to codegen, not merely a file + # that happens to be NAMED "_pending". source_resolver.py (the CLI-facing + # caller) opts in via exclude_pending=True; this test exercises the option + # directly. + (tmp_path / "meta.live.json").write_text("{}", encoding="utf-8") + pending = tmp_path / "_pending" + pending.mkdir() + (pending / "meta.draft.json").write_text("{}", encoding="utf-8") + # Nested: _pending/ excluded at ANY depth, not just top-level. + nested_pending = tmp_path / "nested" / "_pending" + nested_pending.mkdir(parents=True) + (nested_pending / "meta.deep-draft.json").write_text("{}", encoding="utf-8") + + ids = [s.id for s in DirectorySource(tmp_path, exclude_pending=True).expand()] + assert ids == ["meta.live.json"] + + +def test_directory_source_follows_a_symlinked_root(tmp_path: Path) -> None: + # I1: the SOURCE path itself is a symlink to a directory. `Path.is_dir()` + # follows symlinks (the existence guard passes), so the walk must too, or + # the root resolves to zero files, silently. + real = tmp_path / "real" + real.mkdir() + (real / "meta.a.json").write_text("{}", encoding="utf-8") + link = tmp_path / "link" + link.symlink_to(real, target_is_directory=True) + + ids = [s.id for s in DirectorySource(link).expand()] + assert ids == ["meta.a.json"] + + +def test_directory_source_follows_a_symlinked_subdirectory(tmp_path: Path) -> None: + # I1, second arm: a symlinked SUBDIRECTORY inside a walked tree. + (tmp_path / "meta.top.json").write_text("{}", encoding="utf-8") + external = tmp_path.parent / f"{tmp_path.name}-external" + external.mkdir() + (external / "meta.linked.json").write_text("{}", encoding="utf-8") + (tmp_path / "linked").symlink_to(external, target_is_directory=True) + + ids = sorted(s.id for s in DirectorySource(tmp_path).expand()) + assert ids == ["meta.linked.json", "meta.top.json"] + + +def test_directory_source_symlink_cycle_fails_loudly_rather_than_hanging( + tmp_path: Path, +) -> None: + from metaobjects.loader.sources import SymlinkLoopError + + (tmp_path / "meta.top.json").write_text("{}", encoding="utf-8") + # A directory symlinked to its own ancestor — a cycle. + (tmp_path / "loop").symlink_to(tmp_path, target_is_directory=True) + + with pytest.raises(SymlinkLoopError): + list(DirectorySource(tmp_path).expand()) + + def test_uri_source_file_scheme_reads_content(tmp_path: Path) -> None: p = tmp_path / "x.json" p.write_text("{}", encoding="utf-8") diff --git a/server/typescript/packages/cli/package.json b/server/typescript/packages/cli/package.json index 6182f1f63..08dcd731d 100644 --- a/server/typescript/packages/cli/package.json +++ b/server/typescript/packages/cli/package.json @@ -27,7 +27,7 @@ "scripts": { "build": "tsc -p .", "typecheck": "tsc -p tsconfig.typecheck.json", - "build:binary": "bun build ./bin/meta.ts --compile --outfile dist/meta --external @biomejs/wasm-bundler --external @biomejs/wasm-web" + "build:binary": "bun build ./bin/meta.ts --compile --outfile dist/meta --external @biomejs/wasm-bundler --external @biomejs/wasm-web --external @electric-sql/pglite" }, "license": "Apache-2.0", "author": "Doug Mealing ", diff --git a/server/typescript/packages/cli/src/commands/init.ts b/server/typescript/packages/cli/src/commands/init.ts index f103023f3..25134f17d 100644 --- a/server/typescript/packages/cli/src/commands/init.ts +++ b/server/typescript/packages/cli/src/commands/init.ts @@ -137,6 +137,14 @@ export interface InitOptions { wireRoot?: boolean; /** Scaffold ONLY the agent-context (always-on + skills + root wiring), skipping the metaobjects/ project scaffold — for dropping context into an existing/polyglot repo. */ docsOnly?: boolean; + /** + * Write ONLY `.metaobjects/config.json` — no TypeScript scaffold (metaobjects.config.ts, + * codegen/generators/, package.json edits, .gitignore, agent-context files, or a + * metaobjects/ directory). For a Maven- or pip-rooted project that needs the Node CLI + * (which owns `migrate` and `verify --db` under ADR-0015) to discover its metadata + * without acquiring a TypeScript project it will never use. + */ + configOnly?: boolean; } export interface InitResult { @@ -304,6 +312,65 @@ async function writeOwnedGenerators(opts: InitOptions, result: InitResult): Prom } } +/** + * .metaobjects/config.json — write fresh defaults, or preserve+merge an existing + * valid config. Shared by the full scaffold and `--config-only` so the two paths + * cannot drift on the config's default content. + */ +async function writeConfigFile(opts: InitOptions, result: InitResult, agentDir: string, agentDirExists: boolean): Promise { + const freshConfig = opts.d1 + ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: buildD1MigrateBlock(opts.cwd) }) + : DEFAULT_CONFIG; + const writeFresh = (): Promise => + writeFile(join(agentDir, "config.json"), JSON.stringify(freshConfig, null, 2) + "\n", "utf8"); + + if (!agentDirExists) { + await writeFresh(); + result.created.push(".metaobjects/config.json"); + return; + } + + const configPath = join(agentDir, "config.json"); + let priorContent: string | undefined; + try { + priorContent = await readFile(configPath, "utf8"); + const parsed = ConfigSchema.parse(JSON.parse(priorContent)); + const merged = ConfigSchema.parse({ ...DEFAULT_CONFIG, ...parsed }); + // When a valid .metaobjects/config.json already exists and the user passes --force, + // we preserve the existing config and only re-scaffold support files. The --d1 flag + // only takes effect on fresh inits — retro-fitting D1 onto an existing project is + // the user's job (edit migrate.dialect and migrate.d1 in config.json directly). + await saveConfig(agentDir, merged); + result.preserved.push(".metaobjects/config.json"); + return; + } catch { + if (priorContent === undefined) { + // The .metaobjects/ dir existed but config.json itself did not — a fresh write. + await writeFresh(); + result.created.push(".metaobjects/config.json"); + return; + } + + // In the full-scaffold path this is only reachable once the caller has + // already required --force (the exists-guard at the top of `init()` + // throws before writeConfigFile runs otherwise), so opts.force is always + // true there. `--config-only` calls this function directly with no such + // guard, so without this check it would silently destroy an existing, + // merely-unparseable config on every run — the one thing `--force` is + // supposed to gate. + if (!opts.force) { + throw new Error( + `existing .metaobjects/config.json exists but could not be parsed; refusing to overwrite it. ` + + `Use --force to replace it with defaults. Prior content:\n${priorContent}`, + ); + } + log.warn("existing .metaobjects/config.json was invalid — writing fresh defaults. Prior content:"); + log.warn(priorContent); + result.warnings.push("invalid .metaobjects/config.json replaced with defaults"); + await writeFresh(); + } +} + export async function init(opts: InitOptions): Promise { const result: InitResult = { created: [], preserved: [], warnings: [] }; const agentDir = join(opts.cwd, DEFAULT_METAOBJECTS_DIR); @@ -319,6 +386,22 @@ export async function init(opts: InitOptions): Promise { return result; } + if (opts.configOnly) { + // --print-only must win outright: a documented dry run must never write, and + // this branch used to return ABOVE the printOnly guard the full-scaffold path + // uses below, so `--config-only --print-only` silently wrote the real file. + if (opts.printOnly) { + result.created.push(".metaobjects/config.json"); + return result; + } + // Config only: write/preserve .metaobjects/config.json and nothing else — no + // metaobjects/ dir, no agent-context, no TypeScript scaffold. `agentDirExists` is + // captured before the mkdir below so an existing valid config is still preserved. + await mkdir(agentDir, { recursive: true }); + await writeConfigFile(opts, result, agentDir, agentDirExists); + return result; + } + if (opts.refreshDocs && exists) { // Refresh-only path: (re)write the agent-context docs and NOTHING else — never // the project scaffold (metaobjects/, config.json, codegen/generators/, @@ -372,45 +455,7 @@ export async function init(opts: InitOptions): Promise { } // .metaobjects/config.json - const freshConfig = opts.d1 - ? ConfigSchema.parse({ ...DEFAULT_CONFIG, migrate: buildD1MigrateBlock(opts.cwd) }) - : DEFAULT_CONFIG; - if (agentDirExists) { - const configPath = join(agentDir, "config.json"); - let priorContent: string | undefined; - try { - priorContent = await readFile(configPath, "utf8"); - const parsed = ConfigSchema.parse(JSON.parse(priorContent)); - const merged = ConfigSchema.parse({ ...DEFAULT_CONFIG, ...parsed }); - // When a valid .metaobjects/config.json already exists and the user passes --force, - // we preserve the existing config and only re-scaffold support files. The --d1 flag - // only takes effect on fresh inits — retro-fitting D1 onto an existing project is - // the user's job (edit migrate.dialect and migrate.d1 in config.json directly). - await saveConfig(agentDir, merged); - result.preserved.push(".metaobjects/config.json"); - } catch { - if (priorContent !== undefined) { - log.warn("existing .metaobjects/config.json was invalid — writing fresh defaults. Prior content:"); - log.warn(priorContent); - result.warnings.push("invalid .metaobjects/config.json replaced with defaults"); - } - await writeFile( - join(agentDir, "config.json"), - JSON.stringify(freshConfig, null, 2) + "\n", - "utf8", - ); - if (priorContent === undefined) { - result.created.push(".metaobjects/config.json"); - } - } - } else { - await writeFile( - join(agentDir, "config.json"), - JSON.stringify(freshConfig, null, 2) + "\n", - "utf8", - ); - result.created.push(".metaobjects/config.json"); - } + await writeConfigFile(opts, result, agentDir, agentDirExists); // .metaobjects/.gitignore await writeFile(join(agentDir, ".gitignore"), METAOBJECTS_GITIGNORE_BODY, "utf8"); @@ -653,6 +698,7 @@ export async function initCommand(args: string[], cwd: string): Promise noSkills: flags.noSkills, wireRoot: flags.wireRoot, docsOnly: flags.docsOnly, + configOnly: flags.configOnly, }); if (flags.printOnly) { @@ -666,6 +712,13 @@ export async function initCommand(args: string[], cwd: string): Promise log.info(`Scaffolded the MetaObjects agent context (${result.created.length} files): .metaobjects/AGENTS.md + .claude/skills/metaobjects-*.`); for (const w of result.warnings) log.info(` ${w}`); log.info("Re-run --docs-only --refresh-docs to update; --no-wire-root to skip the root CLAUDE.md @import."); + } else if (flags.configOnly) { + if (result.created.includes(".metaobjects/config.json")) { + log.info("Wrote .metaobjects/config.json — declare your metadata sources there for the Node CLI (migrate, verify --db)."); + } else { + log.info(".metaobjects/config.json already exists — left untouched."); + } + for (const w of result.warnings) log.warn(w); } else { log.info(nextStepsBlock()); // Surface any scaffold warnings (e.g. the #77 monorepo-subdir agent-context diff --git a/server/typescript/packages/cli/src/commands/migrate.ts b/server/typescript/packages/cli/src/commands/migrate.ts index 3d6041145..5becfd328 100644 --- a/server/typescript/packages/cli/src/commands/migrate.ts +++ b/server/typescript/packages/cli/src/commands/migrate.ts @@ -29,6 +29,7 @@ import { snapshotPath, readSnapshot, writeSnapshot, + qualifiedDbName, BlockedChangesError, PrimaryKeyChangeError, renderD1, @@ -47,6 +48,7 @@ import { type EmitResult, type D1Runner, type SchemaProvenance, + type SchemaSnapshot, } from "@metaobjectsdev/migrate-ts"; import { buildWranglerExecuteArgs, @@ -66,8 +68,11 @@ SUBCOMMANDS: (use with --from-db). NOTE: for a brand-new/empty database use the greenfield example below, NOT baseline — an offline baseline records your metadata as already-applied and emits no CREATE TABLE. - apply-pending Replay committed migration files against --db (no diff); - provisions a fresh/CI database. postgres/sqlite only. + apply-pending Replay committed migration files against --db (no diff). + Provisions a fresh/CI database when the chain BUILDS the + schema — 'meta verify --replay' proves that it does. A + database adopted via 'baseline --from-db' has no such + chain. postgres/sqlite only. MIGRATE FLAGS: --db DB connection URL (required for live-introspect / --apply / --rollback) @@ -87,7 +92,13 @@ MIGRATE FLAGS: --allow Comma-separated destructive-change permissions: drop-column,drop-table,type-change,drop-index,drop-fk, drop-check,drop-view,drop-view-cascade, - adopt-view,nullable-to-not-null,drop-identity-default + adopt-view,nullable-to-not-null,drop-identity-default, + drop-unmanaged + drop-unmanaged permits dropping a table/view the committed + snapshot never contained — one this toolchain never managed. + Without it that drop is refused, because the migration it + writes cannot replay against a database where the object + never existed. --on-ambiguous abort|rename|drop-add How to handle ambiguous renames (default: abort) --from-db Introspect live DB instead of using the committed snapshot @@ -291,6 +302,45 @@ function summarizeChanges(changes: Change[]): Record { return counts; } +/** + * The qualified names of tables/views this diff proposes to DROP that the committed + * snapshot never contained — i.e. objects this toolchain never managed (#313). + * + * FAILS OPEN. No snapshot on disk, or one that cannot be read, yields an empty list: + * a project that has never generated one is not in an error state, and refusing there + * would break the first `meta migrate` of every greenfield project. A parse failure is + * migrate's own error to raise elsewhere, with its own message, not a silent refusal + * here. + * + * Names come from `qualifiedDbName` and nothing else. Three independent sets already + * have to agree on this spelling — the diff's identity maps, the `@unmanaged` + * exclusion set, and the out-of-scope set — and a fourth encoding of "absent schema + * means public" would silently un-guard every object it disagreed about. + */ +async function snapshotAbsentDrops(changes: Change[], snapPath: string): Promise { + let snapshot: SchemaSnapshot | null; + try { + snapshot = await readSnapshot(snapPath); + } catch { + return []; + } + if (snapshot === null) return []; + + const managed = new Set(); + for (const t of snapshot.tables) managed.add(qualifiedDbName(t)); + for (const v of snapshot.views) managed.add(qualifiedDbName(v)); + + const absent: string[] = []; + for (const c of changes) { + const name = + c.kind === "drop-table" ? qualifiedDbName({ name: c.table, schema: c.schema }) + : c.kind === "drop-view" ? qualifiedDbName({ name: c.view, schema: c.schema }) + : undefined; + if (name !== undefined && !managed.has(name)) absent.push(name); + } + return absent; +} + function allowFlagFor(kind: string): string { switch (kind) { case "drop-column": return "drop-column"; @@ -659,6 +709,44 @@ export async function migrateCommand( changeCounts = summarizeChanges(diffResult.changes); + // #313 — refuse to AUTHOR a drop for an object the committed snapshot never + // contained. This path diffs metadata against introspection and never reads the + // snapshot, so an object another tool owns reads as "in the DB, not in the model" + // and is proposed for a drop; the migration that results cannot replay against a + // database where that object never existed, which is how a chain stays broken for + // months. `classify.ts` already states the doctrine — objects present in the DB + // but not the snapshot "must never be treated as actionable drift or + // auto-dropped" — and this is where it is finally enforced. + // + // It does not false-fire on the brownfield cases, because both of them ADD to the + // snapshot: a `baseline --from-db` snapshot contains the foreign table, and a + // scoped project carries its out-of-scope entries forward. The guard fires + // precisely when nothing ever claimed the object. + // + // Only on THIS path, and that is not an omission: the offline path diffs metadata + // against the committed snapshot, so it proposes a drop only for an object the + // snapshot HAS. A snapshot-absent drop is unreachable there by construction. + const unmanagedDrops = await snapshotAbsentDrops( + diffResult.changes, + snapshotPath(resolvePath(metaRoot, config.outDir), kysely.dialect), + ); + if (unmanagedDrops.length > 0 && tokensToAllowOptions(config.allow).dropUnmanaged !== true) { + const named = unmanagedDrops.join(", "); + log.error( + `migrate: refusing to drop ${named} — absent from the committed schema snapshot, so this ` + + `toolchain never managed ${unmanagedDrops.length === 1 ? "it" : "them"} and the migration ` + + `could not replay against a database where ${unmanagedDrops.length === 1 ? "it" : "they"} ` + + `never existed. Re-run with '--allow drop-unmanaged' if the drop is intended.`, + ); + emitStructuredError( + `migrate: refusing to drop ${named} — absent from the committed schema snapshot`, + "re-run with '--allow drop-unmanaged' if the drop is intended", + fmt, + ); + await kysely.close(); + return 2; + } + // All changes — tables AND views — are emitted by the one schema-diff path. // View DDL (create/drop/replace) is produced by diff()'s view passes (2b body // comparison, 2c dependency-recreate) and rendered by every dialect's emitter; diff --git a/server/typescript/packages/cli/src/commands/verify.ts b/server/typescript/packages/cli/src/commands/verify.ts index f0019baa5..f3c702ddd 100644 --- a/server/typescript/packages/cli/src/commands/verify.ts +++ b/server/typescript/packages/cli/src/commands/verify.ts @@ -36,8 +36,14 @@ import { collectUnmanagedNames, excludeFromSnapshot, scopedDiffInputs, + scopeExpectedSchema, buildExpectedSchemaWithProvenance, type GovernedScope, + applyPending, + type ApplyPendingResult, + openReplayEngine, + type ReplayEngine, + verifyReplay, introspect, diff, readSnapshot, @@ -80,9 +86,14 @@ const ERR_UNKNOWN_ATTR = "ERR_UNKNOWN_ATTR"; /** * A no-flags MigrateFlags, so `resolveMigrateConfig` yields exactly what `meta migrate` - * would use with nothing passed on the command line — config value, else default. verify - * consumes only `outDir` from the result (#292); the other fields exist to satisfy the - * shared shape, and reading any of them here would be reaching into migrate's decisions. + * would use with nothing passed on the command line — config value, else default. + * + * verify consumes `outDir` (#292) and — for the replay gate ONLY — `dialect`. The #292 + * restriction that reading anything else "would be reaching into migrate's decisions" + * was written about the DRIFT gate, whose dialect comes from the live `--db` URL. The + * replay gate has no `--db` at all, and the dialect a committed chain was EMITTED for + * is a migrate decision by definition, so migrate's own resolution is the only correct + * source for it. Everything else here exists to satisfy the shared shape. */ const EMPTY_MIGRATE_FLAGS = { db: undefined, dialect: undefined, format: undefined, outDir: undefined, slug: undefined, @@ -125,7 +136,8 @@ export async function verifyCommand( if (!flags.anyExplicit) { log.info( "meta verify — running --templates (default). Explicit subverbs: " + - "--templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift).", + "--templates (prompt drift), --db/--dialect d1 (schema drift), --codegen (codegen drift), " + + "--replay/--replay-snapshot (the committed migration chain replays from empty).", ); } @@ -229,6 +241,11 @@ export async function verifyCommand( // are checked on every `meta verify`. Opt-in by DECLARATION — a model with no // requirement nodes is silent, not in drift. const requirementExit = runRequirementVerify(); + // #313 — BOTH replay flags select this gate. `--replay-snapshot` implies + // `--replay`'s work, so a broken chain must fail under it even when `--replay` + // was not passed; naming only `flags.replay` here is how `--replay-snapshot` + // would parse cleanly and do nothing at all. + const replayExit = flags.replay || flags.replaySnapshot ? await runReplayVerify() : 0; // Advisory verify-as-teacher pass: surface hand-rolled work the metadata could // model. Warnings ONLY — never changes the exit code (bias to under-flagging). @@ -236,7 +253,179 @@ export async function verifyCommand( // noisy project (both opt-outs work on `meta verify` and `meta gen`). if (!flags.noAntipatterns && process.env.META_NO_ANTIPATTERNS !== "1") runAntiPatternAdvisory(); - return Math.max(templateExit, schemaExit, codegenExit, requirementExit); + return Math.max(templateExit, schemaExit, codegenExit, requirementExit, replayExit); + + // -- replay (#313) --------------------------------------------------------- + /** + * Replay the committed migration chain into an EMPTY throwaway database and assert + * it applies. `--replay-snapshot` additionally asserts the result equals the + * committed snapshot. + * + * This exists because `meta migrate` could write a chain that cannot be replayed — + * a bare `DROP TABLE "x"` for an object no migration ever created — and nothing + * noticed until someone tried to provision a fresh database, which for the reporter + * was three months later. The two tiers are separate because a project adopted via + * `migrate baseline --from-db` passes the first trivially and CANNOT pass the second + * by construction: its snapshot is the whole introspected database and its chain is + * empty. + * + * Exit codes follow verify's convention: a chain that fails to apply, or a snapshot + * mismatch, is drift → 1; an engine that will not start is operational → 2. + */ + async function runReplayVerify(): Promise { + // Resolve the migrations directory and the chain's dialect through MIGRATE's own + // precedence, never a second derivation — verify must not look somewhere migrate + // does not write, nor assume a dialect the chain was not emitted for. + const migrateConfig = await resolveMigrateConfig(EMPTY_MIGRATE_FLAGS, projectRoot); + + if (migrateConfig.format === "flyway") { + log.error( + `meta verify --replay is not supported with --migration-format flyway — run 'flyway migrate' against a scratch database to replay`, + ); + return 2; + } + + // --dialect wins; else migrate's resolved dialect; else refuse. There is no --db + // to infer from, so guessing would replay a postgres chain through sqlite. + const dialect: Dialect | undefined = flags.dialect ?? migrateConfig.dialect; + if (dialect === undefined) { + log.error( + `meta verify --replay: no dialect — pass --dialect , or set migrate.dialect in .metaobjects/config.json`, + ); + return 2; + } + if (dialect === "d1") { + log.error( + `meta verify --replay is not supported for dialect 'd1' — use 'wrangler d1 migrations apply' against a scratch database to replay committed migrations`, + ); + return 2; + } + + const dir = resolvePath(projectRoot, migrateConfig.outDir); + + let engine: ReplayEngine; + try { + engine = await openReplayEngine(dialect); + } catch (err) { + // A missing optional driver lands here, and its message already carries the + // install hint. Operational, not drift. + log.error(`meta verify --replay: ${(err as Error).message}`); + return 2; + } + + try { + let applied: ApplyPendingResult; + try { + applied = await applyPending(engine.db, dir, { dryRun: false, dialect }); + } catch (err) { + log.error(`meta verify --replay: ${(err as Error).message}`); + log.error( + `meta verify --replay: the committed chain does not apply to an empty database. ` + + `Applied migrations are immutable, so fix this with a NEW migration that creates the ` + + `missing object — not by editing a committed up.sql.`, + ); + return 1; + } + + // Not a silent pass. `discoverMigrations` returns [] for a missing directory, so + // a run over an empty chain would otherwise "succeed" having proved nothing — + // and a gate that is quiet when it checked nothing cannot be told from one that + // passed. Every migration is pending against a fresh engine, so an empty + // `pending` means the directory held none. + if (applied.pending.length === 0) { + log.info(`meta verify --replay: no committed migrations — nothing to replay`); + return 0; + } + + log.info( + `meta verify --replay — the committed chain applies to an empty ${dialect} database ` + + `(${applied.applied.length} migration(s)).`, + ); + + if (!flags.replaySnapshot) return 0; + return await runReplaySnapshotTier(engine, dialect, dir); + } finally { + await engine.dispose(); + } + } + + /** + * The second tier: the replayed schema must EQUAL the committed snapshot. + * + * This is the 2026-05-31 §8 integrity aid, finally wired — `verifyReplay` has been + * built and exported with no CLI caller since then. What it catches that tier 1 + * cannot is hand-edited structural DDL: a committed up.sql someone changed so the + * chain still applies but no longer produces the schema the snapshot records. + * + * It does NOT support a project adopted via `migrate baseline --from-db`, and does + * not try to detect one. The only candidate signal (`BASELINE_NAME`/`recordBaseline`) + * has no production caller and would live in the TARGET database's ledger, while + * this runs against a fresh engine with no ledger at all. So the failure message + * names baseline adoption as the first thing to rule out. + */ + async function runReplaySnapshotTier( + engine: ReplayEngine, + dialect: Extract, + dir: string, + ): Promise { + // Fails OPEN on a missing snapshot: a project that has never generated one + // offline is not in an error state, and an unreadable/unparseable file is + // migrate's error to raise with its own message, not a drift verdict. It still + // SAYS so — silence here would be indistinguishable from a pass. + let snapshot: SchemaSnapshot | null; + try { + snapshot = await readSnapshot(snapshotPath(dir, dialect)); + } catch { + log.info(`meta verify --replay-snapshot: the committed snapshot could not be read — nothing to compare`); + return 0; + } + if (snapshot === null) { + log.info(`meta verify --replay-snapshot: no committed snapshot — nothing to compare`); + return 0; + } + + // A scoped project carries the OTHER owner's tables into its snapshot on purpose + // and its chain never creates them, so they must leave the comparison. The + // committed snapshot alone cannot be scoped — `scopeExpectedSchema` decides on a + // qualified-name → metadata-FQN provenance map the snapshot does not carry — so + // the expected side is rebuilt from metadata purely to derive that decision. + // + // Only for a project that actually declares `migrate.scope`. An unscoped project + // passes no `governed` and gets the comparison exactly as it was. + let governed: GovernedScope | undefined; + if (schemaScope !== undefined) { + const viewStrategy = forgeConfig?.columnNamingStrategy ?? "snake_case"; + const built = buildExpectedSchemaWithProvenance(root, { + dialect, + columnNamingStrategy: viewStrategy, + views: buildProjectionViews(root, { dialect, columnNamingStrategy: viewStrategy }), + }); + governed = scopeExpectedSchema(built, schemaScope); + } + + // `verifyReplay` calls `applyPending` itself. That is NOT a second replay: the + // first one recorded every migration in this engine's ledger, so the call finds + // nothing pending and returns immediately. + const result = await verifyReplay({ + db: engine.db, + dialect, + migrationsDir: dir, + snapshot, + ...(governed !== undefined ? { governed } : {}), + }); + if (result.ok) { + log.info(`meta verify --replay-snapshot — the replayed chain reproduces the committed snapshot.`); + return 0; + } + + log.error( + `meta verify --replay-snapshot: the replayed chain does not reproduce the committed snapshot. ` + + `If this project was adopted with 'migrate baseline --from-db', its chain does not build the ` + + `schema and this tier does not apply — use --replay instead.`, + ); + for (const line of summarizeDrift([...result.drift, ...result.unmanaged])) log.error(` ${line}`); + return 1; + } // -- requirements (#290) --------------------------------------------------- function runRequirementVerify(): number { diff --git a/server/typescript/packages/cli/src/index.ts b/server/typescript/packages/cli/src/index.ts index 24cda0466..eeb6ad5df 100644 --- a/server/typescript/packages/cli/src/index.ts +++ b/server/typescript/packages/cli/src/index.ts @@ -16,6 +16,7 @@ USAGE: COMMANDS: init Scaffold a MetaObjects project in the current repo init --refresh-docs Refresh .metaobjects/AGENTS.md + CLAUDE.md after CLI upgrades + init --config-only Write only .metaobjects/config.json — for a Maven- or pip-rooted project agent-docs Scaffold only the agent-context (.metaobjects/ + .claude/skills/) — canonical redirect target for all language ports gen [...] Codegen TS targets from your declared metadata types [query] Search the metadata vocabulary (types, subtypes, @attrs) by name or description @@ -110,7 +111,7 @@ or META_NO_ANTIPATTERNS=1. NOTE: outDir, dialect, dbImport, extStyle are read from metaobjects.config.ts `, - verify: `meta verify — drift gate (templates / DB schema / codegen) + verify: `meta verify — drift gate (templates / DB schema / codegen / migration replay) USAGE: meta verify [flags] @@ -122,6 +123,17 @@ FLAGS: --db Schema drift — live DB URL enables the schema-drift gate. Supports: file:, libsql:, postgres:, postgresql: D1 has no URL — use --dialect d1 / --d1 instead. + --replay Migration-chain drift — replay the committed chain into an + EMPTY throwaway database and assert it applies. Needs no --db: + the engine is in-process (PGlite for postgres, a temp file for + sqlite) and provisions nothing. Dialect comes from --dialect, + else migrate.dialect. flyway and d1 are refused. + Postgres needs the optional peer '@electric-sql/pglite'. + --replay-snapshot ...and assert the replayed schema EQUALS the committed + snapshot — catches a hand-edited up.sql that still applies but + no longer builds the recorded schema. Does NOT apply to a + project adopted with 'migrate baseline --from-db' (its chain + does not build the schema); use --replay there. --prompts Directory of provider-resolved template text (default: prompts) --dialect sqlite|postgres|d1 Optional override (auto-detected from --db URL scheme) --allow Accepted for parity with 'migrate'; does NOT affect the drift gate @@ -177,6 +189,8 @@ FLAGS: --print-only Print what would be written, don't write --d1 Include D1 (Cloudflare) migration config --no-wire-root Skip wiring root metaobjects.config.ts + --config-only Write only .metaobjects/config.json (no TS scaffold) — declares + metadata sources for the Node CLI from a Maven- or pip-rooted project --help, -h Print this help `, "agent-docs": `meta agent-docs — scaffold the agent-context (.metaobjects/ always-on files + .claude/skills/) diff --git a/server/typescript/packages/cli/src/lib/allow.ts b/server/typescript/packages/cli/src/lib/allow.ts index 947606f0f..dfb5c9865 100644 --- a/server/typescript/packages/cli/src/lib/allow.ts +++ b/server/typescript/packages/cli/src/lib/allow.ts @@ -33,6 +33,10 @@ export const ALLOW_TOKEN_MAP: Record = { // @generation at all — ambiguous between "never declared it" and // "deliberately removing auto-increment", so migrate refuses without it. "drop-identity-default": "dropIdentityDefault", + // Gates dropping an object the committed snapshot never contained (#313). Read by + // migrate's generation-time provenance guard, not by diff()'s status pass — see + // AllowOptions.dropUnmanaged for why it still belongs in that shape. + "drop-unmanaged": "dropUnmanaged", }; /** Translate parsed `--allow` tokens into the migrate-ts `AllowOptions` shape. */ diff --git a/server/typescript/packages/cli/src/lib/args.ts b/server/typescript/packages/cli/src/lib/args.ts index 4c232a66c..9a14786ec 100644 --- a/server/typescript/packages/cli/src/lib/args.ts +++ b/server/typescript/packages/cli/src/lib/args.ts @@ -15,6 +15,7 @@ export interface InitFlags { noSkills: boolean; wireRoot: boolean; docsOnly: boolean; + configOnly: boolean; } export function parseInitArgs(argv: string[]): InitFlags { @@ -31,6 +32,7 @@ export function parseInitArgs(argv: string[]): InitFlags { "no-skills": { type: "boolean", default: false }, "no-wire-root": { type: "boolean", default: false }, "docs-only": { type: "boolean", default: false }, + "config-only": { type: "boolean", default: false }, }, strict: true, allowPositionals: false, @@ -46,6 +48,7 @@ export function parseInitArgs(argv: string[]): InitFlags { noSkills: !!values["no-skills"], wireRoot: !values["no-wire-root"], docsOnly: !!values["docs-only"], + configOnly: !!values["config-only"], }; } @@ -197,6 +200,13 @@ export const ALLOW_TOKENS = [ // metadata declares no @generation at all — ambiguous between "never // declared it" and "deliberately removing auto-increment". "drop-identity-default", + // drop-unmanaged permits dropping an object the COMMITTED SNAPSHOT never + // contained — i.e. one this toolchain never managed, typically a table another + // tool owns. Without it such a drop is refused at generation time, because the + // migration it writes cannot replay against a database where that object never + // existed (#313). Unlike its neighbours this one is enforced by `migrate` itself + // rather than by `diff()`'s status pass; see AllowOptions.dropUnmanaged. + "drop-unmanaged", ] as const; type AllowToken = (typeof ALLOW_TOKENS)[number]; @@ -232,7 +242,20 @@ export interface VerifyFlags { templates: boolean; /** Run the codegen-drift gate (regenerate-to-temp and diff committed output). */ codegen: boolean; - /** Whether ANY explicit subverb flag (--templates/--db/--codegen) was passed. */ + /** + * Replay the committed migration chain into an empty throwaway database and assert + * it applies (#313). Needs no `--db`: the engine is local and disposable (PGlite + * for postgres, a temp sqlite file), so the gate provisions nothing. + */ + replay: boolean; + /** + * `--replay` plus: assert the replayed schema EQUALS the committed snapshot. A + * separate subverb rather than a `--strict` modifier, because `--lax` below is a + * different axis (ADR-0023 attribute strictness) and `--strict` beside it would + * read as that flag's opposite rather than as a replay depth. + */ + replaySnapshot: boolean; + /** Whether ANY explicit subverb flag (--templates/--db/--codegen/--replay*) was passed. */ anyExplicit: boolean; /** Suppress the advisory anti-pattern (verify-as-teacher) pass. */ noAntipatterns: boolean; @@ -255,6 +278,8 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { "skip-schema": { type: "boolean", default: false }, templates: { type: "boolean", default: false }, codegen: { type: "boolean", default: false }, + replay: { type: "boolean", default: false }, + "replay-snapshot": { type: "boolean", default: false }, "no-antipatterns": { type: "boolean", default: false }, lax: { type: "boolean", default: false }, "d1": { type: "string" }, @@ -283,11 +308,15 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { const templates = !!values.templates; const codegen = !!values.codegen; + const replay = !!values.replay; + const replaySnapshot = !!values["replay-snapshot"]; // --db is itself an explicit subverb selector: passing a connection URL means // "run the schema-drift mode". So is `--dialect d1` (D1 has no --db connection - // URL — see the `d1` field doc above). So "any explicit subverb" is - // templates|codegen|db|dialect==d1. - const anyExplicit = templates || codegen || values.db !== undefined || dialect === "d1"; + // URL — see the `d1` field doc above). The replay flags are subverbs too, and + // must be listed here or `meta verify --replay` would ALSO run the template gate + // as the bare-verify default. + const anyExplicit = + templates || codegen || values.db !== undefined || dialect === "d1" || replay || replaySnapshot; return { prompts: values.prompts, @@ -297,6 +326,8 @@ export function parseVerifyArgs(argv: string[]): VerifyFlags { skipSchema: !!values["skip-schema"], templates, codegen, + replay, + replaySnapshot, anyExplicit, noAntipatterns: !!values["no-antipatterns"], lax: !!values.lax, diff --git a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap index 99fa1137c..c1efedcfc 100644 --- a/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap +++ b/server/typescript/packages/cli/test/__snapshots__/cli.test.ts.snap @@ -9,6 +9,7 @@ USAGE: COMMANDS: init Scaffold a MetaObjects project in the current repo init --refresh-docs Refresh .metaobjects/AGENTS.md + CLAUDE.md after CLI upgrades + init --config-only Write only .metaobjects/config.json — for a Maven- or pip-rooted project agent-docs Scaffold only the agent-context (.metaobjects/ + .claude/skills/) — canonical redirect target for all language ports gen [...] Codegen TS targets from your declared metadata types [query] Search the metadata vocabulary (types, subtypes, @attrs) by name or description diff --git a/server/typescript/packages/cli/test/cli.test.ts b/server/typescript/packages/cli/test/cli.test.ts index b05d977b1..1a03ee9e6 100644 --- a/server/typescript/packages/cli/test/cli.test.ts +++ b/server/typescript/packages/cli/test/cli.test.ts @@ -6,7 +6,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; // wireRoot defaults TRUE (root-wiring is on by default so the scaffolded always-on actually loads; --no-wire-root opts out). -const defaultInitFlags = { force: false, quiet: false, printOnly: false, refreshDocs: false, d1: false, servers: [], clients: [], noSkills: false, wireRoot: true, docsOnly: false }; +const defaultInitFlags = { force: false, quiet: false, printOnly: false, refreshDocs: false, d1: false, servers: [], clients: [], noSkills: false, wireRoot: true, docsOnly: false, configOnly: false }; describe("parseInitArgs", () => { test("default flags (wireRoot on by default, others off)", () => { @@ -27,6 +27,9 @@ describe("parseInitArgs", () => { test("--d1 toggles d1", () => { expect(parseInitArgs(["--d1"])).toEqual({ ...defaultInitFlags, d1: true }); }); + test("--config-only toggles configOnly", () => { + expect(parseInitArgs(["--config-only"])).toEqual({ ...defaultInitFlags, configOnly: true }); + }); test("throws on unknown flag", () => { expect(() => parseInitArgs(["--foo"])).toThrow(/Unknown option '--foo'/); }); diff --git a/server/typescript/packages/cli/test/init.test.ts b/server/typescript/packages/cli/test/init.test.ts index 11ce1f68f..de0ba0e26 100644 --- a/server/typescript/packages/cli/test/init.test.ts +++ b/server/typescript/packages/cli/test/init.test.ts @@ -218,6 +218,89 @@ describe("init() — monorepo-subdir agent-context warning (#77)", () => { }); }); +describe("init() --config-only", () => { + test("writes just the config, no TypeScript scaffold", async () => { + const result = await init({ cwd, configOnly: true }); + + // The one file it writes. + expect(result.created).toContain(".metaobjects/config.json"); + const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(cfg.schema_version).toBe(1); + expect(cfg.sources).toEqual([]); + + // None of the TypeScript scaffold, none of the metaobjects/ metadata dir, none + // of the agent-context — this is the whole point of the flag: a Maven- or + // pip-rooted project declares its sources for the Node CLI without acquiring a + // TS project it will not use. + for (const unwanted of [ + "metaobjects.config.ts", + "codegen/generators/entity.ts", + "package.json", + ".gitignore", + "metaobjects", + ".metaobjects/.gitignore", + ".metaobjects/AGENTS.md", + ]) { + expect(existsSync(join(cwd, unwanted))).toBe(false); + } + }); + + test("--print-only writes nothing to disk", async () => { + // --config-only used to return ABOVE the --print-only guard the full-scaffold + // path checks below it, so this documented dry run silently wrote the real file. + const result = await init({ cwd, configOnly: true, printOnly: true }); + + expect(result.created).toContain(".metaobjects/config.json"); + expect(existsSync(join(cwd, ".metaobjects"))).toBe(false); + expect(existsSync(join(cwd, ".metaobjects", "config.json"))).toBe(false); + }); + + test("leaves an existing valid config untouched", async () => { + mkdirSync(join(cwd, ".metaobjects"), { recursive: true }); + const existing = { schema_version: 1, sources: [{ path: "model" }] }; + writeFileSync(join(cwd, ".metaobjects", "config.json"), JSON.stringify(existing)); + + const result = await init({ cwd, configOnly: true }); + + expect(result.preserved).toContain(".metaobjects/config.json"); + const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(cfg.sources).toEqual([{ path: "model" }]); + }); + + test("refuses to overwrite an existing config it cannot parse, without --force", async () => { + mkdirSync(join(cwd, ".metaobjects"), { recursive: true }); + // Not valid against ConfigSchema.strict() — e.g. written by a newer `meta` + // or a typo'd key. Before --config-only existed, reaching this failure + // required an explicit --force; --config-only must not have quietly + // regressed that safety net. + writeFileSync(join(cwd, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, unknownKey: true })); + + await expect(init({ cwd, configOnly: true })).rejects.toThrow(/could not be parsed/); + // Unmodified — the refusal must be provable, not just declared. + const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(cfg.unknownKey).toBe(true); + }); + + test("--force still replaces an existing unparseable config with defaults", async () => { + mkdirSync(join(cwd, ".metaobjects"), { recursive: true }); + writeFileSync(join(cwd, ".metaobjects", "config.json"), JSON.stringify({ schema_version: 1, unknownKey: true })); + + const result = await init({ cwd, configOnly: true, force: true }); + + expect(result.warnings).toContain("invalid .metaobjects/config.json replaced with defaults"); + const cfg = JSON.parse(readFileSync(join(cwd, ".metaobjects", "config.json"), "utf8")); + expect(cfg.sources).toEqual([]); + }); +}); + +describe("initCommand --config-only", () => { + test("returns 0 and writes only the config", async () => { + expect(await initCommand(["--config-only"], cwd)).toBe(0); + expect(existsSync(join(cwd, ".metaobjects", "config.json"))).toBe(true); + expect(existsSync(join(cwd, "metaobjects.config.ts"))).toBe(false); + }); +}); + describe("init --d1", () => { test("scaffolds config with migrate.dialect = 'd1' and prefilled binding from wrangler.toml", async () => { writeFileSync(join(cwd, "wrangler.toml"), [ diff --git a/server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts b/server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts new file mode 100644 index 000000000..bc326014c --- /dev/null +++ b/server/typescript/packages/cli/test/migrate-drop-unmanaged.test.ts @@ -0,0 +1,147 @@ +/** + * `--allow drop-unmanaged` — refuse to AUTHOR a drop for an object the committed + * snapshot never contained (#313). + * + * The live migrate path diffs metadata against introspection and never reads the + * snapshot, so a table another tool owns reads as "in the DB, not in the model" and + * is proposed for a drop. The migration that results cannot replay against a database + * where that object never existed — which is how the reported chain stayed broken for + * three months. `classify.ts` already stated the doctrine; this is where it is + * enforced. + */ +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdtemp, rm, mkdir, writeFile, readdir } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { sql } from "kysely"; +import { migrateCommand } from "../src/commands/migrate.js"; +import { buildKyselyFromUrl } from "../src/lib/kysely.js"; +import { ALLOW_TOKENS } from "../src/lib/args.js"; +import { ALLOW_TOKEN_MAP, tokensToAllowOptions } from "../src/lib/allow.js"; +import { snapshotPath, writeSnapshot, type TableDescriptor } from "@metaobjectsdev/migrate-ts"; + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +const MODEL = JSON.stringify({ + "metadata.root": { + package: "acme::platform", + children: [{ + "object.entity": { + name: "Job", + children: [ + { "source.rdb": { name: "src", "@table": "jobs" } }, + { "field.long": { name: "id" } }, + { "identity.primary": { name: "pk", "@fields": ["id"], "@generation": "increment" } }, + ], + }, + }], + }, +}); + +interface Fixture { root: string; db: string } + +/** + * A project modelling only `jobs`, against a database that ALSO holds `theirs` — + * another tool's table. `seedSnapshot` decides the case: `managed` writes a snapshot + * containing both (what `baseline --from-db` produces), `unmanaged` writes one + * containing only `jobs`, and `none` writes no snapshot at all. + */ +async function fixture(seedSnapshot: "managed" | "unmanaged" | "none"): Promise { + const root = await mkdtemp(join(tmpdir(), "drop-unmanaged-")); + dirs.push(root); + await mkdir(join(root, "metaobjects"), { recursive: true }); + await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); + await mkdir(join(root, ".metaobjects", "migrations"), { recursive: true }); + await writeFile( + join(root, ".metaobjects", "config.json"), + JSON.stringify({ schema_version: 1, migrate: { dialect: "sqlite" } }), + "utf8", + ); + + const db = join(root, "t.db"); + const k = await buildKyselyFromUrl(`file:${db}`, "sqlite"); + try { + await sql`CREATE TABLE "jobs" (id INTEGER NOT NULL PRIMARY KEY)`.execute(k.db); + await sql`CREATE TABLE "theirs" (id INTEGER NOT NULL PRIMARY KEY)`.execute(k.db); + } finally { + await k.close(); + } + + if (seedSnapshot !== "none") { + // Written through migrate's own writer, not hand-rolled JSON: the on-disk shape + // is versioned and nested, and a fixture that guessed it would be testing the + // guard's error path instead of the guard. + const t = (name: string): TableDescriptor => ({ + name, + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], foreignKeys: [], checks: [], primaryKey: ["id"], + }); + await writeSnapshot(snapshotPath(join(root, ".metaobjects", "migrations"), "sqlite"), { + tables: seedSnapshot === "managed" ? [t("jobs"), t("theirs")] : [t("jobs")], + views: [], + }); + } + return { root, db }; +} + +const migrationCount = async (root: string): Promise => + (await readdir(join(root, ".metaobjects", "migrations"))) + .filter((e) => !e.startsWith(".")).length; + +/** + * `--from-db` is required, not incidental: a bare `meta migrate --db ` takes the + * OFFLINE path, which diffs metadata against the committed snapshot. There, a drop is + * proposed only for an object the snapshot HAS — so the offline path cannot produce a + * snapshot-absent drop by construction, and the guard belongs to the live path alone. + */ +const run = (f: Fixture, extra: string[] = []): Promise => + migrateCommand( + ["--from-db", "--db", `file:${f.db}`, "--dialect", "sqlite", "--slug", "x", + "--allow", ["drop-table", ...extra].join(",")], + f.root, + ); + +describe("drop-unmanaged is wired through every token structure", () => { + test("is a recognised allow token", () => { + expect(ALLOW_TOKENS).toContain("drop-unmanaged"); + }); + + // A token in ALLOW_TOKENS but absent from the map validates cleanly and then grants + // NOTHING — the silent-failure mode allow-tokens-pinned.test.ts exists to prevent. + test("grants a permission rather than validating into nothing", () => { + expect(ALLOW_TOKEN_MAP["drop-unmanaged"]).toBe("dropUnmanaged"); + expect(tokensToAllowOptions(["drop-unmanaged"]).dropUnmanaged).toBe(true); + }); +}); + +describe("meta migrate refuses a snapshot-absent drop", () => { + test("refuses, and writes nothing", async () => { + const f = await fixture("unmanaged"); + expect(await run(f)).toBe(2); + expect(await migrationCount(f.root)).toBe(0); + }); + + test("--allow drop-unmanaged lets it through", async () => { + const f = await fixture("unmanaged"); + expect(await run(f, ["drop-unmanaged"])).toBe(0); + expect(await migrationCount(f.root)).toBe(1); + }); + + // The brownfield non-false-fire, and the reason the guard is safe to ship on by + // default: `baseline --from-db` puts the foreign table IN the snapshot, so the + // guard reads it as managed. Without this case the guard would look correct while + // breaking every adopted project. + test("a drop for a table the snapshot DOES contain proceeds without the flag", async () => { + const f = await fixture("managed"); + expect(await run(f)).toBe(0); + expect(await migrationCount(f.root)).toBe(1); + }); + + // Fails OPEN: refusing here would break the first `meta migrate` of every + // greenfield project, which has no snapshot yet by definition. + test("no snapshot on disk does not trigger the refusal", async () => { + const f = await fixture("none"); + expect(await run(f)).toBe(0); + }); +}); diff --git a/server/typescript/packages/cli/test/unit/args-verify.test.ts b/server/typescript/packages/cli/test/unit/args-verify.test.ts index 33d018f92..b85276fa3 100644 --- a/server/typescript/packages/cli/test/unit/args-verify.test.ts +++ b/server/typescript/packages/cli/test/unit/args-verify.test.ts @@ -6,6 +6,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs([])).toEqual({ prompts: undefined, db: undefined, dialect: undefined, allow: [], skipSchema: false, templates: false, codegen: false, anyExplicit: false, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: undefined, remote: false, }); }); @@ -13,6 +14,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--prompts", "templates"])).toEqual({ prompts: "templates", db: undefined, dialect: undefined, allow: [], skipSchema: false, templates: false, codegen: false, anyExplicit: false, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: undefined, remote: false, }); }); @@ -20,6 +22,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--db", "file:x.db", "--dialect", "sqlite", "--skip-schema"])).toEqual({ prompts: undefined, db: "file:x.db", dialect: "sqlite", allow: [], skipSchema: true, templates: false, codegen: false, anyExplicit: true, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: undefined, remote: false, }); }); @@ -29,6 +32,7 @@ describe("parseVerifyArgs", () => { expect(parseVerifyArgs(["--dialect", "d1", "--d1", "DB", "--remote"])).toEqual({ prompts: undefined, db: undefined, dialect: "d1", allow: [], skipSchema: false, templates: false, codegen: false, anyExplicit: true, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: "DB", remote: true, }); const bare = parseVerifyArgs(["--dialect", "d1"]); @@ -52,6 +56,7 @@ describe("parseVerifyArgs", () => { prompts: undefined, db: undefined, dialect: undefined, allow: ["drop-column", "drop-table"], skipSchema: false, templates: false, codegen: false, anyExplicit: false, noAntipatterns: false, lax: false, + replay: false, replaySnapshot: false, d1: undefined, remote: false, }); }); diff --git a/server/typescript/packages/cli/test/verify-replay.test.ts b/server/typescript/packages/cli/test/verify-replay.test.ts new file mode 100644 index 000000000..0686a6188 --- /dev/null +++ b/server/typescript/packages/cli/test/verify-replay.test.ts @@ -0,0 +1,225 @@ +/** + * `meta verify --replay` / `--replay-snapshot` (#313). + * + * `meta migrate` could write a chain that cannot be replayed — a bare + * `DROP TABLE "x"` for an object no migration ever created, because another tool + * owns it — and nothing noticed until someone provisioned a fresh database. These + * gates replay the committed chain into an empty throwaway engine and say so. + * + * The flag-parse cases below are necessary and nowhere near sufficient: a flag that + * parses correctly and reaches no gate is exactly the failure mode this feature is + * most exposed to, so every tier has a case that drives `verifyCommand` end to end + * against a real project on disk. + */ +import { describe, test, expect, afterAll } from "bun:test"; +import { mkdtemp, mkdir, writeFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { parseVerifyArgs } from "../src/lib/args.js"; +import { verifyCommand } from "../src/commands/verify.js"; +import { runBaseline, runOfflineGenerate } from "../src/commands/migrate.js"; + +const dirs: string[] = []; +afterAll(async () => { for (const d of dirs) await rm(d, { recursive: true, force: true }); }); + +const MIGRATIONS = "./.metaobjects/migrations"; + +/** A shape carrying no `source.*`, so it is not persistable and the schema is empty. */ +const NO_TABLES = JSON.stringify({ + "metadata.root": { + package: "acme::platform", + children: [{ + "object.value": { + name: "Placeholder", + children: [{ "field.string": { name: "note" } }], + }, + }], + }, +}); + +const MODEL = JSON.stringify({ + "metadata.root": { + package: "acme::platform", + children: [{ + "object.entity": { + name: "Job", + children: [ + { "source.rdb": { name: "src", "@table": "jobs" } }, + { "field.long": { name: "id" } }, + { "field.string": { name: "title", "@maxLength": 80 } }, + { "identity.primary": { name: "pk", "@fields": ["id"], "@generation": "increment" } }, + ], + }, + }], + }, +}); + +const cfg = () => + ({ dialect: "sqlite", outDir: MIGRATIONS, onAmbiguous: "abort", + allow: [], slug: "init", dryRun: false } as never); + +/** A project with metadata and a declared sqlite dialect, but no migrations yet. */ +async function project(): Promise { + const root = await mkdtemp(join(tmpdir(), "verify-replay-")); + dirs.push(root); + await mkdir(join(root, "metaobjects"), { recursive: true }); + await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); + await mkdir(join(root, ".metaobjects", "migrations"), { recursive: true }); + await writeFile( + join(root, ".metaobjects", "config.json"), + JSON.stringify({ schema_version: 1, migrate: { dialect: "sqlite", outDir: MIGRATIONS } }), + "utf8", + ); + return root; +} + +/** + * A genuine greenfield chain: baseline against a model declaring NO tables (so the + * reference snapshot starts empty, as a fresh project's does), then generate. The + * result is a chain that BUILDS the schema — which is the population `--replay` and + * `--replay-snapshot` are for, and the opposite of a `baseline --from-db` adoption + * whose snapshot is the whole database against an empty chain. + */ +async function withGeneratedChain(root: string): Promise { + await writeFile(join(root, "metaobjects", "meta.platform.json"), NO_TABLES, "utf8"); + expect(await runBaseline(cfg(), root)).toBe(0); + await writeFile(join(root, "metaobjects", "meta.platform.json"), MODEL, "utf8"); + expect(await runOfflineGenerate(cfg(), root)).toBe(0); +} + +/** A hand-written chain that applies cleanly. */ +async function withApplyingChain(root: string): Promise { + const dir = join(root, ".metaobjects", "migrations", "20260101000000-init"); + await mkdir(dir, { recursive: true }); + await writeFile(join(dir, "up.sql"), 'CREATE TABLE "jobs" (id INTEGER NOT NULL PRIMARY KEY);', "utf8"); + await writeFile(join(dir, "down.sql"), 'DROP TABLE "jobs";', "utf8"); +} + +/** Hand-write a migration whose up.sql drops something the chain never creates. */ +async function withBrokenChain(root: string): Promise { + const dir = join(root, ".metaobjects", "migrations", "20260101000000-init"); + await mkdir(dir, { recursive: true }); + await writeFile( + join(dir, "up.sql"), + 'CREATE TABLE "jobs" (id INTEGER NOT NULL PRIMARY KEY);\n\nDROP TABLE "theirs";', + "utf8", + ); + await writeFile(join(dir, "down.sql"), 'DROP TABLE "jobs";', "utf8"); +} + +describe("verify --replay / --replay-snapshot flags", () => { + test("both are parsed", () => { + expect(parseVerifyArgs(["--replay"]).replay).toBe(true); + expect(parseVerifyArgs(["--replay-snapshot"]).replaySnapshot).toBe(true); + }); + + test("both default off", () => { + expect(parseVerifyArgs([]).replay).toBe(false); + expect(parseVerifyArgs([]).replaySnapshot).toBe(false); + }); + + // Without this, `meta verify --replay` would ALSO run the template gate as the + // bare-verify default, and report drift the user never asked about. + test("each counts as an explicit subverb", () => { + expect(parseVerifyArgs(["--replay"]).anyExplicit).toBe(true); + expect(parseVerifyArgs(["--replay-snapshot"]).anyExplicit).toBe(true); + }); + + // `--lax` is ADR-0023 attribute strictness — a different axis entirely, which is + // why the second tier is its own subverb and not `--strict`. + test("does not collide with --lax", () => { + const f = parseVerifyArgs(["--replay-snapshot", "--lax"]); + expect(f.replaySnapshot).toBe(true); + expect(f.lax).toBe(true); + }); +}); + +describe("verify --replay runs the gate", () => { + test("a chain that drops an object it never creates fails", async () => { + const root = await project(); + await withBrokenChain(root); + expect(await verifyCommand(["--replay"], root)).toBe(1); + }); + + test("a generated chain applies to an empty database", async () => { + const root = await project(); + await withGeneratedChain(root); + expect(await verifyCommand(["--replay"], root)).toBe(0); + }); + + // Not a silent pass: a run over an empty chain proves nothing, and a gate that is + // quiet when it checked nothing cannot be told from one that passed. + test("no committed migrations reports that, and passes", async () => { + const root = await project(); + expect(await verifyCommand(["--replay"], root)).toBe(0); + }); + + // `migrate.dialect` genuinely defaults to undefined, so this refusal is reachable + // rather than dead code — with no --db there is no URL to infer from, and guessing + // would replay a postgres chain through sqlite. + test("no dialect anywhere is refused, operationally", async () => { + const root = await project(); + await withApplyingChain(root); + await writeFile( + join(root, ".metaobjects", "config.json"), + JSON.stringify({ schema_version: 1, migrate: { outDir: MIGRATIONS } }), + "utf8", + ); + expect(await verifyCommand(["--replay", "--skip-schema"], root)).toBe(2); + // The control: naming the dialect on the command line makes the same run pass. + expect(await verifyCommand(["--replay", "--dialect", "sqlite", "--skip-schema"], root)).toBe(0); + }); + + // `--skip-schema` is load-bearing here, not incidental: without it the D1 SCHEMA + // gate also returns 2 (no wrangler.toml), so the assertion would pass whether or + // not the replay gate refuses at all. + test("d1 is refused, operationally", async () => { + const root = await project(); + await withApplyingChain(root); + expect(await verifyCommand(["--replay", "--dialect", "d1", "--skip-schema"], root)).toBe(2); + // The control: the same run on the project's real dialect passes, so the 2 above + // is the d1 refusal and not something wrong with the fixture. + expect(await verifyCommand(["--replay", "--skip-schema"], root)).toBe(0); + }); +}); + +describe("verify --replay-snapshot runs BOTH tiers", () => { + // The regression that would let the flag ship dead: --replay-snapshot implies + // --replay's work, so a broken chain must fail under it even though --replay was + // never passed. + test("a broken chain fails under --replay-snapshot alone", async () => { + const root = await project(); + await withBrokenChain(root); + expect(await verifyCommand(["--replay-snapshot"], root)).toBe(1); + }); + + test("a generated chain reproduces its own committed snapshot", async () => { + const root = await project(); + await withGeneratedChain(root); + expect(await verifyCommand(["--replay-snapshot"], root)).toBe(0); + }); + + // Tier 2's whole reason for existing, and what keeps the case above honest: a chain + // that APPLIES but no longer produces the schema the snapshot records. This is the + // hand-edited-structural-DDL case — tier 1 cannot see it. + test("a chain that applies but diverges from the snapshot fails", async () => { + const root = await project(); + await withGeneratedChain(root); + expect(await verifyCommand(["--replay"], root)).toBe(0); // tier 1 is blind to it + const extra = join(root, ".metaobjects", "migrations", "29991231000000-hand-edit"); + await mkdir(extra, { recursive: true }); + await writeFile(join(extra, "up.sql"), 'CREATE TABLE "unrecorded" (id INTEGER NOT NULL PRIMARY KEY);', "utf8"); + await writeFile(join(extra, "down.sql"), 'DROP TABLE "unrecorded";', "utf8"); + expect(await verifyCommand(["--replay"], root)).toBe(0); // still applies fine + expect(await verifyCommand(["--replay-snapshot"], root)).toBe(1); + }); + + // Fails OPEN: a project that has never generated a snapshot offline is not in an + // error state — but the tier says so rather than passing in silence. + test("no committed snapshot is not a failure", async () => { + const root = await project(); + // A chain that APPLIES, so tier 1 passes and this isolates the snapshot half. + await withApplyingChain(root); + expect(await verifyCommand(["--replay-snapshot"], root)).toBe(0); + }); +}); diff --git a/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts b/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts index a3f16bc43..89bd3ceeb 100644 --- a/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts +++ b/server/typescript/packages/integration-tests/test/view-lifecycle-pg.test.ts @@ -579,7 +579,8 @@ describe("view lifecycle — real Postgres", () => { // The banner lives in the committed migration file, where a reviewer sees it. expect(up).toContain("WARNING: CASCADE DROP"); expect(up).toContain("reporting.downstream_report (view)"); - expect(up).toContain(`DROP VIEW "v_program_summary" CASCADE;`); + // #313 — forward drops carry IF EXISTS so a committed chain replays from empty. + expect(up).toContain(`DROP VIEW IF EXISTS "v_program_summary" CASCADE;`); // Ours came back; theirs did not — which is exactly what the operator opted into. expect(await relationExists("v_program_summary")).toBe(true); diff --git a/server/typescript/packages/migrate-ts/package.json b/server/typescript/packages/migrate-ts/package.json index f31b558cf..b74df704d 100644 --- a/server/typescript/packages/migrate-ts/package.json +++ b/server/typescript/packages/migrate-ts/package.json @@ -48,9 +48,20 @@ "@metaobjectsdev/metadata": "workspace:*" }, "peerDependencies": { - "kysely": ">=0.27.0 <0.30.0" + "kysely": ">=0.27.0 <0.30.0", + "@electric-sql/pglite": ">=0.3.0 <0.6.0", + "@libsql/kysely-libsql": ">=0.4.0 <0.5.0" + }, + "peerDependenciesMeta": { + "@electric-sql/pglite": { + "optional": true + }, + "@libsql/kysely-libsql": { + "optional": true + } }, "devDependencies": { + "@electric-sql/pglite": "^0.5.0", "@libsql/kysely-libsql": "^0.4.1", "@types/pg": "^8.20.0", "bun-types": "latest", diff --git a/server/typescript/packages/migrate-ts/src/emit/postgres.ts b/server/typescript/packages/migrate-ts/src/emit/postgres.ts index db4ec92ef..68ea5d2ec 100644 --- a/server/typescript/packages/migrate-ts/src/emit/postgres.ts +++ b/server/typescript/packages/migrate-ts/src/emit/postgres.ts @@ -54,16 +54,53 @@ export function renderPostgres(changes: Change[]): EmitResult { } // Down runs in reverse order (so creates undo correctly w.r.t. FKs). return { - up: upStmts.join("\n\n"), + up: [...createSchemaStmts(sorted), ...upStmts].join("\n\n"), down: [...downStmts].reverse().join("\n\n"), recreatedTables: new Set(), // postgres alters in place; no recreate-and-copy }; } +/** + * `CREATE SCHEMA IF NOT EXISTS` for every non-default schema this migration creates + * an object in, ahead of everything else it emits. + * + * A chain must be appliable to a VIRGIN database (#313), and `CREATE TABLE "s"."x"` + * fails there unless `s` exists — yet `CREATE SCHEMA` was emitted nowhere in either + * emitter, only by the ledger's own setup. So an `@schema` project's chain could + * never be replayed, and the first `apply-pending` against a fresh CI database died. + * + * VIEWS count, not only tables: a first migration that creates only a view in a + * non-default schema fails identically. A `create-view` carries the schema in two + * places and the change's own key wins, matching `renderCreateView(c.view, c.schema)`. + * + * `IF NOT EXISTS` because a later migration in the same chain, or an operator, may + * have created it already. Sorted so output is deterministic — the committed snapshot + * and the golden tests depend on that. Deliberately NOT dropped in `down`: the schema + * may hold objects this tool does not own and cannot restore. + */ +function createSchemaStmts(sorted: readonly Change[]): string[] { + const schemas = new Set(); + for (const c of sorted) { + const s = + c.kind === "create-table" ? c.table.schema + : c.kind === "create-view" ? (c.schema ?? c.view.schema) + : undefined; + if (s !== undefined && s !== DEFAULT_DB_SCHEMA_POSTGRES) schemas.add(s); + } + return [...schemas].sort().map((s) => `CREATE SCHEMA IF NOT EXISTS ${quote(s)};`); +} + function renderUp(c: Change): string { switch (c.kind) { case "create-table": return renderCreateTable(c.table); - case "drop-table": return `DROP TABLE ${quoteQualified(c.table, c.schema)};`; + // #313 — every FORWARD drop is `IF EXISTS`. A committed chain must apply to a + // VIRGIN database, and the diff legitimately proposes dropping an object that + // exists in the live DB but was never created by any migration in the chain (a + // table another tool owns, say). Bare, that statement kills the replay with + // `table "x" does not exist`. The DOWN renderer below is deliberately NOT + // guarded: `rollbackTo` runs down.sql and the ledger delete in ONE transaction, + // so a no-op down would still record the rollback as done. + case "drop-table": return `DROP TABLE IF EXISTS ${quoteQualified(c.table, c.schema)};`; case "rename-table": return `ALTER TABLE ${quoteQualified(c.from, c.schema)} RENAME TO ${quote(c.to)};`; case "add-column": { const base = `ALTER TABLE ${quoteQualified(c.table, c.schema)} ADD COLUMN ${renderColumn(c.column)};`; @@ -90,18 +127,28 @@ function renderUp(c: Change): string { // descriptor (both diff producers populate it), which is where the marker lives. // Matters broadly, not marginally: Drizzle's `unique()` emits constraints, so every // schema adopted from Drizzle has constraint-backed unique indexes. + // Both arms carry the #313 `IF EXISTS`: they are two renderings of the SAME + // `drop-index` change, and guarding one would leave the change kind half-covered. case "drop-index": return c.restore?.constraint !== undefined - ? `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT ${quote(c.index)};` - : `DROP INDEX ${quoteIndexQualified(c.index, c.schema)};`; + ? `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.index)};` + : `DROP INDEX IF EXISTS ${quoteIndexQualified(c.index, c.schema)};`; case "add-fk": return renderAddFk(c.table, c.schema, c.fk); - case "drop-fk": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT ${quote(c.fk)};`; - // add-check / drop-check are declared but NOT yet produced by the diff — - // checks are create-time-only (inlined in CREATE TABLE via renderCreateTable). - // These arms exist for future existing-table CHECK evolution support, mirroring - // the create-view/drop-view "declared, not yet produced" pattern. + case "drop-fk": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.fk)};`; + // `drop-check` IS produced by the diff — diff/index.ts:579 and :592 both push it, + // and an evolved `field.enum @values` is a live producer. (A comment here used to + // claim these arms were unreachable "declared, not yet produced" stubs; that was + // false, and two tests already asserted the emitted statement.) `add-check` is the + // paired ADD and rides the same passes. + // + // `drop-fk`/`drop-check` are guarded on Postgres ONLY, and that is not a dialect + // split: SQLite emits no standalone statement for either kind — `renderUpNative` + // throws, because SQLite constraints are create-time-only and inline, so the change + // folds into a table recreate that rebuilds from the EXPECTED descriptor and never + // references the dropped constraint. SQLite is already replay-safe by construction; + // guarding Postgres makes the two dialects agree. case "add-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} ADD CONSTRAINT ${quote(c.check.name)} CHECK (${c.check.expression});`; - case "drop-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT ${quote(c.check)};`; + case "drop-check": return `ALTER TABLE ${quoteQualified(c.table, c.schema)} DROP CONSTRAINT IF EXISTS ${quote(c.check)};`; case "create-view": return renderCreateView(c.view, c.schema, /* orReplace */ false); case "drop-view": return renderDropView(c); case "replace-view": return renderCreateView(c.view, c.schema, /* orReplace */ true); @@ -372,7 +419,9 @@ function renderViewComment(qualifiedView: string, comment: string | null): strin function renderDropView(c: Extract): string { const qualified = quoteQualifiedView(c.view, c.schema); const dependents = c.dependents ?? []; - if (dependents.length === 0) return `DROP VIEW ${qualified};`; + // #313 `IF EXISTS` on both forms — this is the FORWARD renderer. `renderRestoreView` + // below stays bare: it is reached only from `renderDown`. + if (dependents.length === 0) return `DROP VIEW IF EXISTS ${qualified};`; const listed = dependents .map((d) => `-- ${d.schema}.${d.name} (${d.relkind === "m" ? "materialized view" : "view"})`) @@ -385,7 +434,7 @@ function renderDropView(c: Extract): string { "-- restore them:", listed, rule, - `DROP VIEW ${qualified} CASCADE;`, + `DROP VIEW IF EXISTS ${qualified} CASCADE;`, ].join("\n"); } diff --git a/server/typescript/packages/migrate-ts/src/emit/sqlite.ts b/server/typescript/packages/migrate-ts/src/emit/sqlite.ts index 958dba81f..ba022b508 100644 --- a/server/typescript/packages/migrate-ts/src/emit/sqlite.ts +++ b/server/typescript/packages/migrate-ts/src/emit/sqlite.ts @@ -216,13 +216,20 @@ function renderRecreate( function renderUpNative(c: Change): string { switch (c.kind) { case "create-table": return renderCreateTable(c.table); - case "drop-table": return `DROP TABLE ${quote(c.table)};`; + // #313 — FORWARD drops are `IF EXISTS` so a committed chain applies to a VIRGIN + // database: the diff legitimately proposes dropping an object present in the live + // DB that no migration in the chain ever created. `renderDownNative` stays bare + // (a no-op rollback would still be recorded as done), and so does the + // recreate-and-copy rebuild's DROP above — that one drops a table the same recipe + // just INSERT…SELECTed from, where IF EXISTS turns a caught corruption into a + // silent one. + case "drop-table": return `DROP TABLE IF EXISTS ${quote(c.table)};`; case "rename-table": return `ALTER TABLE ${quote(c.from)} RENAME TO ${quote(c.to)};`; case "add-column": return `ALTER TABLE ${quote(c.table)} ADD COLUMN ${renderColumnInline(c.column)};`; case "drop-column": return `ALTER TABLE ${quote(c.table)} DROP COLUMN ${quote(c.column)};`; case "rename-column": return `ALTER TABLE ${quote(c.table)} RENAME COLUMN ${quote(c.from)} TO ${quote(c.to)};`; case "add-index": return renderCreateIndex(c.table, c.index); - case "drop-index": return `DROP INDEX ${quote(c.index)};`; + case "drop-index": return `DROP INDEX IF EXISTS ${quote(c.index)};`; case "add-check": case "drop-check": case "change-column-type": diff --git a/server/typescript/packages/migrate-ts/src/index.ts b/server/typescript/packages/migrate-ts/src/index.ts index b4c8c9208..154bb7005 100644 --- a/server/typescript/packages/migrate-ts/src/index.ts +++ b/server/typescript/packages/migrate-ts/src/index.ts @@ -112,6 +112,10 @@ export { export { verifyReplay } from "./verify/replay.js"; export type { VerifyReplayArgs, VerifyReplayResult } from "./verify/replay.js"; +// An empty in-process database to replay a committed chain into (#313). +export { openReplayEngine } from "./verify/replay-engine.js"; +export type { ReplayEngine } from "./verify/replay-engine.js"; + // Wrangler config helpers export { findWranglerConfig, diff --git a/server/typescript/packages/migrate-ts/src/types.ts b/server/typescript/packages/migrate-ts/src/types.ts index 3006d6712..58d352980 100644 --- a/server/typescript/packages/migrate-ts/src/types.ts +++ b/server/typescript/packages/migrate-ts/src/types.ts @@ -347,6 +347,23 @@ export interface AllowOptions { * skips the default-diff for a live auto-sequence default entirely.) */ dropIdentityDefault?: boolean; + /** + * Permits dropping an object the COMMITTED SNAPSHOT never contained — i.e. one + * this toolchain never managed. Without it, such a drop is refused at generation + * time, because the migration it would write cannot replay against a database + * where that object never existed (#313). `classify.ts` already states the + * doctrine: objects present in the DB but not the snapshot "must never be treated + * as actionable drift or auto-dropped". + * + * The ONE field here read by the CLI's generation-time provenance guard rather + * than by `diff()`'s status pass — `diff` compares metadata against introspection + * and never sees the snapshot, which is precisely why the doctrine was not + * enforced where it mattered. It lives in `AllowOptions` anyway so `--allow` keeps + * ONE token list and ONE grant map (`ALLOW_TOKENS` / `ALLOW_TOKEN_MAP`, pinned + * together by `cli/test/unit/allow-tokens-pinned.test.ts`): a second parallel + * validation path for a single token is the exact drift that pin exists to catch. + */ + dropUnmanaged?: boolean; } export type AmbiguousChange = diff --git a/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts b/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts new file mode 100644 index 000000000..5762d6867 --- /dev/null +++ b/server/typescript/packages/migrate-ts/src/verify/replay-engine.ts @@ -0,0 +1,184 @@ +// An empty, throwaway database that lives INSIDE this process. +// +// The replay gate has to apply a whole committed chain from nothing. Doing that +// against the user's server would mean CREATE DATABASE — which needs CREATEDB, +// breaks behind a connection pooler, is restricted on managed Postgres, collides +// between parallel CI jobs sharing one server, and puts a DROP DATABASE next to a +// name derived from a real one (Postgres truncates identifiers at 63 bytes, so a +// long enough target derives a scratch name that truncates back ONTO the target). +// None of that is worth it when the engine runs locally and disposably: PGlite is +// real Postgres compiled to WASM and lives in this process; sqlite is a throwaway +// file in a private temp directory (see `openMemorySqlite` for why not `:memory:`). +// Nothing to provision, nothing to name, nothing to drop by mistake. +// +// Both drivers are OPTIONAL peers imported lazily. PGlite is ~22 MB of WASM and must +// not land in the node_modules of every adopter who only ever runs `meta gen`; the +// install hints mirror `buildKyselyFromUrl`'s. `cli` already depends on +// `@libsql/kysely-libsql` outright, so only a direct embedder can miss that one. +import { Kysely } from "kysely"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +export interface ReplayEngine { + /** An empty database. The caller owns applying migrations into it. */ + db: Kysely>; + /** Release the engine. Safe to call more than once. */ + dispose: () => Promise; +} + +/** Open an empty in-process database of the given dialect. */ +export async function openReplayEngine( + dialect: "postgres" | "sqlite", +): Promise { + return dialect === "postgres" ? openPglite() : openMemorySqlite(); +} + +/** + * A throwaway sqlite database in a private temp directory, removed on dispose. + * + * NOT `:memory:`, and that is the whole point of this comment. Under + * `@libsql/kysely-libsql`, `:memory:` gives every CONNECTION its own database — so a + * table created inside a transaction is invisible the moment the transaction's + * connection is released. `applyPending` runs each migration file in a transaction, + * which means an in-memory engine would replay a whole chain into a series of + * throwaway databases, introspect an empty one, and never let migration 2 see + * migration 1's tables. The gate would pass having proved nothing. + * + * `file::memory:?cache=shared` fixes the visibility and breaks isolation instead — + * two engines in one process land in the SAME database — and libsql rejects the + * named `?mode=memory&cache=shared` form outright (`URL_PARAM_NOT_SUPPORTED`). A + * unique temp file is correct on both counts, and it is what the existing + * `test/integrity/replay.test.ts` has always used. + */ +async function openMemorySqlite(): Promise { + type LibsqlDialectCtor = new (opts: { url: string }) => + ConstructorParameters>>[0]["dialect"]; + let LibsqlDialect: LibsqlDialectCtor; + try { + const mod = await import("@libsql/kysely-libsql"); + LibsqlDialect = mod.LibsqlDialect as unknown as LibsqlDialectCtor; + } catch { + throw new Error( + `the sqlite replay engine requires '@libsql/kysely-libsql'; install it to run 'meta verify --replay'`, + ); + } + const dir = mkdtempSync(join(tmpdir(), "meta-replay-")); + const db = new Kysely>({ + dialect: new LibsqlDialect({ url: `file:${join(dir, "replay.db")}` }), + }); + return disposable(db, async () => { + rmSync(dir, { recursive: true, force: true }); + }); +} + +async function openPglite(): Promise { + let PGliteCtor: new () => PgliteInstance; + try { + const mod = await import("@electric-sql/pglite"); + PGliteCtor = mod.PGlite as unknown as new () => PgliteInstance; + } catch { + throw new Error( + `the postgres replay engine requires '@electric-sql/pglite' (in-process WASM Postgres); ` + + `install it to run 'meta verify --replay' against a postgres chain`, + ); + } + const { PostgresDialect } = await import("kysely"); + + // PGlite is Postgres compiled to WASM, and Emscripten propagates the WASM + // program's internal exit status into `process.exitCode` — it becomes 99 on the + // FIRST QUERY (not on teardown) and stays there for the life of the process. + // Opening an engine must not decide what the HOST process exits with, so the + // caller's value is captured here and restored on dispose. + // + // `bin/meta.ts` ends with `process.exit(code)`, which overrides this, so the + // shipped CLI never showed it. Anything that does NOT force its own exit did: + // this package's `bun test` exited 99 on 0 failures, turning two + // `ci-local.sh --only ts` gates red with no failing test to point at, and an + // embedder calling `openReplayEngine` directly would exit non-zero on success. + const hostExitCode = process.exitCode; + const pg = new PGliteCtor(); + const db = new Kysely>({ + dialect: new PostgresDialect({ pool: pgliteAsPool(pg) as never }), + }); + + // `?? 0` is load-bearing, not defensive: assigning `undefined` to + // `process.exitCode` is a NO-OP under Bun (measured — set 99, assign + // `undefined`, it stays 99; assign 0 and it clears). The pristine value IS + // `undefined`, so restoring it literally runs and changes nothing — which is + // the shape this bug already took once during the fix. + // + // The restore sits in a `finally` because this close is frequently the SECOND: + // `disposable` runs `db.destroy()` first, which drives the pool's `end()`, + // which already called `pg.close()`, so this call throws `PGlite is closed`. + return disposable(db, async () => { + try { + await pg.close(); + } finally { + process.exitCode = hostExitCode ?? 0; + } + }); +} + +/** The slice of PGlite's surface this file uses. */ +interface PgliteInstance { + query( + sql: string, + params?: unknown[], + ): Promise<{ rows: unknown[]; affectedRows?: number; statement?: string }>; + close(): Promise; +} + +/** + * Adapt PGlite to the `pg.Pool` shape kysely's `PostgresDialect` expects: `connect()` + * returning a client with `query()`/`release()`, plus `end()`. PGlite offers only + * `query`/`close`, so without this the dialect cannot drive it at all. + * + * PGlite is a SINGLE session, so every `connect()` hands back the same underlying + * instance. That is correct here — a replay is strictly sequential — and it is what + * makes a session advisory lock taken on one kysely connection visible to the next. + * + * `command` is read by kysely only to decide whether to report numAffectedRows; the + * replay path never reads it, so PGlite's `statement` (or a SELECT default) suffices. + */ +function pgliteAsPool(pg: PgliteInstance): unknown { + return { + async connect() { + return { + async query(sqlText: unknown, params?: readonly unknown[]) { + if (typeof sqlText !== "string") { + throw new Error(`the PGlite replay engine does not support cursors`); + } + const r = await pg.query(sqlText, params ? [...params] : []); + return { + command: r.statement ?? "SELECT", + rowCount: r.affectedRows ?? r.rows.length, + rows: r.rows, + }; + }, + release() { /* single session — there is no pool to return to */ }, + }; + }, + async end() { + await pg.close(); + }, + }; +} + +function disposable( + db: Kysely>, + closeEngine: () => Promise, +): ReplayEngine { + let disposed = false; + return { + db, + dispose: async () => { + if (disposed) return; + disposed = true; + // Both swallow: the engine is throwaway, and a teardown error must not mask + // the replay verdict the caller is about to report. + try { await db.destroy(); } catch { /* ignore */ } + try { await closeEngine(); } catch { /* ignore */ } + }, + }; +} diff --git a/server/typescript/packages/migrate-ts/src/verify/replay.ts b/server/typescript/packages/migrate-ts/src/verify/replay.ts index 05a145fb5..ddf46f58b 100644 --- a/server/typescript/packages/migrate-ts/src/verify/replay.ts +++ b/server/typescript/packages/migrate-ts/src/verify/replay.ts @@ -4,6 +4,7 @@ import { applyPending } from "../apply/apply.js"; import { MIGRATIONS_TABLE } from "../apply/ledger.js"; import { introspect } from "../introspect/index.js"; import { driftAgainstSnapshot, type DriftClassification } from "../drift/classify.js"; +import { excludeFromSnapshot, type GovernedScope } from "../scope.js"; import type { Dialect, SchemaSnapshot } from "../types.js"; export interface VerifyReplayArgs { @@ -14,6 +15,20 @@ export interface VerifyReplayArgs { migrationsDir: string; /** The committed snapshot the migrations are expected to reproduce. */ snapshot: SchemaSnapshot; + /** + * The scope decision the run made, as `scopeExpectedSchema` reports it. + * + * A project declaring `migrate.scope` carries the OTHER owner's tables into its + * committed snapshot on purpose (`carryForwardOutOfScope`), and its chain — also on + * purpose — never creates them. Without this they read as missing on every replay, + * so a scoped project could never use this check at all. + * + * Excluded from the SNAPSHOT side only: the replayed database never had them + * either, so there is nothing to suppress on the actual side. Omitted ⇒ the + * comparison is byte-for-byte what it was, which is what every unscoped project + * gets. + */ + governed?: GovernedScope; } export interface VerifyReplayResult extends DriftClassification { @@ -35,7 +50,12 @@ export async function verifyReplay(args: VerifyReplayArgs): Promise t.name !== MIGRATIONS_TABLE), }; - const classification = await driftAgainstSnapshot(args.snapshot, actual, args.dialect); + // `excludeFromSnapshot` returns a ScopedExpectedSchema, so take `.snapshot`. With an + // empty `outOfScope` it returns the SAME object, not an equal copy. + const expected = args.governed !== undefined + ? excludeFromSnapshot(args.snapshot, args.governed).snapshot + : args.snapshot; + const classification = await driftAgainstSnapshot(expected, actual, args.dialect); return { ...classification, ok: classification.drift.length === 0 && classification.unmanaged.length === 0, diff --git a/server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts b/server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts index 76b574e2a..485adb6b0 100644 --- a/server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts +++ b/server/typescript/packages/migrate-ts/test/check-evolution/drop-check-down.test.ts @@ -9,7 +9,8 @@ describe("drop-check: restore down + allow gating", () => { test("drop-check with restore → down re-adds the constraint", () => { const c = { kind: "drop-check", table: "orders", check: CHK.name, restore: CHK, status: { state: "allowed" } } as unknown as Change; const r = emit([c], { dialect: "postgres" }); - expect(r.up).toContain(`ALTER TABLE "orders" DROP CONSTRAINT "orders_qty_numeric_chk";`); + // #313 — the forward drop carries IF EXISTS; the down (asserted below) stays bare. + expect(r.up).toContain(`ALTER TABLE "orders" DROP CONSTRAINT IF EXISTS "orders_qty_numeric_chk";`); expect(r.down).toContain(`ALTER TABLE "orders" ADD CONSTRAINT "orders_qty_numeric_chk" CHECK (qty >= 1);`); }); test("drop-check is blocked unless allow.dropCheck", () => { diff --git a/server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts b/server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts index 91a9bc7f6..52e03a2d6 100644 --- a/server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts +++ b/server/typescript/packages/migrate-ts/test/check/emit-postgres-check.test.ts @@ -22,6 +22,8 @@ describe("emit postgres — checks", () => { }); test("drop-check → ALTER TABLE DROP CONSTRAINT", () => { const r = emit([{ kind: "drop-check", table: "orders", check: "orders_status_chk", status: ALLOWED } as unknown as Change], { dialect: "postgres" }); - expect(r.up).toContain(`ALTER TABLE "orders" DROP CONSTRAINT "orders_status_chk";`); + // #313 — the forward drop carries IF EXISTS. The add-check test above asserts the + // matching DOWN, which stays bare; the two together pin the direction split. + expect(r.up).toContain(`ALTER TABLE "orders" DROP CONSTRAINT IF EXISTS "orders_status_chk";`); }); }); diff --git a/server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts b/server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts new file mode 100644 index 000000000..c45ac7c59 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/emit-drop-if-exists.test.ts @@ -0,0 +1,152 @@ +// Forward drops must tolerate an absent object so a committed chain replays into +// an empty database (#313). Down statements must NOT — `rollbackTo` runs down.sql +// and the ledger delete in one transaction, so a silently-no-op down would record +// the rollback as done. +import { describe, test, expect } from "bun:test"; +import { renderPostgres } from "../src/emit/postgres.js"; +import { renderSqlite } from "../src/emit/sqlite.js"; +import type { ChangeStatus, SchemaSnapshot, TableDescriptor } from "../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +const GONE: TableDescriptor = { + name: "gone", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], +}; + +describe("forward drops tolerate an absent object (#313)", () => { + test("postgres drop-table", () => { + const { up } = renderPostgres([{ kind: "drop-table", table: "gone", status: ALLOWED }]); + expect(up).toContain('DROP TABLE IF EXISTS "gone";'); + }); + + test("postgres drop-view", () => { + const { up } = renderPostgres([{ kind: "drop-view", view: "v_gone", status: ALLOWED }]); + expect(up).toContain('DROP VIEW IF EXISTS "v_gone";'); + }); + + test("postgres drop-index, plain", () => { + const { up } = renderPostgres([ + { kind: "drop-index", table: "t", index: "idx_gone", status: ALLOWED }, + ]); + expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); + }); + + test("postgres drop-index, constraint-backed (#285)", () => { + const { up } = renderPostgres([ + { + kind: "drop-index", + table: "t", + index: "uq_gone", + status: ALLOWED, + restore: { name: "uq_gone", columns: ["a"], unique: true, constraint: "unique" }, + }, + ]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "uq_gone";'); + }); + + test("postgres drop-fk", () => { + const { up } = renderPostgres([{ kind: "drop-fk", table: "t", fk: "fk_gone", status: ALLOWED }]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "fk_gone";'); + }); + + // drop-check IS produced by the diff (diff/index.ts:579, :592) — an evolved + // `field.enum @values` is a live producer. The `renderUp` comment that claimed + // otherwise is deleted by this change. + test("postgres drop-check", () => { + const { up } = renderPostgres([ + { kind: "drop-check", table: "t", check: "t_qty_chk", status: ALLOWED }, + ]); + expect(up).toContain('ALTER TABLE "t" DROP CONSTRAINT IF EXISTS "t_qty_chk";'); + }); + + test("sqlite drop-table", () => { + const { up } = renderSqlite([{ kind: "drop-table", table: "gone", status: ALLOWED }]); + expect(up).toContain('DROP TABLE IF EXISTS "gone";'); + }); + + test("sqlite drop-index", () => { + const { up } = renderSqlite([ + { kind: "drop-index", table: "t", index: "idx_gone", status: ALLOWED }, + ]); + expect(up).toContain('DROP INDEX IF EXISTS "idx_gone";'); + }); + + // Already guarded before this change; pinned so a later sweep cannot un-guard it. + test("sqlite drop-view was already guarded", () => { + const { up } = renderSqlite([{ kind: "drop-view", view: "v_gone", status: ALLOWED }]); + expect(up).toContain('DROP VIEW IF EXISTS "v_gone";'); + }); +}); + +describe("down statements stay bare — a rollback must fail loudly", () => { + test("postgres create-table down", () => { + const { down } = renderPostgres([{ kind: "create-table", table: GONE, status: ALLOWED }]); + expect(down).toContain('DROP TABLE "gone";'); + expect(down).not.toContain("DROP TABLE IF EXISTS"); + }); + + test("postgres create-view down", () => { + const { down } = renderPostgres([ + { + kind: "create-view", + status: ALLOWED, + view: { + name: "v", + sql: "SELECT 1 AS one", + columns: [{ name: "one", sqlType: { kind: "integer", bits: 32 } }], + }, + }, + ]); + expect(down).toContain('DROP VIEW "v";'); + expect(down).not.toContain("DROP VIEW IF EXISTS"); + }); + + test("sqlite create-table down", () => { + const { down } = renderSqlite([{ kind: "create-table", table: GONE, status: ALLOWED }]); + expect(down).toContain('DROP TABLE "gone";'); + expect(down).not.toContain("DROP TABLE IF EXISTS"); + }); +}); + +describe("the recreate-and-copy rebuild drop stays bare — deliberately", () => { + test("sqlite recreate emits a bare DROP TABLE for the table it just copied from", () => { + const expectedSchema: SchemaSnapshot = { + tables: [ + { + name: "orders", + columns: [ + { name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }, + { name: "amount", sqlType: { kind: "integer", bits: 64 }, nullable: false }, + ], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], + }, + ], + views: [], + }; + const { up } = renderSqlite( + [ + { + kind: "change-column-type", + table: "orders", + column: "amount", + from: { kind: "real" }, + to: { kind: "integer", bits: 64 }, + status: ALLOWED, + }, + ], + expectedSchema, + ); + // IF EXISTS here would turn a caught corruption into a silent one: the recipe + // just INSERT…SELECTed out of this exact table. + expect(up).toContain('DROP TABLE "orders";'); + expect(up).not.toContain('DROP TABLE IF EXISTS "orders";'); + }); +}); diff --git a/server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts b/server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts new file mode 100644 index 000000000..7bd1cf6d8 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/emit-postgres-create-schema.test.ts @@ -0,0 +1,94 @@ +// A committed chain must apply to a VIRGIN database (#313). `CREATE TABLE "s"."x"` +// fails there unless the schema exists, and no migration has ever created one — +// `CREATE SCHEMA` appears nowhere in the emitters, only in the ledger's own setup. +import { describe, test, expect } from "bun:test"; +import { renderPostgres } from "../src/emit/postgres.js"; +import type { ChangeStatus, TableDescriptor, ViewDescriptor } from "../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +const t = (name: string, schema?: string): TableDescriptor => ({ + name, + ...(schema !== undefined ? { schema } : {}), + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], +}); + +const v = (name: string, schema?: string): ViewDescriptor => ({ + name, + ...(schema !== undefined ? { schema } : {}), + sql: "SELECT 1 AS one", + columns: [{ name: "one", sqlType: { kind: "integer", bits: 32 } }], +}); + +describe("a chain that creates an object in a non-default schema creates the schema first", () => { + test("emits CREATE SCHEMA IF NOT EXISTS before the table", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "reporting"), status: ALLOWED }]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')).toBeLessThan(up.indexOf("CREATE TABLE")); + }); + + // The spec says "the first OBJECT", not "the first table": a chain whose first + // migration creates only a view in a non-default schema fails identically. + test("emits it for a create-view too", () => { + const { up } = renderPostgres([{ kind: "create-view", view: v("v_x", "reporting"), status: ALLOWED }]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "reporting";'); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "reporting";')).toBeLessThan(up.indexOf("CREATE VIEW")); + }); + + // `create-view` carries the schema in two places; the change's own key wins, the + // same precedence renderCreateView already uses. + test("a create-view's own schema key wins over the descriptor's", () => { + const { up } = renderPostgres([ + { kind: "create-view", view: v("v_x", "descriptor_schema"), schema: "change_schema", status: ALLOWED }, + ]); + expect(up).toContain('CREATE SCHEMA IF NOT EXISTS "change_schema";'); + expect(up).not.toContain('CREATE SCHEMA IF NOT EXISTS "descriptor_schema";'); + }); + + test("emits it once for several objects in the same schema", () => { + const { up } = renderPostgres([ + { kind: "create-table", table: t("x", "reporting"), status: ALLOWED }, + { kind: "create-table", table: t("y", "reporting"), status: ALLOWED }, + { kind: "create-view", view: v("v_x", "reporting"), status: ALLOWED }, + ]); + expect(up.match(/CREATE SCHEMA IF NOT EXISTS "reporting";/g)).toHaveLength(1); + }); + + test("emits one per distinct schema, in sorted order", () => { + const { up } = renderPostgres([ + { kind: "create-table", table: t("x", "zeta"), status: ALLOWED }, + { kind: "create-table", table: t("y", "alpha"), status: ALLOWED }, + ]); + expect(up.indexOf('CREATE SCHEMA IF NOT EXISTS "alpha";')) + .toBeLessThan(up.indexOf('CREATE SCHEMA IF NOT EXISTS "zeta";')); + }); + + test("emits nothing for the default schema", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x"), status: ALLOWED }]); + expect(up).not.toContain("CREATE SCHEMA"); + }); + + test("emits nothing for an explicit 'public'", () => { + const { up } = renderPostgres([{ kind: "create-table", table: t("x", "public"), status: ALLOWED }]); + expect(up).not.toContain("CREATE SCHEMA"); + }); + + // Not filler: dropping a schema on rollback would destroy objects this tool does + // not own and cannot restore. + test("the down does NOT drop the schema", () => { + const { down } = renderPostgres([{ kind: "create-table", table: t("x", "reporting"), status: ALLOWED }]); + expect(down).not.toContain("DROP SCHEMA"); + }); + + // A migration that only DROPS from a schema must not resurrect it. + test("a drop-only migration emits no CREATE SCHEMA", () => { + const { up } = renderPostgres([ + { kind: "drop-table", table: "x", schema: "reporting", status: ALLOWED }, + ]); + expect(up).not.toContain("CREATE SCHEMA"); + }); +}); diff --git a/server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts b/server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts index 021ec68c8..09c1af2cd 100644 --- a/server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts +++ b/server/typescript/packages/migrate-ts/test/integration/pg-constraint-backed-index-285.test.ts @@ -138,9 +138,15 @@ describe("#285 — constraint-backed index drops as a constraint (PG)", () => { expect(d.changes.some((c) => c.kind === "drop-index")).toBe(true); const sqlText = emit(d.changes, { dialect: "postgres" }).up; - // The whole point: a bare DROP INDEX here is what Postgres refuses. - expect(sqlText).not.toMatch(/DROP INDEX "?work_item_message_id_unique/); - expect(sqlText).toMatch(/ALTER TABLE .*DROP CONSTRAINT "work_item_message_id_unique"/); + // The whole point: a DROP INDEX here is what Postgres refuses. + // + // `(IF EXISTS )?` is load-bearing on the NEGATIVE assertion. #313 put `IF EXISTS` + // on every forward drop, so without the optional group this stops matching because + // the SPELLING changed — passing for a reason unrelated to #285, and continuing to + // pass if #285 fully regressed. Verified by reverting the drop-index arm and + // watching this line go red. + expect(sqlText).not.toMatch(/DROP INDEX (IF EXISTS )?"?work_item_message_id_unique/); + expect(sqlText).toMatch(/ALTER TABLE .*DROP CONSTRAINT IF EXISTS "work_item_message_id_unique"/); // Pre-fix this throws: cannot drop index … because constraint … requires it. await applyRaw(kysely, sqlText); diff --git a/server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts b/server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts new file mode 100644 index 000000000..5206cd2a1 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/integrity/replay-emitted-chain.test.ts @@ -0,0 +1,112 @@ +// #313, end to end: the EMITTER's own output, written by `writeMigration`, applied +// by `applyPending` into an empty database. +// +// Every prior defect in this area (#226/#241, #243, #255, #285, and 0.21.4's +// `BEGIN TRANSACTION` finding) shared one shape — SQL proven statement-by-statement +// and never proven through the tool that applies it. `applyPending` splits and +// rewrites statements before executing them, so an emit-level string assertion +// cannot see this class of bug. A hand-written-SQL test cannot either: it stays +// green no matter what the emitter does. +import { describe, test, expect } from "bun:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { emit } from "../../src/emit/index.js"; +import { writeMigration } from "../../src/write-migration.js"; +import { applyPending } from "../../src/apply/apply.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import type { Change, ChangeStatus } from "../../src/types.js"; + +const ALLOWED: ChangeStatus = { state: "allowed" }; + +// Exactly the reported shape: another tool owned `theirs`, so the diff proposed +// dropping it, and no migration in the chain ever created it. +const REPORTED: Change[] = [ + { + kind: "create-table", + status: ALLOWED, + table: { + name: "mine", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], + }, + }, + { kind: "drop-table", table: "theirs", status: ALLOWED }, +]; + +describe("an EMITTED chain applies to an empty database (#313)", () => { + for (const dialect of ["sqlite", "postgres"] as const) { + test(`${dialect}: emit → writeMigration → applyPending, from empty`, async () => { + const dir = mkdtempSync(join(tmpdir(), `replay-emitted-${dialect}-`)); + const engine = await openReplayEngine(dialect); + try { + await writeMigration(emit(REPORTED, { dialect }), { dir, slug: "init" }); + const applied = await applyPending(engine.db, dir, { dryRun: false, dialect }); + expect(applied.applied).toHaveLength(1); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + } + + // A chain creating a table in a non-default schema needs `CREATE SCHEMA` ahead of + // it, which no migration used to emit. Postgres-only — sqlite has no schemas. + test("postgres: an emitted chain creating a non-default schema's table applies", async () => { + const dir = mkdtempSync(join(tmpdir(), "replay-emitted-schema-")); + const engine = await openReplayEngine("postgres"); + try { + await writeMigration( + emit( + [ + { + kind: "create-table", + status: ALLOWED, + table: { + name: "x", + schema: "reporting", + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], + }, + }, + ], + { dialect: "postgres" }, + ), + { dir, slug: "init" }, + ); + const applied = await applyPending(engine.db, dir, { dryRun: false, dialect: "postgres" }); + expect(applied.applied).toHaveLength(1); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + // The control: without it the cases above could pass because `applyPending` + // swallows a failing statement rather than because the emitter stopped writing + // one. This proves the assertion has teeth. + test("the control: a HAND-WRITTEN bare drop still fails the replay", async () => { + const dir = mkdtempSync(join(tmpdir(), "replay-emitted-control-")); + const engine = await openReplayEngine("sqlite"); + try { + await writeMigration( + { + up: 'CREATE TABLE "mine" (id INTEGER NOT NULL PRIMARY KEY);\n\nDROP TABLE "theirs";', + down: 'DROP TABLE "mine";', + }, + { dir, slug: "init" }, + ); + await expect(applyPending(engine.db, dir, { dryRun: false, dialect: "sqlite" })) + .rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts b/server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts new file mode 100644 index 000000000..9ad101dfd --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/integrity/replay-scoped.test.ts @@ -0,0 +1,95 @@ +// A project declaring `migrate.scope` writes the OTHER owner's tables into its +// committed snapshot on purpose (`carryForwardOutOfScope`), and its chain — also on +// purpose — never creates them. Without threading the scope decision, every replay +// of such a project reports those tables as missing. +import { describe, test, expect } from "bun:test"; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { verifyReplay } from "../../src/verify/replay.js"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import type { SchemaSnapshot, TableDescriptor } from "../../src/types.js"; + +function chainWith(upSql: string): string { + const dir = mkdtempSync(join(tmpdir(), "replay-scoped-")); + mkdirSync(join(dir, "20260101000000-init"), { recursive: true }); + writeFileSync(join(dir, "20260101000000-init", "up.sql"), upSql, "utf8"); + writeFileSync(join(dir, "20260101000000-init", "down.sql"), 'DROP TABLE "mine";', "utf8"); + return dir; +} + +// `id INTEGER NOT NULL PRIMARY KEY`, not a bare `INTEGER PRIMARY KEY`: sqlite reports +// notnull=0 for the latter, so `nullable: false` below would read as drift and both +// tests would answer a question about column nullability rather than about scope. +const CHAIN = 'CREATE TABLE "mine" (id INTEGER NOT NULL PRIMARY KEY);'; + +const table = (name: string): TableDescriptor => ({ + name, + columns: [{ name: "id", sqlType: { kind: "integer", bits: 64 }, nullable: false }], + indexes: [], + foreignKeys: [], + checks: [], + primaryKey: ["id"], +}); + +const SNAPSHOT: SchemaSnapshot = { tables: [table("mine"), table("theirs")], views: [] }; + +describe("verifyReplay honours migrate.scope", () => { + test("an out-of-scope table in the snapshot is not reported as missing", async () => { + const dir = chainWith(CHAIN); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, + dialect: "sqlite", + migrationsDir: dir, + snapshot: SNAPSHOT, + // Qualified `.`; an absent schema normalizes to the Postgres + // default, and every sqlite object lands under that same constant prefix. + governed: { outOfScope: ["public.theirs"] }, + }); + expect(result.ok).toBe(true); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + // The control is what makes the case above non-vacuous: it proves the difference + // comes from `governed`, not from a fixture that was trivially green. + test("without `governed`, the same case reports drift — the control", async () => { + const dir = chainWith(CHAIN); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, + dialect: "sqlite", + migrationsDir: dir, + snapshot: SNAPSHOT, + }); + expect(result.ok).toBe(false); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + // An empty `outOfScope` must not become a way to suppress a real difference. + test("an empty scope still reports drift", async () => { + const dir = chainWith(CHAIN); + const engine = await openReplayEngine("sqlite"); + try { + const result = await verifyReplay({ + db: engine.db, + dialect: "sqlite", + migrationsDir: dir, + snapshot: SNAPSHOT, + governed: { outOfScope: [] }, + }); + expect(result.ok).toBe(false); + } finally { + await engine.dispose(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts b/server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts index cbe5b52ff..afcf65795 100644 --- a/server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts +++ b/server/typescript/packages/migrate-ts/test/unit/emit-postgres.test.ts @@ -133,7 +133,8 @@ describe("renderPostgres — table-level", () => { test("drop-table", () => { const changes: Change[] = [{ kind: "drop-table", table: "legacy", status: ALLOWED }]; const { up } = emit(changes, { dialect: "postgres" }); - expect(norm(up)).toBe(`DROP TABLE "legacy";`); + // #313 — forward drops carry IF EXISTS so a committed chain replays from empty. + expect(norm(up)).toBe(`DROP TABLE IF EXISTS "legacy";`); }); }); @@ -161,7 +162,7 @@ describe("renderPostgres — indexes + FKs", () => { test("drop-index", () => { const changes: Change[] = [{ kind: "drop-index", table: "users", index: "old_idx", status: ALLOWED }]; const { up } = emit(changes, { dialect: "postgres" }); - expect(norm(up)).toBe(`DROP INDEX "old_idx";`); + expect(norm(up)).toBe(`DROP INDEX IF EXISTS "old_idx";`); }); test("add-fk with ON DELETE CASCADE", () => { @@ -184,7 +185,7 @@ describe("renderPostgres — indexes + FKs", () => { test("drop-fk", () => { const changes: Change[] = [{ kind: "drop-fk", table: "weeks", fk: "weeks_program_id_fk", status: ALLOWED }]; const { up } = emit(changes, { dialect: "postgres" }); - expect(norm(up)).toBe(`ALTER TABLE "weeks" DROP CONSTRAINT "weeks_program_id_fk";`); + expect(norm(up)).toBe(`ALTER TABLE "weeks" DROP CONSTRAINT IF EXISTS "weeks_program_id_fk";`); }); }); diff --git a/server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts b/server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts index 0a20d9b8f..f40b1ac6b 100644 --- a/server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts +++ b/server/typescript/packages/migrate-ts/test/unit/emit-sqlite.test.ts @@ -58,7 +58,8 @@ describe("renderSqlite — create-table", () => { describe("renderSqlite — table-level + indexes", () => { test("drop-table", () => { const { up } = emit([{ kind: "drop-table", table: "old", status: ALLOWED }], { dialect: "sqlite" }); - expect(norm(up)).toBe(`DROP TABLE "old";`); + // #313 — forward drops carry IF EXISTS so a committed chain replays from empty. + expect(norm(up)).toBe(`DROP TABLE IF EXISTS "old";`); }); test("rename-table", () => { const { up } = emit([{ kind: "rename-table", from: "p", to: "a", status: ALLOWED }], { dialect: "sqlite" }); @@ -74,7 +75,7 @@ describe("renderSqlite — table-level + indexes", () => { }); test("drop-index", () => { const { up } = emit([{ kind: "drop-index", table: "u", index: "i", status: ALLOWED }], { dialect: "sqlite" }); - expect(norm(up)).toBe(`DROP INDEX "i";`); + expect(norm(up)).toBe(`DROP INDEX IF EXISTS "i";`); }); }); @@ -360,7 +361,7 @@ describe("renderSqlite — drop-index vs drop-column (#255 generalized)", () => ], { dialect: "sqlite" }, ); - const idxDropIndex = up.indexOf('DROP INDEX "uniqueCode"'); + const idxDropIndex = up.indexOf('DROP INDEX IF EXISTS "uniqueCode"'); const idxDropColumn = up.indexOf('ALTER TABLE "programs" DROP COLUMN "code"'); expect(idxDropIndex).toBeGreaterThanOrEqual(0); expect(idxDropColumn).toBeGreaterThanOrEqual(0); diff --git a/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts b/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts new file mode 100644 index 000000000..9f4ca0877 --- /dev/null +++ b/server/typescript/packages/migrate-ts/test/unit/replay-engine.test.ts @@ -0,0 +1,194 @@ +import { describe, test, expect } from "bun:test"; +import { sql } from "kysely"; +import { openReplayEngine } from "../../src/verify/replay-engine.js"; +import { introspect } from "../../src/introspect/index.js"; + +describe("openReplayEngine", () => { + // PGlite is Postgres compiled to WASM, and Emscripten propagates the WASM + // program's internal exit status into `process.exitCode` — it becomes 99 on the + // FIRST QUERY, not on teardown, and stays there. `bin/meta.ts` ends with + // `process.exit(code)`, so the shipped CLI overrides it and was never affected; + // anything that does NOT force its own exit inherits it. That is why this whole + // FILE used to exit 99 with 11 pass / 0 fail, turning two `ci-local.sh --only ts` + // gates red while reporting no failing test — and why an embedder calling + // `openReplayEngine` programmatically would silently exit non-zero on success. + // + // This must run on the postgres engine specifically; sqlite never touches WASM. + // + // It runs in a CHILD PROCESS, and that is the whole design. Two in-process + // shapes were tried first and both prove nothing: + // - `const before = process.exitCode; … expect(after).toBe(before)` passes + // vacuously once any earlier test has opened a postgres engine, because + // 99 === 99. + // - Pinning a clean baseline with `process.exitCode = 0` first fixes that but + // then captures 0 rather than the pristine `undefined`, so it never + // exercises the `?? 0` that the real fix turns on — it passed against the + // broken implementation. + // The only faithful assertion is the one the CI gate itself makes: what a fresh + // process actually EXITS with, starting from a pristine `process.exitCode`. + test("postgres: does not leak PGlite's WASM exit status into the host process", async () => { + const script = ` + const { openReplayEngine } = await import("${import.meta.dir}/../../src/verify/replay-engine.ts"); + const { sql } = await import("kysely"); + const engine = await openReplayEngine("postgres"); + try { await sql\`SELECT 1\`.execute(engine.db); } finally { await engine.dispose(); } + `; + const proc = Bun.spawn(["bun", "-e", script], { + cwd: `${import.meta.dir}/../..`, + stdout: "pipe", + stderr: "pipe", + }); + const code = await proc.exited; + expect(code).toBe(0); + }, 60_000); + + test("sqlite: gives an empty, usable database", async () => { + const engine = await openReplayEngine("sqlite"); + try { + await sql`CREATE TABLE t (id integer primary key)`.execute(engine.db); + await sql`INSERT INTO t (id) VALUES (1)`.execute(engine.db); + const rows = await sql<{ id: number }>`SELECT id FROM t`.execute(engine.db); + expect(rows.rows).toHaveLength(1); + } finally { + await engine.dispose(); + } + }); + + test("postgres: gives an empty, usable database with real PG DDL", async () => { + const engine = await openReplayEngine("postgres"); + try { + // Schema namespacing + a CHECK — neither is expressible in sqlite, so this + // proves the postgres engine really is Postgres. + await sql`CREATE SCHEMA IF NOT EXISTS "reporting"`.execute(engine.db); + await sql`CREATE TABLE "reporting"."t" (id integer primary key, n integer CHECK (n > 0))`.execute(engine.db); + const rows = await sql<{ table_name: string }>` + SELECT table_name FROM information_schema.tables WHERE table_schema = 'reporting' + `.execute(engine.db); + expect(rows.rows.map((r) => r.table_name)).toContain("t"); + } finally { + await engine.dispose(); + } + }); + + // THE defect that nearly shipped, and the reason this case exists on BOTH dialects: + // `applyPending` runs every migration file inside a transaction, so a table created + // by migration 1 must be visible to migration 2 and to the introspection that + // follows. Under `@libsql/kysely-libsql`, `:memory:` gives each CONNECTION its own + // database, so a tx-created table vanishes the instant the transaction's connection + // is released — a whole chain would replay into a series of throwaway databases and + // the gate would pass having proved nothing. Only a cross-transaction assertion + // catches it; the non-transactional cases above all passed. + for (const dialect of ["sqlite", "postgres"] as const) { + test(`${dialect}: a table created inside a transaction survives it`, async () => { + const engine = await openReplayEngine(dialect); + try { + await engine.db.transaction().execute(async (trx) => { + await sql`CREATE TABLE in_tx (id integer primary key)`.execute(trx); + }); + // A second transaction must SEE the first one's table, the way migration 2 + // sees migration 1's. + await engine.db.transaction().execute(async (trx) => { + await sql`INSERT INTO in_tx (id) VALUES (1)`.execute(trx); + }); + const snap = await introspect(engine.db, dialect); + expect(snap.tables.map((x) => x.name)).toContain("in_tx"); + } finally { + await engine.dispose(); + } + }); + } + + // applyPending runs each migration file inside a kysely transaction and takes a + // pg advisory lock on postgres. Both must work through the shim, or the gate + // fails for a reason that has nothing to do with the chain under test. + test("postgres: transactions roll back, and advisory locks work", async () => { + const engine = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE t (id integer primary key)`.execute(engine.db); + await expect( + engine.db.transaction().execute(async (trx) => { + await sql`INSERT INTO t (id) VALUES (1)`.execute(trx); + throw new Error("boom"); + }), + ).rejects.toThrow(/boom/); + const after = await sql<{ c: string }>`SELECT count(*)::text AS c FROM t`.execute(engine.db); + expect(after.rows[0]?.c).toBe("0"); + + await sql`SELECT pg_advisory_lock(hashtext('meta'))`.execute(engine.db); + await sql`SELECT pg_advisory_unlock(hashtext('meta'))`.execute(engine.db); + } finally { + await engine.dispose(); + } + }); + + // The whole gate rests on this: a statement against a missing object must REJECT. + test("postgres: dropping a missing table rejects — the #313 signal", async () => { + const engine = await openReplayEngine("postgres"); + try { + await expect(sql`DROP TABLE "theirs"`.execute(engine.db)).rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + } + }); + + test("sqlite: dropping a missing table rejects — the #313 signal", async () => { + const engine = await openReplayEngine("sqlite"); + try { + await expect(sql`DROP TABLE "theirs"`.execute(engine.db)).rejects.toThrow(/theirs/); + } finally { + await engine.dispose(); + } + }); + + // `--replay-snapshot` introspects the replayed database, so the postgres + // introspector — information_schema, pg_catalog, pg_get_viewdef — has to work + // against the in-process engine or that whole tier is unreachable. + test("postgres: introspect reads back a table and a view", async () => { + const engine = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE t (id integer primary key, n integer NOT NULL)`.execute(engine.db); + await sql`CREATE VIEW v AS SELECT id FROM t`.execute(engine.db); + const snap = await introspect(engine.db, "postgres"); + expect(snap.tables.map((x) => x.name)).toContain("t"); + expect(snap.views.map((x) => x.name)).toContain("v"); + } finally { + await engine.dispose(); + } + }); + + test("two engines of the same dialect do not share state", async () => { + const a = await openReplayEngine("sqlite"); + const b = await openReplayEngine("sqlite"); + try { + await sql`CREATE TABLE only_in_a (id integer)`.execute(a.db); + const rows = await sql<{ name: string }>`SELECT name FROM sqlite_master WHERE name = 'only_in_a'`.execute(b.db); + expect(rows.rows).toHaveLength(0); + } finally { + await a.dispose(); + await b.dispose(); + } + }); + + test("two postgres engines do not share state either", async () => { + const a = await openReplayEngine("postgres"); + const b = await openReplayEngine("postgres"); + try { + await sql`CREATE TABLE only_in_a (id integer)`.execute(a.db); + const rows = await sql<{ c: string }>` + SELECT count(*)::text AS c FROM information_schema.tables WHERE table_name = 'only_in_a' + `.execute(b.db); + expect(rows.rows[0]?.c).toBe("0"); + } finally { + await a.dispose(); + await b.dispose(); + } + }); + + // A caller disposes in a `finally` after an early return, so a double dispose + // must not throw. + test("dispose is idempotent", async () => { + const engine = await openReplayEngine("sqlite"); + await engine.dispose(); + await engine.dispose(); + }); +}); diff --git a/server/typescript/packages/sdk/src/config.ts b/server/typescript/packages/sdk/src/config.ts index 5f16a466e..0e70467d7 100644 --- a/server/typescript/packages/sdk/src/config.ts +++ b/server/typescript/packages/sdk/src/config.ts @@ -31,6 +31,7 @@ export const AllowTokenEnum = z.enum([ "adopt-view", "nullable-to-not-null", "drop-identity-default", + "drop-unmanaged", ]); // .strict(), like every other object in this schema: `.partial()` alone leaves diff --git a/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts b/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts new file mode 100644 index 000000000..f9f387a93 --- /dev/null +++ b/server/typescript/packages/sdk/test/source-resolution-conformance.test.ts @@ -0,0 +1,102 @@ +// Runs the shared source-resolution corpus against the TypeScript reference +// implementation. Every port ships an equivalent runner reading this same file. +import { describe, expect, test } from "bun:test"; +import { mkdtemp, mkdir, readFile, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, relative, resolve, sep } from "node:path"; +import { resolveCollection } from "../src/collection.js"; + +interface Case { + readonly name: string; + readonly tree: Record; + readonly config: unknown | null; + /** Project-root-relative directory the resolver is invoked against; default + * ".". `config` (when non-null) is written under this directory's + * `.metaobjects/config.json`. See the corpus README, "Shape". */ + readonly resolveFrom?: string; + readonly expectFiles?: readonly string[]; + /** A string pins the exact error code raised; `true` pins only that + * resolution RAISES — the malformed-config error code is deliberately not + * pinned cross-port (see the corpus README). */ + readonly expectError?: string | true; + /** Optional: linkPath -> targetPath, both project-root-relative, materialized + * AFTER `tree` (I1 — a symlinked source root, or a symlinked subdirectory + * inside a walked tree). */ + readonly symlinks?: Record; +} + +const CORPUS = resolve( + import.meta.dir, + "../../../../../fixtures/source-resolution-conformance/cases.json", +); + +/** Materializes `c.tree` under a fresh temp root and, when `c.config` is + * non-null, writes it to `/.metaobjects/config.json`. Returns + * both the project root (`expectFiles` is relative to this) and the + * directory the resolver should be invoked against (`resolveFrom`-relative + * to the root, defaulting to the root itself). */ +async function materialize(c: Case): Promise<{ root: string; resolveDir: string }> { + const root = await mkdtemp(join(tmpdir(), "mo-src-conf-")); + for (const [rel, content] of Object.entries(c.tree)) { + const abs = join(root, rel); + await mkdir(dirname(abs), { recursive: true }); + await writeFile(abs, content); + } + // Materialized AFTER tree — see the Case.symlinks doc. + for (const [linkRel, targetRel] of Object.entries(c.symlinks ?? {})) { + const linkAbs = join(root, linkRel); + await mkdir(dirname(linkAbs), { recursive: true }); + await symlink(join(root, targetRel), linkAbs, "dir"); + } + const resolveDir = resolve(root, c.resolveFrom ?? "."); + if (c.config !== null) { + await mkdir(join(resolveDir, ".metaobjects"), { recursive: true }); + await writeFile( + join(resolveDir, ".metaobjects", "config.json"), + JSON.stringify(c.config, null, 2), + ); + } + return { root, resolveDir }; +} + +const cases: Case[] = JSON.parse(await readFile(CORPUS, "utf8")).cases; + +describe("source-resolution conformance", () => { + test("corpus is non-empty (a silent zero-case run is a failed gate)", () => { + expect(cases.length).toBeGreaterThan(0); + }); + + for (const c of cases) { + test(c.name, async () => { + const { root, resolveDir } = await materialize(c); + if (c.expectError !== undefined) { + let thrown: unknown; + let threw = false; + try { + await resolveCollection(resolveDir, { explicitDir: resolveDir }); + } catch (e) { + threw = true; + thrown = e; + } + expect(threw).toBe(true); + // A string pins the exact code; `true` only pins that it raises — see + // the `expectError` type doc above. + if (typeof c.expectError === "string") { + expect((thrown as { code?: string }).code).toBe(c.expectError); + } + return; + } + // A case with neither `expectFiles` nor `expectError` is a malformed corpus + // entry, not "expect zero files" — `?? []` here would silently pass such a + // case instead of failing loudly on it (this is the TS-runner-specific half + // of the "assert count, not just presence" family of fixes; see the C#/Java/ + // Python runners' analogous length assertions below the set comparison). + if (c.expectFiles === undefined) { + throw new Error(`corpus case "${c.name}" has neither expectFiles nor expectError`); + } + const collection = await resolveCollection(resolveDir, { explicitDir: resolveDir }); + const got = collection.files.map((f) => relative(root, f).split(sep).join("/")).sort(); + expect(got).toEqual([...c.expectFiles].sort()); + }); + } +});