Skip to content

CONFORMANCE.md

Latest commit

 

History

History
262 lines (216 loc) · 19 KB

CONFORMANCE.md

File metadata and controls

262 lines (216 loc) · 19 KB
Raw

Conformance coverage

The MetaObjects standard ships 21 shared conformance corpora under fixtures/. Every port runs every corpus that is applicable to it and asserts the same expected behaviour against the same fixtures. This page is the inverse index: fixture → feature doc + per-port pass status, and it is the single place per-corpus counts are maintained (the port READMEs deliberately point here instead of restating them).

Not every corpus is a five-port corpus, and that asymmetry is deliberate rather than a gap — schema migration is TypeScript-owned (ADR-0015), Kotlin runs on the JVM and inherits the Java loader/render/extract engines instead of re-implementing them, and a few corpora gate tooling that only one port ships. The dark cells below say which is which.

If you are coming from a feature doc's ## Verified by section, you are in the right place. If you are wondering whether a particular fixture has a human-readable explanation somewhere, look it up in the "Fixture-to-doc mapping" table below.

Per-corpus totals

Counts are fixture directories (or scenario files, where a corpus is file-shaped); regenerate with ls -d fixtures/<corpus>/*/ | wc -l.

Corpus Fixtures TS Java Kotlin C# Python
fixtures/conformance/ (metamodel) 286 ✓ ✓ inherits via metadata-ktx ✓ ✓
fixtures/yaml-conformance/ 15 15 / 15 14 / 15 (1 ledgered: yaml-quoted-leading-zero — Java pipeline strips quotes off "007") inherits via Java 14 / 15 (1 ledgered: error-yaml-coerced-hex-in-string — YamlDotNet doesn't coerce 0xFF) 15 / 15
fixtures/verify-conformance/ 31 ✓ ✓ inherits via Java ✓ ✓
fixtures/verify-strict-conformance/ 1 ✓ — — — ✓
fixtures/render-conformance/ 15 ✓ ✓ inherits via Java ✓ ✓
fixtures/extract-conformance/ 33 ✓ ✓ inherits the shared JVM engine ✓ ✓
fixtures/output-prompt-conformance/ 14 ✓ ✓ ✓ ✓ ✓
fixtures/persistence-conformance/ 33 (27 query + 6 migration) all 33 27 query (migrations TS-only, ADR-0015) 27 query (via Exposed) 27 query 27 query
fixtures/api-contract-conformance/ 41 (26 core + 8 tph + 3 m2m + 2 jsonb + 2 write-through) ✓ (Fastify reference + generated lane) ✓ (embedded HTTP + JDBC) ✓ (embedded HTTP + Exposed) ✓ (HttpListener + Npgsql) ✓ (FastAPI + pg8000)
fixtures/validation-conformance/ 16 cases ✓ ✓ ✓ ✓ ✓
fixtures/registry-conformance/ 1 canonical manifest ✓ (reference emitter) ✓ ✓ ✓ ✓
fixtures/object-model-conformance/ 1 shared metadata fixture (per-port scenarios) ✓ ✓ ✓ ✓ ✓
fixtures/codegen-conformance/ 4 ✓ ✓ ✓ ✓ ✓
fixtures/template-codegen-conformance/ 3 ✓ ✓ ✓ ✓ ✓
fixtures/template-output-render-conformance/ 5 ✓ ✓ ✓ ✓ ✓
fixtures/generator-registry-conformance/ 1 canonical manifest ✓ ✓ ✓ ✓ ✓
fixtures/provider-composition-conformance/ 9 (5 error-shape + 4 compose-load) ✓ ✓ — (JVM registry via Java) ✓ ✓
fixtures/source-resolution-conformance/ 25 cases ✓ (reference implementation) ✓ inherits via Java ✓ ✓
fixtures/scope-conformance/ 10 cases ✓ (reference implementation) — — — —
fixtures/agent-context-conformance/ 4 ✓ (the emitter is TS-owned) — — — —
fixtures/metamodel-docs/ 1 ✓ (docs emit is TS-owned) — — — —

A ✓ means the port runs the corpus green; an explicit n / m is used where a port carries a ledgered divergence. The two ledgered YAML fixtures are documented library-vs-pipeline divergences (see the _comment block in each port's yaml-conformance-expected-failures.json for the full reconciliation note). They are tracked as known-gaps rather than silently patched — the runner treats listed fixtures as passing, but a future port-level reconciliation pass would close them.

Split coverage: where a feature is gated, and where it deliberately is not

The matrix is corpus-shaped, so a feature whose coverage splits — across several corpora, or between its vocabulary and its behaviour — never gets a row of its own, and its boundary ends up discoverable only from the feature doc. The split ones are recorded here instead.

Capability requirements (features/requirements.md):

  • Vocabulary — gated in all five ports. The types, their attributes and their closed enums travel in registry-conformance's byte-matched manifest, which every port reproduces exactly, and accept/reject behaviour is pinned by requirement-* fixtures in fixtures/conformance/. A port that drifts on what it will load fails.
  • Checks — TypeScript only, by decision. The meta verify diagnostics over requirements ship in the TypeScript CLI; the other ports load and validate and stop there. Same call as ADR-0015: one implementation of a build-time gate rather than five, where the gate is not a per-port runtime concern. verify-conformance therefore holds no requirement cases, and that absence is deliberate rather than a gap.

Stated as mechanisms rather than as a list of attribute names on purpose — the requirement vocabulary has a breaking change scheduled (FR-038), which moves what the manifest contains without moving the boundary between the two halves.

How to tell a deliberate split from a real parity gap, since the two look identical in the matrix — both show one port covered and four blank. Ask what the uncovered ports claim. Here they claim nothing: they load requirement vocabulary and stop, exactly as the feature doc says. Contrast {{#hasField}} in 0.23.1, where the JVM emitted has<Field>() onto generated payload records and verify accepted the section, while no render engine in any port implemented the other half — two ports shipping halves of one promise, with no fixture that could see it. A split is deliberate when no port makes a claim the corpus would have to check; it is a gap when some port already made one.

A hazard for whoever ships the scheduled removal. Contract prose lives inside attr descriptions in expected-registry.json, where five ports byte-match it and nothing else gates it — @verifiedBy carries "verify checks each name EXISTS and is not skipped; it never runs them", and @trackedBy carries "NOT resolved by verify, which does not reach the network". Deleting an attribute deletes its description, and with it the only gated statement of that guarantee. Sweep every description being removed for contract prose and rehome it in the same change, not after — the same class as the ADR-0047 renumbering trap, where a string that reads like prose is actually a gated artifact.

Per-port runners + commands:

Port Metamodel + YAML + render + verify Persistence API contract
TypeScript cd server/typescript && bun test (per-package, ~3s) scripts/integration-test.sh ts (needs Docker) cd server/typescript/packages/integration-tests && bun test test/api-contract.test.ts (needs Docker)
Java cd server/java && mvn -pl metadata test (and per-tier -pl render, etc.) scripts/integration-test.sh java (needs Docker) mvn -f server/java/integration-tests/pom.xml test -Dtest=ApiContractConformanceTest (needs Docker)
Kotlin cd server/java && mvn -pl codegen-kotlin test (snapshot suite) mvn -f server/java/integration-tests-kotlin/pom.xml test (needs Docker) mvn -f server/java/integration-tests-kotlin/pom.xml test -Dtest=ApiContractConformanceTest (needs Docker)
C# dotnet test (per project) scripts/integration-test.sh csharp (needs Docker) dotnet test server/csharp/MetaObjects.IntegrationTests/MetaObjects.IntegrationTests.csproj --filter "FullyQualifiedName~ApiContractConformanceTest" (needs Docker)
Python pytest (per package) scripts/integration-test.sh python (needs Docker) cd server/python && uv run --extra integration pytest tests/integration/test_api_contract.py (needs Docker)

The persistence-conformance corpus is intentionally on-demand — none of the unit-test runners (bun test, dotnet test, pytest, mvn test) pull Docker. scripts/integration-test.sh is the entry point and is wired into docs/RELEASING.md §2b as the pre-latest gate.

Fixture-to-doc mapping

fixtures/conformance/ — metamodel loader + canonical serializer (286)

Fixture prefix Feature doc
loader-basic-*, error-parse-* features/loaders.md
extends-*, error-extends-* features/entities.md (inheritance)
identity-*, subtype-entity-*, subtype-value-* features/entities.md (identity / entity vs value)
attr-*, error-attr-*, error-reserved-word-as-attr features/entities.md (attributes)
doc-common-attrs-* features/entities.md (documentation common attrs)
auto-set-on-* features/entities.md (auto-set timestamps)
attr-filter-*, loader-filterable-*, warning-filterable-*, layout-data-grid-*, error-data-grid-* features/entities.md (filter / sort / grid)
overlay-* features/entities.md (overlay / merge)
merge-three-way-no-conflict, error-merge-conflict-attr, warning-duplicate-declaration features/loaders.md (multi-file merge attribution, FR5c)
field-string-*, field-decimal-*, field-object-storage-*, error-field-object-storage-* features/field-types.md
currency-* features/field-types.md (currency)
enum-*, error-enum-*, warning-enum-* features/field-types.md (enum)
source-rdb-*, source-db-table-*, source-db-view-*, source-multi-source-*, error-source-* features/source-kinds.md
relationship-*, error-unknown-relationship-* features/relationships.md
template-*, error-template-* features/templates-and-payloads.md
origin-*, error-origin-* features/templates-and-payloads.md (payload origins)
projection-*, error-projection-*, field-readonly-on-view-projection features/source-kinds.md (projections + the object taxonomy, ADR-0028)
requirement-*, error-unknown-attr-requirement features/requirements.md (vocabulary only — the meta verify checks are TS-owned; see "Split coverage" above)
smoke-empty-metadata features/entities.md

fixtures/yaml-conformance/ (15)

All 15 fixtures → features/yaml-authoring.md. The corpus splits into 7 happy-path fixtures (sigil-free attrs, array suffix, anchor/alias, block scalars, mixed bare-and-prefixed, quoted leading zero, etc.) and 6 error-yaml-* fixtures that pin the YAML 1.1 coercion guards (bool / null / hex in string contexts; numeric in enum contexts; reserved-as-attr).

fixtures/render-conformance/ (15)

All 15 fixtures → features/templates-and-payloads.md (render engine output section). 4 are end-to-end shape examples (prompt / email / spreadsheet / CSV-injection escape); 10 pin Mustache-engine semantics — dotted-path lookup, parent-context fallthrough, falsy/empty-array section behavior, inverted sections, nested partials, standalone-tag whitespace stripping, raw-HTML bypass, trailing-newline preservation, and unicode multibyte handling.

fixtures/verify-conformance/ (31)

All 31 fixtures → features/migrations-and-drift.md (template drift section — Renderer.verify).

fixtures/persistence-conformance/ (33 — 27 query + 6 migration)

fixtures/api-contract-conformance/ (41)

All 41 scenarios → features/api-contract.md (cross-port REST API URL grammar + JSON wire format). Verifies every backend's emitted CRUD routes answer identically over HTTP — list / get / create / patch+put / delete, plus pagination (limit/offset), sort (sort=field:dir), the withCount=1 envelope, the not_found / invalid_sort error envelopes, and the 201 / 204 status codes.

The corpus also covers the 9 cross-port filter operators (eq, ne, gt, gte, lt, lte, in, like, isNull) plus the implicit-AND combinator and 2 error shapes (invalid_filter_field / invalid_filter_op) under the URL grammar ?filter[<field>][<op>]=<value> (FR-009). On top of the 26 core scenarios the corpus carries four sub-corpora — tph/ (8, single-table inheritance), m2m/ (3), jsonb/ (2, typed value-object columns) and write-through/ (2, table-write + view-read entities). All 5 ports — TS, Java, Kotlin, C#, Python — run it in BOTH lanes: a hand-rolled reference server and the port's own GENERATED API artifact booted over HTTP.

fixtures/source-resolution-conformance/ (25 cases)

All 25 cases → features/metadata-sources.md (how a declared sources set resolves to a file list). Companion to scope-conformance/ below — sources decides which files are read, scope filters what is emitted from them. The corpus is file-shaped: one committed cases.json, read directly by every port's runner, with no per-port fixture and no ledger.

It pins the resolved file SET for a declared sources list — the default directory, replacement-not-merge, the relative-path base (the directory HOLDING .metaobjects/, never the process cwd), recursive directory walking, case-insensitive extension matching, union-with-de-duplication, and every error condition (an unresolvable path, an unsupported resource/package kind, and a malformed config — "expectError": true pins only that resolution RAISES, since which error code it raises with is deliberately NOT a cross-port contract; see the corpus README). All four CLI surfaces run it — TypeScript (the reference implementation), C#, Python, and Java (Kotlin inherits it, since Kotlin has no CLI entry point of its own and runs through the same Maven plugin as Java): server/typescript/packages/sdk/test/source-resolution-conformance.test.ts, server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs, server/python/tests/conformance/test_source_resolution_conformance.py, and server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java.

fixtures/scope-conformance/ (10 cases)

All 10 cases → features/metadata-sources.md (the scope pattern grammar). The corpus is file-shaped: one committed cases.json, read directly by every port's runner, with no per-port fixture and no ledger.

It pins the semantics of a consumer's include/exclude scope over fully-qualified names — * matches within one :: segment and never crosses it; a segment that is exactly ** matches one or more whole segments (so acme::** does not match the bare acme); every other character is literal, regex metacharacters included; an absent or empty include means everything; multiple include patterns are a union and exclude is applied after it; and matching is case-sensitive. These are exactly the rules four independent implementations would otherwise each get slightly wrong — the failure mode that produced the cross-port LIKE/ILIKE divergence fixed in 0.21.6.

TypeScript is the only port with a runner today. The reference implementation is server/typescript/packages/sdk/src/scope.ts (compilePattern / compileScope / matchesScope), and the corpus was authored against it. Java, Kotlin, C# and Python have no runner yet; when each gains one, this corpus is what it implements against — it exists now precisely so those four land on one grammar rather than four.

Orphaned fixtures (tested but not yet documented)

The fixtures in the eight corpora mapped above (metamodel 255 + yaml 15 + verify 31

  • render 15 + persistence 33 + api-contract 41 + source-resolution 25 + scope 10) each map to a feature doc. None are orphaned today. The remaining corpora in the totals table gate tooling contracts (registry manifests, provider composition, agent context, docs emit) rather than user-facing metamodel behaviour, so they have no feature-doc row.

If you add a new fixture and don't see a clear home for it, either:

  1. Add a section to the closest matching feature doc and reference it from its ## Verified by block, or
  2. Open an issue if the fixture exercises behavior the feature docs haven't yet described.

Orphaned docs (documented but no fixture coverage)

None of the 8 feature docs lack fixture coverage. The closest case is the high-level overview material in each doc (e.g., the "Anatomy of an entity" explainer in entities.md) — these are pedagogical, not behavioral, and don't require pinning.

See also