Repository navigation
fix(driver-memory): $contains on a multi-valued or JSON-stored field is membership, on every face - #20984
Conversation
…ership, on every face A multi-valued or JSON-stored field now answers `$contains` / `$notContains` by whole-element membership (the SQL family's reading), on the query path's two spellings and on the analytics face's `$match` and its SQLite echo. A scalar column keeps the case-exact substring test. The fork is the field's declared storage shape, read from the declaration `syncSchema` recorded. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…ory face; one seam; docblock and sweep - memory-20874-contains-membership.test.ts: driver-sql's #17590 fixture row for row, plus the stored shapes SQLite decides (scalar, object, nested array, null/boolean/fractional members), through find() in both spellings, count(), the analytics query and its SQL echo executed on sql.js, and the nested-relation lowering's driver input. - driver-sql #17590 suite: a multi-valued lookup column and the u1/u10 case. - driver-memory: one public seam, filterContainsTest; the analytics echo reads its member set off the test the $match exit runs. - spec: the $contains docblock's stale driver-memory status and dead tracker link are replaced with the measured status. - the rest suite header and the pending #20802 changeset no longer state the in-memory gap. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift CheckThis PR changes 2 package(s): 4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 4 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 138 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin d5b60fc179b36ee6d41982a8c9bd69b30dc48485 && git checkout d5b60fc179b36ee6d41982a8c9bd69b30dc48485
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 05be3525961a4977ea49eda507e44a4482f87681 1bf8dbb609ca103f2e7b7cd062dfad72a0e2ba17 && git checkout -B drift-repro 05be3525961a4977ea49eda507e44a4482f87681 && git merge --no-ff 1bf8dbb609ca103f2e7b7cd062dfad72a0e2ba17
node scripts/docs-audit/affected-docs.mjs --json 05be3525961a4977ea49eda507e44a4482f87681
|
…ing) filterContainsTest is a new public method on the exported InMemoryDriver class, so the published surface widens; the entry stays minor. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: Inputs: card #20874 (body and all five comments: triage 5914978120, claim 5921357350, report 5922095993, claim amendment 5922144972, patch-round report 5922235042), PR #20984 (body, its 8-file list, the net diff of ① Derived judgments
② Semver level
Clause-②: yes (widening) — one new public method ③ Boundary flags
Implemented-by: VERDICT: PASS |
|
…alued field by membership, as its where twin does (objectstack-ai#21004) Fixes objectstack-ai#20873 Clause-②: no ## What changes The per-aggregation `filter` (`engine.aggregate({ aggregations: [{ …, filter }] })`, and so `POST /api/v1/data/:object/query`) is evaluated by the engine's own walker, `matchesAggregationFilter` in `packages/objectql/src/having-filter.ts`. Its `$contains` arm failed every value that was not a string, so a stored array never matched, and its `$notContains` arm passed every such value, members included. On a DECLARED JSON-stored field both arms now ask MEMBERSHIP, the reading `FILTER_OPERATORS`' `$contains` docblock (`@objectstack/spec`) declares and `where` already gives on every SQL dialect (`SqlDriver.applyJsonMembership`): - `$contains: v` holds when `v` names an element of the stored array. A member stored as a JSON number or boolean is named by its text (`'1'` names `1`, `'1.50'` names `1.5`, `'true'` names `true`, `'null'` names `null`). That is the candidate set `driver-sql`'s `jsonMembershipCandidates` binds on every dialect. Array-only, as the SQL constructs are. - `$notContains: v` is its exact complement, and a row with no value still satisfies it (objectstack-ai#5298), as `col IS NULL OR NOT (…)` does in SQL. - A scalar text column keeps the substring test, unchanged. So does `having`. Files: `having-filter.ts` (the two arms, `storedArrayHasMember`, `declaredJsonStoredFields`, and an optional `jsonStored` set threaded through `matchesHaving` / `matchesAggregationFilter`), plus `in-memory-aggregation.ts`. That file is the one place the engine hands the object's declared field map to the per-aggregation filter, so it reads the declared set once per call beside `declaredFieldClasses`. `having-filter.ts` is not on objectql's published entry points. `in-memory-aggregation.ts` is (`applyInMemoryAggregation` and `bucketDateValue`, from `index.ts` and `core.ts`), and neither exported signature changes; the declared set is threaded through the internal `aggregateBucket` only. ### The card's table, through the REST door, before and after Measured with a real `SqlDriver` on SQLite and on a live PostgreSQL 16.14. Rows: `d1 ['u1','u2']`, `d2 ['u2']`, `d3 ['u3','u1']`, `d4 []`, `d5 ['u10']`, `d6 null`. Base `212d613c`, head `90ba78d9`. | query | `where` twin | per-aggregation `m`, base | `m`, head | |:--|:--|:--|:--| | `owners $contains 'u1'` (the card) | 2 | 0 | 2 | | `owners $contains 'u10'` | 1 | 0 | 1 | | `owners $notContains 'u1'` | 4 | 6 | 4 | | `tags $contains 'red'` (`d3` holds `['redwood']`) | 2 | 0 | 2 | | `tags $notContains 'red'` | 4 | 6 | 4 | | `$or` of `$contains` u1 / u3 (the any-of spelling objectstack-ai#7398's refusal prescribes) | 2 | 0 | 2 | | `$not` over `owners $contains 'u1'` | 4 | 6 | 4 | | `title $contains 'u1'` (text, the control) | 3 | 3 | 3 | On the in-memory driver the per-aggregation `m` is the same evaluator's answer, also 2 now. Memory's own `where` answers 3 for the card at this base (`d5` too, by a per-element substring). That face belongs to objectstack-ai#20874 (in flight), whose branch (`10656601`) moves it to membership and pins `d1, d3`. ## The fork: by the DECLARED column (Zone 2 H2) The fork reads the declaration (`STRUCTURED_JSON_TYPES` or `isMultiValueField`), never the row. That is the contract's sentence: "One operator, two questions, selected by the COLUMN rather than by the caller". It is also `SqlDriver.isJsonColumn`'s population (built from the same two spec sets) and objectstack-ai#20874's `isJsonStoredField`, character for character. Measured: on every fixture reachable through the public doors, the declared reading and a value-shape reading select the same rows. - `$contains` / `$notContains` on a declared structured-JSON field is refused before any row is read: the engine's text-operator declared-type door, `INVALID_FILTER` 400, in `where` and in the per-aggregation filter alike, on all three backends. - A multi-valued field's `find()` value is an array or `null` on memory, SQLite and PostgreSQL alike. The write door wraps a scalar: `'u1'` is stored as `['u1']` on memory and SQLite. The two readings differ only on rows a direct caller hands the walker: a declared multi-valued column holding a scalar string, or an undeclared column holding an array. There the declared reading gives what SQL `where` gives (no member; the substring reading), and a value-shape reading would not. Both cases are pinned. No `open_questions` fork results. ## Zone 2 hypotheses, measured - **H1 — confirmed.** The arms were as hypothesised; the card's table reproduced on all three backends (`m: 0`). - **H2 — declared column**, above. - **H3 — `$in` / `$nin` left as they are.** - `where: { owners: { $in: ['u1','u9'] } }` is refused `INVALID_FILTER` 400 on SQLite and PostgreSQL (the objectstack-ai#7398 JSON-column gate). Memory's `where` answers `d1, d3`. - The drivers disagree and SQL refuses, so no membership semantics are invented here. - The per-aggregation answer stays `m: 0` for `$in` and `m: 6` for `$nin`, a 200 where `where` is a 400. That is reported as an out-of-scope finding, not pinned. - **H4 — `having`.** - A `groupBy` on a multi-valued field is refused 400 on all three backends, and on a structured-JSON field too. - The one aggregated column that can still hold a stored array is a `min` / `max` over a multi-valued field. That is answered three ways: an array on memory, the serialized TEXT on SQLite's native aggregate, `DATABASE_ERROR` 500 on PostgreSQL. So there is no single `where` answer to hold `having` to, and `having` is not handed the declared set. - What is pinned: `having` `$contains` on a `groupBy` text projection keeps substring, on SQLite and PostgreSQL at the REST door and on the engine level. - **H5 — confirmed, so the mirror arm moved under os-dev rule 3's bounded in-place exemption.** Base per-aggregation `$notContains 'u1'` counted 6 where `where` counts 4. - All four conditions hold. Same defect class, same arm pair. The shape is pinned by `applyJsonMembership`'s complement. No other claim holds `having-filter.ts`: objectstack-ai#20822 group 3 is unclaimed, and objectstack-ai#20981 is filed bare. Same gate family. - The NULL row `d6` is counted, as SQL's `col IS NULL OR NOT (…)` counts it. - The claim's file surface does not name this arm. PM: please amend it, together with `in-memory-aggregation.ts` and the two test files. - **H6 — no importable predicate.** `driver-sql`'s `jsonMembershipCandidates` and `driver-memory`'s `containsMemberCandidates` (landed by PR objectstack-ai#20984 after this branch's merge base) are both module-private in driver packages, which objectql does not depend on. So `storedArrayHasMember` is the third copy of the rule on `main`; see Acceptance notes. ## Compile-surface conclusions | # | face | conclusion | |:--|:--|:--| | 1 | `driver-sql` `applyFilterCondition` | **already compliant (evidence)** — `$contains` / `$notContains` on a JSON column go through `applyJsonMembership`; the `where` twin numbers in the table above are this face, measured on SQLite and PostgreSQL 16.14. `driver-sqlite-wasm` and `driver-turso` local inherit it (not measured separately). | | 2 | turso `RemoteTransport.buildWhereSQL` | **out of scope (reason)** — an independent compiler this card does not touch. Read at `212d613c`: its `$contains` / `$notContains` arms go `pushLike` (substring over the stored text) with no JSON-column fork. Not measured (no remote libsql here). In the out-of-scope finding below. | | 3 | service-analytics `compileScopedFilterToSql` | **out of scope (reason)** — a different compiler. Measured function-level at `212d613c`: `{ owners: { $contains: 'u1' } }` compiles to `instr("t"."owners", ?) > 0` on SQLite, which admits a row holding `["u10"]`. On PostgreSQL it compiles to `"t"."owners" LIKE ? ESCAPE ?` over a json column. In the out-of-scope finding below (an RLS read scope). | | 4 | service-analytics `lowerAnalyticsWhere` | **out of scope (reason)** — it lowers the analytics `where` to a `FilterCondition` and adds no `$contains` reading of its own. The ObjectQL strategy hands that to the driver (face 1). The native SQL strategy maps `contains` to the substring LIKE shape (`native-sql-strategy.ts`, read, not measured), in the same finding as face 3. | | 5 | `formula` `matchesFilterCondition` | **out of scope (reason: fenced; PR objectstack-ai#20972 landed on this file during this run)** — its arm is `typeof actual === 'string' && typeof v === 'string' && actual.includes(v)` (unchanged by objectstack-ai#20972). Measured: `['u1','u2']` → false, `['u10']` → false, `'u1 memo'` → true. In the out-of-scope finding below. | | half | objectql `having-filter` | **changed** — the per-aggregation filter, as above; `applyHaving` / `matchesHaving` without a declared set unchanged (H4). | | unfrozen | `driver-memory` / `driver-mongodb` | **out of scope (reason: fenced, objectstack-ai#20874 / objectstack-ai#20897 in flight).** Memory measured above, and objectstack-ai#20874's fork matches this one. Mongo's `translateFieldOperators` compiles `$contains` to a bare `$regex`, which MongoDB applies per array element (per-element substring). Read, not measured. | ## Tests - `pnpm --filter @objectstack/objectql exec vitest run --project local --maxWorkers=2 src/engine-aggregate-filter-array-membership.test.ts` — 30 passed. The card's rows with the rows themselves, empty table, per group, `having` control, the member-text reading (number / exponent / boolean / null / non-JSON-number spellings / nested / object / scalar), the declared fork, the declared population. - `OS_TEST_POSTGRES_URL=… pnpm --filter @objectstack/rest exec vitest run --project local --maxWorkers=2 src/aggregation-filter-array-membership.test.ts` — 18 passed (9 SQLite, 9 live PostgreSQL 16.14), 9 named skips (MySQL). Each row runs beside its live `where` twin, populated and empty. - objectql whole `local` project on the merged head `90ba78d9`: 349 files, 6851 tests passed; `repo` project 1 file / 5 passed. - REST aggregation-adjacent files on `90ba78d9` with the PostgreSQL cell live: 9 files, 106 passed, 34 skipped. - `pnpm --filter @objectstack/objectql typecheck` and `pnpm --filter @objectstack/rest typecheck`: green. Both new test files are in their package's test program (`tsc -p tsconfig.test.json --listFiles`). ### Reverse verification Each leg ran through `scripts/ablation-replace.mjs` (anchor hits proven on disk; restore proven blob == HEAD and `git diff HEAD` empty), from the committed change. - **A1** — the `$contains` arm put back to the substring test: 15 of 30 red (every membership `$contains` row, the member-text rows, the declared-fork row); the `$notContains` rows and controls green, as predicted. - **A2** — the `$notContains` arm put back: 3 red (its two rows and the complement row). - **B** — the REST suite reads objectql through `dist/`. `declaredJsonStoredFields` was emptied, objectql rebuilt, and `ablation-dist-preflight.mjs` found the marker in 4 built files. 12 red: the 6 membership rows on each of SQLite and PostgreSQL. Text control, `having` and empty-table rows stayed green. Restore leg: rebuilt, marker absent from all 14 built files, tree clean, 18 passed. ### Driver conformance ledger `node scripts/check-driver-conformance.mjs`: before (`212d613c`) "50 covered cell(s), 0 in the DEBT ledger, 0 exempt"; after (`90ba78d9`) the same. ### Gates - `node scripts/pm/dispatch-gates.mjs --commands` re-derived with no paths at `90ba78d9` gives 63 commands, all run, exit codes recorded to disk. `--ran` reconciliation: 63 derived, 61 run (all exit 0), 2 NOT MEASURED, 0 unrun. - NOT MEASURED: `check:dual-build-cjs-loads` and `check:type-check-debt`. Both are PREREQUISITE NOT MET (exit 3): they need the whole-workspace build `lint.yml` performs first, and 42 / 5 packages have no `dist/` in this worktree. - `check-engine-split-ratio` first refused on the shallow checkout. It was green after a deepen to its window (`git fetch --shallow-since=2026-06-26 origin main`). - Lint, a declared narrowing over the four touched TS files at `90ba78d9`: `eslint --no-inline-config --format json` reports 4 files, 0 errors, 0 warnings. - Each file maps to a config (`--print-config`). - `eslint.config.mjs` never enables type-aware linting (no `parserOptions.project`, no typed rules; its own lines 327-328 say so), so this diff cannot move an untouched file's verdict. ## Acceptance notes - **The member rule now has three copies on `main`.** - `storedArrayHasMember` restates `driver-sql`'s `jsonMembershipCandidates` as a predicate. `driver-memory`'s `containsMemberCandidates` (PR objectstack-ai#20984) is the other JS copy. - None can be imported by the others. The one shared home would be `@objectstack/spec/data`, beside `asciiCaseInsensitiveContains` and `isEmptyFilterValue`, the value-level filter rules every JS face already reads from there. - Carrier: the convergence item recorded on objectstack-ai#20987. - **No shared conformance kit for stored-array membership.** - `FILTER_TEXT_CASES` has no array rows. The membership fixtures are literal per package: `sql-driver-17590-json-column-membership.test.ts`, objectstack-ai#20874's `memory-20874-contains-membership.test.ts`, and the two files here. These use the same `u1` / `u10` / `redwood` disagreement rows. - A spec `*_CASES` kit beside `FILTER_TEXT_CASES` would let all three faces be driven by one table. - Here the cross-face invariant is held by running each row beside its live `where` twin. - **The memory cell is engine-level.** It runs over the read shape `find()` presents (measured identical on all three backends). A real `InMemoryDriver` consumer would need a ruled entry in the `check:driver-memory-census` ledger. - **The PostgreSQL / MySQL REST cells are a named skip in CI.** No job provisions `OS_TEST_POSTGRES_URL` / `OS_TEST_MYSQL_URL` for `packages/rest`, the same note `data-group-by-json-door.test.ts` carries. The PostgreSQL cell's local run is above. - **Out-of-scope findings, for the seat to route** (full evidence in the report on the card): - per-aggregation `$in` / `$nin` on a multi-valued field answer 200 (`m: 0` / `m: 6`, the latter counting the rows it was asked to exclude) where `where` is a 400 on the SQL family, and memory's `where` answers membership; - the `$contains` membership contract is not answered on the turso remote transport, the service-analytics SQL compilers (an RLS read scope over-reaches), `driver-mongodb` and `formula`; - `min` / `max` over a multi-valued field: three answers (array / serialized text / PostgreSQL 500); - `$startsWith` / `$icontains` on a multi-valued field: PostgreSQL `where` 500, SQLite over the serialized text, memory per element. --- _Generated by [Claude Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
…lared JSON-stored field, in the SQL family's words (objectstack-ai#21066) (objectstack-ai#21159) Fixes objectstack-ai#21066 Clause-②: yes (narrowing) On a field the object declares JSON-stored (a `multiple: true` field, `tags` / `multiselect` / `checkboxes`, or a structured-JSON type such as `json`), `driver-memory` now refuses the scalar-comparison family that `driver-sql`'s `where` refuses: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$nin` and implicit equality, whatever the comparand, at any depth. The answer is `INVALID_FILTER` / 400 with the same message. The operator set and the sentence are read from `@objectstack/core` (`JSON_COLUMN_INCOMPATIBLE_OPERATORS`, `jsonColumnOperatorRefusalText`, homed by PR objectstack-ai#21097). There is no third copy. `$contains` / `$notContains` (membership), `$null`, `$exists` and `$empty` keep answering. ## What was wrong (H1, measured at `origin/main` `670680e93` through `engine.find`) A real `ObjectQL` over `InMemoryDriver`, objectstack-ai#21004's six rows (`owners` is a `multiple: true` lookup, `tags` is a `tags` field). Every row reproduces the card: | `where` | before | now | |:--|:--|:--| | `owners` `$eq 'u1'` | `d1`, `d3` (per element) | 400 `INVALID_FILTER` | | `owners` `$in ['u1','u9']` | `d1`, `d3` | 400 | | `owners` `$nin ['u1','u9']` | `d2`, `d4`, `d5`, `d6` | 400 | | `owners` `$gt 'u1'` | `d1`, `d2`, `d3`, `d5` | 400 | | `tags` `$gt 'red'` | `d3` | 400 | | also: bare `{ owners: 'u1' }`, `$ne`, `$gte`, `$lt`, `$lte`, `$between`, `tags $eq`, `{ owners: null }`, `$eq null`, `$ne null`, `$in []`, `$nin []` | rows, per element | 400 | | controls: `owners $contains 'u1'` / `$notContains` / `$null` / `$exists` / `$empty`, `title $in` | rows | unchanged rows | The engine hands the driver the operators as written, except `$ne` / `$nin`. Those arrive inside the spec's null-safe lowering (`$and` of `$or` of `$null: true` and the operator). The gate walks `$and` / `$or` / `$not`, so that shape is refused too. The analytics face (`MemoryAnalyticsService`) answered the same per-element rows. Its SQL echo rendered `owners = 'u1'`, which matches no row over the JSON text the SQL family stores. It now refuses in `query()` and `generateSql()` alike. ## What changed - `filter-refusal.ts`: the shape gate (`assertFilterConditionShape`) takes an optional `FilterFieldDeclarations` (`isJsonStoredField`, `reportWithheld`). It has two arms. Implicit equality on a declared JSON-stored field is refused as `=`, bare. Any operator in the shared set is refused AFTER the existing comparand-shape rules, which is `driver-sql`'s order (comparand gate, then column-type gate). So an array under `$eq` or a one-element `$between` still gets its own refusal first. `jsonStoredFieldOperatorError` builds the error from the shared text: this package's `unsupportedFilterError` envelope, with the withheld diagnostic handed to `reportWithheld` (prefixed `At PATH:`) before the throw. - `memory-driver.ts`: `convertToMongoQuery` passes `this.filterFieldDeclarations(object)`. The population is `isJsonStoredField`, the predicate `$contains` already forks on (`STRUCTURED_JSON_TYPES` or `isMultiValueField`). So the fields where `$contains` asks membership are exactly the fields where the family is refused. The diagnostic goes to the driver's logger at `warn`, the level `driver-sql` uses for its withheld filter diagnostics. That keeps the message's "the full diagnostic is in the server log" true here. - `memory-analytics.ts`: `normalizeFilters` takes the cube. It judges a `where` key (a cube member) by the field it maps to on the cube's table, the same (table, field path) pair `filterContainsTest` reads. Its diagnostic goes to the analytics service's own logger. - `.changeset/21066-memory-json-column-family-refusal.md`: `@objectstack/driver-memory` `minor`, BREAKING banner, `Clause-②: yes (narrowing)`, one ADR-0087 marker `not-required (no-migration-prescription)`. No registered id covers a filter operator on a JSON-stored column. The one migration-registry entry that mentions json columns (`cel-predicate-one-value-comparand-refused`) is the CEL list-comparand surface, not this one. ## Hypotheses, measured - **H1** holds: the table above. - **H2.** The shared home is `@objectstack/core`'s `json-column-operator-refusal.ts`, and both names are read. Before this change `driver-memory` had NO withheld-diagnostic seam: every refusal it raises (the `$null` / `$exists` non-boolean refusals included) names the field in the message, and nothing in the package logged a diagnostic. This change keeps the shared posture: the message names neither field nor operator, and the diagnostic goes to the server log. - **H3.** `driver-sql` decides a JSON column from `jsonFields`, filled from `JSON_COLUMN_TYPES.has(type) or isMultiValueField(field)`. `JSON_COLUMN_TYPES` is `STRUCTURED_JSON_TYPES` plus `MULTI_OPTION_TYPES` plus the driver-internal `object` / `array` aliases. Memory's population is the same predicate less those aliases and less a single-value media field on an unmoved deployment (both recorded on `isJsonStoredField`). On a schemaless direct call (an object never passed through `syncSchema`), nothing is judged. Every operator answers per element as before, as `SqlDriver.isJsonColumn` answers `false` for a table it was never told about. Pinned. A field declared SCALAR (`text`) that holds an array is not judged either. - **H4.** `@objectstack/formula`'s `ORDERING_OPERATORS` docblock does NOT declare a per-element reading for the query plane. It records a non-alignment ("driver-memory's read, a frozen test driver, compares a stored list element by element and keeps returning those rows ... declared on objectstack-ai#15104"). objectstack-ai#15104 is the `$field` cross-field reference card, shut as `not_planned` under the driver-memory investment freeze. It rules nothing about the equality or ordering family on a stored list. So this is a formula-plane record of observed behaviour, not a query-plane contract, and no contract conflict stops the card. That docblock sentence goes stale on declared fields once this lands (see Acceptance notes). - **H5.** objectstack-ai#21009 widens the same shared set to the text operators. Both gates here read the set live, and the new suite iterates `JSON_COLUMN_INCOMPATIBLE_OPERATORS` intersected with this driver's vocabulary, with a floor of the nine `$`-spellings. So once both land, memory refuses `$startsWith` / `$endsWith` / `$icontains` on these fields with no edit here, and the suite pins them. Whichever of the two lands second merges `main` and checks the other's members on its face. The suite's `$contains` control is outside objectstack-ai#21009's scope. ## Pin sweep - The ONE per-element pin the package carried on a declared field flipped: `memory-20444-empty-operator.test.ts` had `{ tags: { $empty: true, $ne: null } }` giving `r2`. It is now a refusal pin (`code` + `status` + the shared message). The composition (`$empty` beside a has-a-value sibling on one multi-value field) is kept through `$null: false`, which answers `r2`. `driver-sql`/SQLite answers that row too, and refuses the `$ne: null` spelling with the same body (measured on the built driver). - `memory-matcher-scalar-comparand-array-value.test.ts` pins per-element answers on a column declared `text`. Those cells still hold, and a header note now says the population there is a scalar-declared column. - Repo-wide: only two tests outside this package bind the real driver (`packages/runtime`'s two ruled consumers). Neither filters a JSON-stored field. No other `INVALID_FILTER` pin moves. ## Tests (final head `13407b76f`) - `pnpm --filter @objectstack/driver-memory exec vitest run --maxWorkers=2`: **70 files, 1703 passed**. The first run after the implementation, before any test edit: 69 files, 1 red of 1613, the per-element pin flipped above. - `pnpm --filter @objectstack/driver-memory run typecheck`: exit 0. `tsc --listFiles` includes both edited test files. - New `memory-21066-json-column-family-refusal.test.ts` (89 tests): - the card's five rows; - every family member on `owners` / `tags` / `meta` (`json`); - ten shapes per field (bare, bare null, `$eq null`, `$ne null`, the engine's `$ne` lowering, `$in []`, `$nin []`, under `$not`, an `$or` branch after a holding one, `$eq` beside `$contains`); - `count` / `findOne` / `updateMany` / `deleteMany` refusing with the table untouched; - the withheld message plus the logged `At filter.$or[1].owners.$gte:` diagnostic; - comparand-first ordering; - eleven answered controls; - the declaration boundary (undeclared object, scalar-declared column, the gate with and without declarations); - the analytics face, `query()` and `generateSql()`, including the `cube.member` spelling, plus its log line and a `$contains` control. - Ablations (`node scripts/ablation-replace.mjs`, wrap mode, run at `de1fef341`; the two later merges touched no file in this package; each restore proven blob == HEAD with `git diff HEAD` empty). The subjects are this package's `src`, imported relatively, so no `dist` is involved: - **A** `filterFieldDeclarations`' predicate forced to `() => false`: **73 red / 37 green** of 110 across the new file and 20444. The 17 green in the new file are exactly the answered controls, the declaration-boundary trio, the premise, the comparand-first case and the analytics `$contains` control. - **B** the implicit-equality arm disabled: **7 red**, exactly the six bare cases and the analytics bare case. - **C** the operator arm disabled: **67 red**, every operator-based refusal pin, the direct-gate test and the 20444 flip. - Gate union, derived with `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` (no paths; 7 changed paths, working tree clean) at `13407b76f`: **60 derived, 60 run, every one exit 0**. Reconciled with `--ran`: "60 derived famil(ies) accounted for — 60 run, 0 NOT-MEASURED (a DERIVED zero — all 60 recorded an exit code and none of them is 3)". The same 60 also ran all-zero at the previous merge head `5b75fe461`. At `de1fef341`, `check:dual-build-cjs-loads` exited 3 (PREREQUISITE NOT MET, no dist yet); it measured on both later heads. - Driver conformance ledger (`node scripts/check-driver-conformance.mjs`), before and after: byte-identical. 50 covered cells, 0 DEBT, 0 exempt. The shared matrix has no JSON-column or multi-value case-set, so this invariant is held by the per-package pins, not the matrix. ## Acceptance notes - **The class gains one method.** `InMemoryDriver.filterFieldDeclarations` is tagged `@internal`. It is not private only because the analytics face is another class. `FilterFieldDeclarations` is not exported from the package root, but the method does appear in the published `.d.ts`. objectstack-ai#20984 graded the analogous public `filterContainsTest` as a surface widening (`Clause-②: yes (widening)`). The seat graded it so (5929927010): the line is `yes (narrowing)`, with the semver (`minor`) and the ADR-0087 marker unchanged. - **Surface beyond the claim's list.** `memory-driver.ts` and `memory-analytics.ts` are edited. The gate cannot see a declaration on its own, so the plumbing is the minimum the direction needs, and the analytics face calls the same gate. No open PR touched either file when read before the first edit. - **The AST comparison-node door** (`{ type: 'comparison', field: 'owners', operator: '=', value: 'u1' }`) still answers per element on a declared field: `d1`, `d3`, measured on the built driver. No seam emits that form (the engine and the protocol hand a driver a FilterCondition), so it is reachable only by a direct driver call. Left alone. - **The shared sentence's mechanism clause** ("a field this driver stores as a JSON TEXT column", "$in/$eq matched nothing") is `driver-sql`'s, and is literally untrue of this driver and of the engine's per-aggregation face. The prescription (`$contains`, an `$or` of `$contains`) is right on all three. Inherited as objectstack-ai#21007 shipped it. objectstack-ai#21009 is the PR that next edits the shared home. - `@objectstack/formula`'s `ORDERING_OPERATORS` docblock ("driver-memory's read ... keeps returning those rows") is now true only of undeclared objects. It is a comment, and no claim holds that file. - **Tooling.** With the `turbo` 2.10.10 to 2.11.5 bump now on `main`, every repo-scoped turbo run in an agent session appends a managed "turborepo-agent-rules" block (an HTML-comment-delimited section) to `AGENTS.md`. These include `pnpm exec turbo run build`, `pnpm check:type-check-debt`, `check:query-options-erasure` and `check:slot-lookup`. It happened repeatedly in this worktree and was restored each time, and every gate derivation above was taken on a clean tree; this PR does not touch `AGENTS.md`. Tracked as objectstack-ai#21146 (PR objectstack-ai#21151). --- _Generated by [Claude Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Fixes #20874
Clause-②: yes (widening)
What changes
driver-memorynow answers$containsand$notContainson a declared JSON-stored field by whole-element membership, the readingdriver-sqlcompiles on all three dialects. A field counts as JSON-stored when it ismultiple: true, amultiselect/checkboxes/tagsfield, or aSTRUCTURED_JSON_TYPESmember such asjson. A scalar text column keeps the case-exact substring test. The change covers every face of the package:$-spelling (normalizeFieldOperators) and its AST spelling (convertConditionToMongo):find,count, and every verb that goes throughconvertToMongoQuery;$match;generateSql), which now renders SQLite'sjson_eachmembership construct for such a column, so the echoed statement still returns the rows the chart was drawn from.One rule backs all of them:
InMemoryDriver.filterContainsTest(object, field, value). On a JSON-stored field it returns{ $elemMatch: { $in: members, $not: { $type: 'array' } } }, and on any other field{ $regex: filterSubstringPattern(value) }.$notContainsis$notover that test, on both faces. The members come from the comparand's text, read the waydriver-sql'sjsonMembershipCandidatesreads it:'1'names the string or the number 1,'1.50'names 1.5, and'true'/'false'/'null'name the string or the JSON literal.The
$containsdocblock inpackages/spec/src/data/filter.zod.tslost its stale "driver-memory DOES NOT ANSWER IT YET" bullet. Its dead tracker link went with it (that card answers 404). The replacement text states the measured status, and states that the other text operators over a stored array are not ruled by that section.Measured on
origin/mainf6ccca4abefore the change, and after (HEAD10656601)Fixture:
driver-sql's #17590 fixture plus a multi-valued lookupowners(['u1','u2'],['u10'],['u3','u1'],[]).wheredriver-sql){ owners: { $contains: 'u1' } }{ owners: { $notContains: 'u1' } }{ tags_: { $contains: 'red' } }{ nums: { $contains: '1' } }(members are numbers){ label: { $contains: 'red' } }(scalar control)engine.findon a realInMemoryDriverwith the nested-relation filter{ owners: { region: 'NA' } }(owneru1NA,u10EU; this is PR #20872's lowering):d1, d3, d5before andd1, d3after. Its$notgaved2, d4before andd2, d4, d5after, which matches the rest suite's SQLite rows. That was a one-off run (a throwaway probe file, not committed): this package may not import the engine and the engine's packages may not import this driver (check:driver-memory-census). What is pinned instead is the driver input the engine sends, an$orof one$containsper related id, which objectql'sengine-nested-relation-lowering.test.tsasserts.Mechanism hypotheses (order Zone 2)
H1 confirmed. The
$containsarm lowered to an escaped$regex, and mingo applies a$regexto each element of an array value. Reproduced on the fixture above before any edit.H2: the fork is by the DECLARED field, not the row's runtime shape. Memory has the declaration (
valueShapes, recorded bysyncSchemasince the$emptywork), and SQL forks on its JSON-column registry, which is also filled from the declaration. The population is the spec's JSON-stored classes:STRUCTURED_JSON_TYPESplusisMultiValueField, the two halvesdriver-sql's registry is built from. The two readings DO give different rows on one fixture, so the choice goes to the PM in the report'sopen_questions: a declaredjsonfield holding the scalar string'u1'. SQLite answers no member, because its constructs are array-only. A per-row-shape fork would answer it by substring. The declared fork matches SQLite, and the contract text says the question is "selected by the COLUMN", declared metadata. Undeclared fields (an object never passed throughsyncSchema) keep the substring reading, asSqlDriver.isJsonColumnanswersfalsefor a table it was never told about.H3 confirmed and pinned. Number members answered nothing;
'1','2','10','0'and'1.50'now give SQLite's rows.H4 confirmed:
$notContainsdiverged. It is the mirror arm of the same defect, changed under os-dev rule 3's bounded in-place exemption. All four conditions hold:col IS NULL OR NOT (…):$notover the test admits null and missing rows, pinned on both fixtures.memory-driver.tsis held by no other claim; the sibling$existscard is kept off it by its own claim.The claim's surface should be amended to include it.
H5 confirmed. The cube face borrowed
filterSubstringPatternand wrapped it in its own$regex, so it had the same defect. It now takes the driver's whole test. The echo reads its member set off that same test, so the chart and its echo cannot name different sets.H6 confirmed. See the
engine.findrun above.Compile-surface conclusions
driver-sqlapplyFilterCondition(driver-sqlite-wasm,driver-tursolocal inherit it)applyJsonMembershipemits membership on JSON columns. Evidence: the #17590 suite now carries a multi-valued lookup column and theu1/u10case, green on SQLite here. Its live PostgreSQL/MySQL cells are unprovisioned locally; the required live job runs the whole driver-sql suite. The two inheriting drivers were not separately run.RemoteTransport.buildWhereSQL$containsarm emitspushLike(a GLOB substring) on every column, JSON columns included, so remotelyu1would match["u10"]while the local transport answers membership. Not measured. Reported as a finding.read-scope-sqlcompileScopedFilterToSql{ owners: { $contains: 'u1' } }on a declaredlookup+multiple: truefield compiles toinstr("t"."owners", ?) > 0on SQLite andLIKE '%u1%'on PostgreSQL/MySQL, a substring over the stored JSON text. This is an RLS read-scope face, so it is reported as a security-relevant finding.filter-normalizerlowerAnalyticsWhere$containsto the cubecontainsoperator, which the native SQL strategy renders asLIKE, a substring. Reported.formulamatchesFilterConditiontypeof actual === 'string' && actual.includes(v), so a stored array never matches (fail-closed). Reported.having-filterdriver-memoryquery path and cube face (memory-analytics.ts), behindfilter-refusal.tsfilter-refusal.tsand the$existsarm are untouched (sibling card).driver-mongodbtranslateFieldOperators$containsto a native$regex, which MongoDB applies per array element: the same defect. Not measured. Reported.Why the shared pin is a mirrored literal table, not
FILTER_TEXT_CASES(order Zone 3, not taken)FILTER_TEXT_CASEShas no array column, so adding one changes the fixture of all five enrolled drivers.driver-mongodbimports every row of it and still carries the per-element defect, so its suite would go red. The table's own rule 2 says rows join a driver's suite in the PR that ends that driver's gap. Doing it here would also widen the spec touch beyond the one declared docblock. A new sibling case-set would add DEBT rows to a ledger that only goes down. So the new memory file mirrorsdriver-sql's #17590 fixture row for row and asserts the same literal row sets. That is how the #17590 file already holds its three dialect cells to one answer.Files
packages/drivers/driver-memory/src/memory-driver.ts: the population (isJsonStoredField), the members, the one test (filterContainsTest), both query-path spellings.packages/drivers/driver-memory/src/memory-analytics.ts: the$matchrows take the driver's test; the echo renders membership (sqliteMembershipPredicate).packages/drivers/driver-memory/src/memory-20874-contains-membership.test.ts(new): two fixtures. The first is the driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 fixture plusowners. The second holds the stored shapes SQLite decides: a scalar, an object, a nested array,[null],[[null]],[true, 1.5],['']and a NULL row, each answer measured ondriver-sql/SQLite first. Every case runs onfindin both spellings, oncount, and on the cube's$match. The cube's echo is EXECUTED on sql.js over the same rows. The file also covers the nested-relation driver input and the fork.packages/drivers/driver-sql/src/sql-driver-17590-json-column-membership.test.ts: anownersmulti-valued lookup column and theu1/u10case (the claim's SQLite pin).packages/spec/src/data/filter.zod.ts: the one declared docblock..changeset/20874-memory-contains-membership.md:minor, not thepatchthe order suggested.filterContainsTestis a new public method on the exportedInMemoryDriver. It ships indist/index.d.ts(the existingfilterSubstringPatternappears there as the positive control), and the level ruling quoted inpr-automation.yml(WHICH LEVEL) grades an additive widening of a published package's public surface at leastminor.Clause-②: yes (widening): the new public method widens the published surface; no filter key or operator is added.packages/rest/src/data-nested-object-door.test.ts(a header comment that cited the gap and the removed docblock clause);.changeset/20802-nested-relation-filter-served.md, one sentence of a pending release note that said the in-memory driver still matches per element.Confirmation requested: a pending release note is corrected
This PR changes
.changeset/20802-nested-relation-filter-served.md, a pending release note this PR did not add. This is the DELIBERATE CORRECTION classcheck-empty-changeset.mjsnames, soCheck Changesetstays red on purpose. It is not a required context.skip-changesetmust not be applied. The note's last sentence said:This PR makes that false, so it now reads:
Please confirm the correction on this PR. If the release that consumes that note ships before this PR lands, the edit should be dropped on rebase.
Tests (HEAD
10656601)pnpm --filter @objectstack/driver-memory typecheckpasses, andvitest rungives 68 files / 1553 tests passed. Both typecheck programs include the new test file (--listFiles).pnpm --filter @objectstack/driver-sql typecheckpasses (its program includes the edited test).sql-driver-17590-json-column-membership.test.tsgives 22 passed, 2 skipped; the skips are the live PostgreSQL/MySQL cells, unprovisioned here, NOT MEASURED locally.pnpm --filter @objectstack/spec typecheckpasses.spec buildand thencheck:generatedreport "All 15 generated artifacts are up to date".Reverse verification. The code change was committed first. Then
memory-driver.tsandmemory-analytics.tswere checked out at BASE under an EXIT/INT/TERM restore trap. The landed mutation was verified on disk before the run:filterContainsTestcount 0, twoescapeRegex(val)arms back, and both files byte-equal to BASE.The predicted direction was that the cases where per-element substring and membership disagree go red and the agreeing ones stay green. Measured: 69 failed / 44 passed. The failures were every disagreeing case, on all four faces, plus the executed echo, the relation-lowering input and the fork pin. The
labelcontrols and theredwood,ab,u10and'0'cases stayed green. The restore was proven by blob hashes equal to HEAD (d376dadb…/9de9198b…), an emptygit diff HEADand clean porcelain.Guard ablation (
scripts/ablation-replace.mjs, anchor 1 to 0). Dropping$not: { $type: 'array' }turned exactly the nested-array cases red: 12, the json'u1', its complement and'null', on all four faces. The executed echo stayed green, since SQLite is the oracle there. The file was restored to the HEAD blob with an emptygit diff HEAD.Lint (narrowed, a measurement).
eslint.config.mjslints**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}minusNEVER_LINTED. The six changed TS files are all in it; the two changesets are not.eslint --no-inline-config --format jsonover those six files reports 6 files, 0 errors, 0 warnings at10656601.parserOptions.project, no typed rules; it says so itself), so this diff cannot move any untouched file's verdict.Driver conformance ledger:
node scripts/check-driver-conformance.mjsreads "50 covered cell(s), 0 in the DEBT ledger, 0 exempt" both atf6ccca4a(before) and at10656601(after).Gates:
dispatch-gates --commandswas re-derived at10656601. That gives 84 commands, six more than the dispatch list:engine-double-contract,objectql-double-limit,query-options-erasure,type-check-coverage,type-check-debtandwhere-matcher. The run also covered the eight roster gates flagged as sharing a directory with these paths. All exit 0 except:check-empty-changeset: exit 1, the deliberate correction above.check:dual-build-cjs-loadsandcheck:type-check-debt: exit 3,PREREQUISITE NOT MET(they need the whole workspace built). NOT MEASURED; CI's Build/Lint jobs own those.check:lean-entry-closure: exit 3 at first. It passed after building objectql.Acceptance notes (observed, not filed by this PR)
{ owners: { $startsWith: 'u1' } }givesd1, d3, d5on memory (per element) and no rows on SQLite (GLOB over the serialized text). The spec docblock now says so.engine.ts's delete-probe docblock (the$containspushdown) still says every backend answers a substring SUPERSET. That is stale fordriver-sqlsince driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 and for memory after this PR. The code stays correct because it narrows exactly throughstoredReferenceIncludes. Comment only, no carrier.driver-memoryholds no declaration for keeps the substring reading, and mingo still applies that$regexper element of an array value. This is the deliberate counterpart ofdriver-sql'sisJsonColumnreturning false for an unknown table.Generated by Claude Code