Repository navigation
fix(objectql): a per-aggregation filter counts $contains on a multi-valued field by membership, as its where twin does - #21004
Conversation
…ed multi-valued field by membership The engine's aggregation evaluator failed `$contains` on every value that was not a string, so a stored array never matched and `$notContains` matched every row. On a declared JSON-stored field (STRUCTURED_JSON_TYPES or isMultiValueField) both now ask membership, the reading `where` gives on every SQL dialect; a scalar string column keeps the substring test. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…lti-valued field beside its where twin Engine-level over the read shape find() presents on memory, SQLite and PostgreSQL; REST door on a real SqlDriver, SQLite always and the live PostgreSQL / MySQL cells where their URL is set, each row beside its where twin. having keeps the substring reading on a text projection. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…ship Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
…mbership Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift Check8 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run. What this run could not see
Coarse fallback — 17 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 9b6a0192b0271d3310efbabdeca28e2801694fd8 && git checkout 9b6a0192b0271d3310efbabdeca28e2801694fd8
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin a5bce408883b81a6e2382ebe709a03cc3a5b40b4 90ba78d9b150faf40e31f68841e9ac94aa1e13a1 && git checkout -B drift-repro a5bce408883b81a6e2382ebe709a03cc3a5b40b4 && git merge --no-ff 90ba78d9b150faf40e31f68841e9ac94aa1e13a1
node scripts/docs-audit/affected-docs.mjs --json a5bce408883b81a6e2382ebe709a03cc3a5b40b4 |
Contract reviewServed-tier: Inputs: card #20873 (body and all five comments: triage 5914962540, the blocked transition 5921472356, claim 5921990075, report 5922812043, claim amendment 5922857157); PR #21004 body, its five-file list, its four commits and the net diff against the merge base with ① Derived judgments
② Semver levelClause-②: no
③ Boundary flags
Implemented-by: VERDICT: PASS |
…on a declared JSON-stored field, in where's words (objectstack-ai#21097) Fixes objectstack-ai#21007 Clause-②: yes (widening) A per-aggregation `filter` now refuses a scalar comparison on a declared JSON-stored field (`$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$nin`, implicit equality) with `INVALID_FILTER` / 400, in the words `where` refuses the same filter in. It no longer counts rows the stored arrays cannot support. The operator set and the refusal text move from `driver-sql` to `@objectstack/core`, byte for byte, so both faces read one set and one sentence. **Clause-② has two halves.** It is `yes (widening)` because `@objectstack/core`'s root gains three exports (`JSON_COLUMN_INCOMPATIBLE_OPERATORS`, `jsonColumnOperatorRefusalText` and its return type `JsonColumnOperatorRefusalText`), and `applyInMemoryAggregation` gains an optional trailing `reportWithheld` parameter. It also narrows: `@objectstack/objectql` refuses queries it used to answer 200, at `engine.aggregate` and at the published `applyInMemoryAggregation` given a field map. The changeset therefore carries `minor` for objectql with a BREAKING banner and one ADR-0087 marker, `minor` for core, and `patch` for driver-sql, whose output is unchanged. The seat answer on the card (`## Seat answer — objectstack-ai#21007`, comment 5924546829) amended the claim to this surface and this `Clause-②`. ## What was wrong (measured before this change, `d1f8ce865`) `POST /api/v1/data/:object/query` on SQLite and a live PostgreSQL 16.14, over the card's six rows (`owners` is a `multiple: true` lookup, and `d1` and `d3` hold `u1`). Both dialects answered identically: | filter | `where` twin | per-aggregation `m` before | now | |:--|:--|:--|:--| | `owners $in ['u1','u9']` (the card) | 400 `INVALID_FILTER` | 0 | 400, same body | | `owners $nin ['u1','u9']` (the card) | 400 | 6, with `d1` and `d3` counted | 400, same body | | `owners $eq 'u1'` / `{ owners: 'u1' }` | 400 | 0 | 400 | | `owners $ne` / `$gt` / `$lte` / `$between` | 400 | 6 / 4 / 1 / 5 | 400 | | `tags $eq 'red'` | 400 | 1 (`['red']` loosely `==` `'red'`) | 400 | | `meta` (json) `$eq` / `$in` | 400 | 0 / 0 | 400 | | `owners $contains 'u1'` (the prescribed spelling) | 2 | 2 | 2 (unchanged) | | `title $in` / `$nin` / `$eq` (controls) | 2 / 4 / 1 | 2 / 4 / 1 | unchanged | ## What changed - **`@objectstack/core`:** a new `src/utils/json-column-operator-refusal.ts`, exported from the root beside `temporal-storage-form.js`. The `export *` publishes three names: `JSON_COLUMN_INCOMPATIBLE_OPERATORS` (driver-sql's 22 spellings, member for member), `jsonColumnOperatorRefusalText(field, op, bare)`, and its return type `JsonColumnOperatorRefusalText` (`{ message, diagnostic }`). These are the two strings `jsonColumnOperatorError` built, and nothing else. Each face keeps its own error constructor: driver-sql keeps its objectstack-ai#8220 provenance seam, and objectql keeps its ADR-0112 envelope. - **`driver-sql`** (`sql-driver.ts`): the module-private set and the two template strings are gone. `jsonColumnOperatorError` keeps its name and signature, and now calls the core builder (hunk at `:3298`). One import line (with its comment) sits at `:153`–`:156`, after the top import block. It is outside the declared `:3376`–`:3460` region on purpose, so as not to touch the `@objectstack/core` import block that objectstack-ai#20988 and objectstack-ai#20987 edit. `assertOperatorAppliesToColumn` (in objectstack-ai#20988's former region) is untouched: it reads the imported set under the same name. - **`objectql`** (`having-filter.ts`): `assertAggregationFilterIsEvaluable` gains `assertAggregationFilterSparesJsonStoredFields`. It runs once on the filter after the reference rule, against `declaredJsonStoredFields(declared.fields)`, and before any driver is asked for a row, so an empty table refuses too (objectstack-ai#20122's rule). It walks `$and` / `$or` / `$not` and refuses implicit equality (reported as `=`, bare, as driver-sql does) and every operator in the shared set, whatever the comparand (`null` and `[]` included). The withheld message is thrown, and the diagnostic, with the aggregation position, goes to `reportWithheld`, as objectstack-ai#20148 does. `$contains` / `$notContains` / `$exists` / `$null` / `$empty` keep answering. `checkCondition` carries the same refusal above its no-value exit, but only as a backstop for a row that reaches the arm. Its reach is the row's: an empty row set never gets there, and spec lowering rule 3 puts a `$null` arm ahead of every negation that the walker's `$or` short-circuit takes first. The gate is the complete door, and the docblocks now say so (round 2, from the review's ①.5). - **`objectql`** (`in-memory-aggregation.ts`, round 2, the review's F10 (b)): the published `applyInMemoryAggregation(rows, ast, timezone, fields, reportWithheld?)` now calls the same `assertAggregationFilterSparesJsonStoredFields` (exported from `having-filter.ts`, not from the package root) once per `aggregations[i].filter` when it is handed `fields`, before any row is judged. Before this, a direct caller reached only the backstop and got a row-dependent answer. This entry point holds no logger, so the closest seam is a new optional trailing `reportWithheld(diagnostic)`, which receives the field, operator and position; without it the diagnostic is dropped and the 400 is unchanged. `engine.aggregate` passes none, because it has already judged and logged the same filter. The gate's declaration parameter is narrowed to just the `fields` and `reportWithheld` members of `AggregationFilterDeclaration`, since `object` is the reference rule's. - **`objectql`** (`engine.ts`): one comment block and the `reportWithheld` log line at `assertAggregationFilterIsEvaluable`'s call site. The log line now reads "as it is for the same refusal in a where" instead of "…cross-field comparison…", since it carries two refusals now. ## Measured findings behind the shape (H1–H5) - **H1, the premise, holds.** `SET_MEMBER_DESCRIPTION`, the `$in` / `$nin` entries of `FILTER_OPERATORS` and the `$contains` docblock give no per-element reading. driver-sql's `where` refuses (it does not answer membership), so triage's "membership, as driver-sql does" misread it. - **H2, the set.** All ten operators, plus null and empty-list comparands, were answered with a wrong count; none was already refused. The bare infix spellings (`in`, `=`, `nin`) are refused earlier at both positions by the nested-relation door, so the evaluator only meets the `$` forms. - **H3, the home.** None existed; per the seat answer, the home is `@objectstack/core`. - **H4, where it fires.** The engine's in-memory lowering is the only evaluator of `aggregations[i].filter`: every driver's aggregate face refuses a per-aggregation filter 501, and the analytics ObjectQL strategy hands measure filters to `engine.aggregate`. - **H5, `having`.** After objectstack-ai#21037 (landed, merged here), `min` / `max` over a multi-valued field is refused `INVALID_FIELD` at the aggregate door. Measured on InMemoryDriver at the merged head: `max(owners)` with or without `having` gives 400 `INVALID_FIELD`. So `having` cannot meet a JSON-stored column, and it is left alone. ## Tests Round 1 numbers were read at `6e541b101`; round 2 numbers are marked with the head `f66bed950` (main merged). - `@objectstack/core` `json-column-operator-refusal.test.ts`: 6 passed. It pins the set member for member, and the message and three diagnostics by SHA-256 and length against what driver-sql printed at `8f784959c`. Hashes avoid a third literal copy of the sentence. - `driver-sql` `sql-driver-json-column-refusal-shared-text.test.ts`, run beside the existing JSON-column, compile-refusal-seam and provenance suites: 248 passed. It checks every `FILTER_OPERATORS` member against the shared set: the driver's thrown message and withheld diagnostic equal the core builder's output. - **Byte identity of the move.** A scratch capture through the built driver-sql (`SqlDriver` over SQLite) covered all 22 spellings plus bare equality, unmarked and author-marked, message and diagnostic, 46 entries. Before (`8f784959c`) and after: `cmp` identical, sha256 `dbcf32f5…534b2` on both. driver-sql's `dist` no longer contains the sentence. - `objectql` `engine-aggregate-filter-json-column-refusal.test.ts` (engine-level cell over the `find()` read shape): 74 passed. It covers 18 family cases on each of `owners`, `tags` and `meta` (`code`, `status`, the `$contains` / `$or` prescription, the field absent from the message, field and operator in the logged diagnostic, and the driver never asked for a row), an empty table (pure and grouped), the logged position, 11 answered cases (membership, null predicates, `title` controls) and the per-row floor. - `rest` `aggregation-filter-json-column-refusal.test.ts`: 52 per cell. SQLite passes and a live PostgreSQL 16.14 passes locally; MySQL is a named skip. Every family case asserts that the per-aggregation 400 body's `error` is **the same string** as its `where` twin's. objectstack-ai#21004's `aggregation-filter-array-membership.test.ts` still passes beside it. - **Full suites.** Read at merge `1a226419e`: objectql local 6948 passed, rest local 5072 passed / 247 skipped, core 1809 passed. Read before the first merge: driver-sql 3285 passed / 188 skipped. `typecheck` passed for core, driver-sql, objectql and rest. At head `6e541b101`: core, driver-sql (refusal suites), objectql `engine-aggregate*` (571 passed) and rest `aggregation-filter*` (150 passed with PostgreSQL) re-ran green. - **Ablation A: the engine gate call deleted** (`ablation-replace`, plus a rebuilt objectql `dist`, plus `ablation-dist-preflight --absent`): - objectql suite: 56 of 74 red. The 54 family cases, the empty table and the logged position failed; the 11 answered cases and the 7 floor cases stayed green. - rest suite: 92 of 104 red (46 per dialect). Populated `owners` / `tags` cases still got a 400 from the per-row floor, without the logged diagnostic. `meta` negations (`$ne`, `$nin`, `$nin []`, `$not $in`, where `meta` is null on every row) answered 200 `{ n: 6, m: 6 }`; the mechanism was not traced. The empty table answered 200. - Restored: blob equals HEAD, `git diff HEAD` empty, objectql rebuilt, preflight shows the marker present in 4 dist files with a clean tree, and both suites green again (74 and 104). - **Ablation B: the per-row operator floor replaced by a no-op** (src, engine suite): 5 red, the 4 operator floor cases and the no-value row; restored blob equals HEAD. - **Round 2: the direct-caller pins** (`engine-aggregate-filter-json-column-refusal.test.ts`, 92 passed). For `meta` (json) `$ne`, `$nin` and `$not $in`, each on four cells: an empty row set, an empty grouped row set, `meta` null in every row with the filter as spec `lowerFilterCondition` lowers it (the shape that carries the `$null` arm), and the same rows with the filter as written. Each must refuse 400 `INVALID_FILTER` with exactly `engine.aggregate`'s message, and hand the diagnostic (field, operator, `At aggregations[1].filter.…`) to `reportWithheld`. Also pinned: no reporter means the same refusal; no field map means nothing judged (`m: 0`, as before); and `$contains` still answers. - **Ablation C: the new `applyInMemoryAggregation` call deleted** (`ablation-replace`, anchor 1 to 0, blob `c65412761a90` to `f530761c0559`): 13 of 92 red. - The 9 empty, empty-grouped and lowered-null cells, plus the no-reporter case, answered instead of refusing. That is the backstop's 200. - The 3 as-written null-row cells were refused by the backstop but with no diagnostic reported. - Restored: blob equals HEAD `c65412761a90`, `git diff HEAD` empty, 92 passed again. - **Round 2 at `f66bed950`:** objectql local full suite 7008 passed (356 files), rest `aggregation-filter*` 150 passed / 61 skipped with a live PostgreSQL 16.14, core refusal pin 6 passed, driver-sql refusal pins 141 passed. - **Driver conformance ledger:** 50 covered cells, 0 in the DEBT ledger, 0 exempt, both before and after, in both rounds. ## Gates - `node scripts/pm/dispatch-gates.mjs --commands` (no paths) derived 70 families at `6e541b101`. - **68 ran, exit 0.** Among them: `check:adr-0087-registration`, `check:changeset-no-major`, `check:engine-double-contract`, `check:nul-bytes`, `check:doc-authoring`, `check:driver-conformance`, `check:driver-memory-census`, `check:query-options-erasure` and `check:test-source-alias`. - **2 NOT MEASURED (exit 3, prerequisite not met):** `check:dual-build-cjs-loads` and `check:type-check-debt`. Both need the whole workspace built; two attempts at that build timed out in the shared verify-lock queue. CI's lint job builds first. - `--ran` reconciliation: 70 derived, 68 run, 2 NOT MEASURED, 0 unrun. - **Round 2 at `f66bed950`:** re-derived with no paths, the same 70 families; 68 ran with exit 0 and the same 2 were NOT MEASURED (exit 3). `--ran`: 70 derived, 68 run, 2 NOT MEASURED, 0 unrun. `typecheck` passed for core and objectql. Narrowed lint: 10 changed `.ts` files, 0 errors, 0 warnings. - An earlier run caught one real finding, fixed in `e7bd7f667` ("type the shared-text pin's find options"): `check:query-options-erasure`'s test surface grew 236 to 237 because of an `as any` on a `find` options bag in the new driver-sql test. - **Lint, narrowed and declared:** `eslint --no-inline-config --format json` over the 9 changed `.ts` files reports 9 files, 0 errors, 0 warnings. `eslint.config.mjs` sets no `parserOptions.project` and registers no typed rule, so linting is not type-aware and this diff cannot move a verdict on an untouched file. The full `pnpm lint` is CI's. ## Acceptance notes - **Round 2, from the at-tier review (5926186339).** - F10 (b): `applyInMemoryAggregation` is gated (above). - F10 (a): the backstop docblocks are corrected. - F12 and ①.6: the export count is three, and the changeset's "Who is affected" names `applyInMemoryAggregation` direct callers and the new optional `reportWithheld`. - **The "flip the `m: 0` / `m: 6` pins" step had nothing to flip.** No suite on `main` pinned a per-aggregation `$in` / `$nin` count on a JSON-stored field (objectstack-ai#21004's two suites pin only `$contains` / `$notContains`). The full objectql, rest and driver-sql runs found no other pin that this change turns. The refusal pins are new files beside objectstack-ai#21004's. - **A `{ $field }` comparand on a JSON-stored field** (`{ owners: { $eq: { $field: 'title' } } }`) is refused by this gate in the JSON-column words. driver-sql's `where` refuses it through its cross-field class rule, in that rule's words. Both answers are `INVALID_FILTER` / 400 with the field withheld, so the two faces disagree only on which sentence they print. - **The REST envelope truncates the shared message at 500 characters** on both faces, so it ends "…because the answ…". That is unchanged here by direction, and filed separately by the seat. - **Findings for the seat, not filed here:** - driver-memory's `where` answers the family per element on a multi-valued field, while the SQL family refuses it (engine-level measurement). The seat files it. - service-analytics' native measure-filter compiler has no JSON-column gate (read at source, not measured). Carrier: objectstack-ai#20987. --- _Generated by [Claude Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
…lared JSON-stored field, in the SQL family's words (objectstack-ai#21066) (objectstack-ai#21159) Fixes objectstack-ai#21066 Clause-②: yes (narrowing) On a field the object declares JSON-stored (a `multiple: true` field, `tags` / `multiselect` / `checkboxes`, or a structured-JSON type such as `json`), `driver-memory` now refuses the scalar-comparison family that `driver-sql`'s `where` refuses: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$nin` and implicit equality, whatever the comparand, at any depth. The answer is `INVALID_FILTER` / 400 with the same message. The operator set and the sentence are read from `@objectstack/core` (`JSON_COLUMN_INCOMPATIBLE_OPERATORS`, `jsonColumnOperatorRefusalText`, homed by PR objectstack-ai#21097). There is no third copy. `$contains` / `$notContains` (membership), `$null`, `$exists` and `$empty` keep answering. ## What was wrong (H1, measured at `origin/main` `670680e93` through `engine.find`) A real `ObjectQL` over `InMemoryDriver`, objectstack-ai#21004's six rows (`owners` is a `multiple: true` lookup, `tags` is a `tags` field). Every row reproduces the card: | `where` | before | now | |:--|:--|:--| | `owners` `$eq 'u1'` | `d1`, `d3` (per element) | 400 `INVALID_FILTER` | | `owners` `$in ['u1','u9']` | `d1`, `d3` | 400 | | `owners` `$nin ['u1','u9']` | `d2`, `d4`, `d5`, `d6` | 400 | | `owners` `$gt 'u1'` | `d1`, `d2`, `d3`, `d5` | 400 | | `tags` `$gt 'red'` | `d3` | 400 | | also: bare `{ owners: 'u1' }`, `$ne`, `$gte`, `$lt`, `$lte`, `$between`, `tags $eq`, `{ owners: null }`, `$eq null`, `$ne null`, `$in []`, `$nin []` | rows, per element | 400 | | controls: `owners $contains 'u1'` / `$notContains` / `$null` / `$exists` / `$empty`, `title $in` | rows | unchanged rows | The engine hands the driver the operators as written, except `$ne` / `$nin`. Those arrive inside the spec's null-safe lowering (`$and` of `$or` of `$null: true` and the operator). The gate walks `$and` / `$or` / `$not`, so that shape is refused too. The analytics face (`MemoryAnalyticsService`) answered the same per-element rows. Its SQL echo rendered `owners = 'u1'`, which matches no row over the JSON text the SQL family stores. It now refuses in `query()` and `generateSql()` alike. ## What changed - `filter-refusal.ts`: the shape gate (`assertFilterConditionShape`) takes an optional `FilterFieldDeclarations` (`isJsonStoredField`, `reportWithheld`). It has two arms. Implicit equality on a declared JSON-stored field is refused as `=`, bare. Any operator in the shared set is refused AFTER the existing comparand-shape rules, which is `driver-sql`'s order (comparand gate, then column-type gate). So an array under `$eq` or a one-element `$between` still gets its own refusal first. `jsonStoredFieldOperatorError` builds the error from the shared text: this package's `unsupportedFilterError` envelope, with the withheld diagnostic handed to `reportWithheld` (prefixed `At PATH:`) before the throw. - `memory-driver.ts`: `convertToMongoQuery` passes `this.filterFieldDeclarations(object)`. The population is `isJsonStoredField`, the predicate `$contains` already forks on (`STRUCTURED_JSON_TYPES` or `isMultiValueField`). So the fields where `$contains` asks membership are exactly the fields where the family is refused. The diagnostic goes to the driver's logger at `warn`, the level `driver-sql` uses for its withheld filter diagnostics. That keeps the message's "the full diagnostic is in the server log" true here. - `memory-analytics.ts`: `normalizeFilters` takes the cube. It judges a `where` key (a cube member) by the field it maps to on the cube's table, the same (table, field path) pair `filterContainsTest` reads. Its diagnostic goes to the analytics service's own logger. - `.changeset/21066-memory-json-column-family-refusal.md`: `@objectstack/driver-memory` `minor`, BREAKING banner, `Clause-②: yes (narrowing)`, one ADR-0087 marker `not-required (no-migration-prescription)`. No registered id covers a filter operator on a JSON-stored column. The one migration-registry entry that mentions json columns (`cel-predicate-one-value-comparand-refused`) is the CEL list-comparand surface, not this one. ## Hypotheses, measured - **H1** holds: the table above. - **H2.** The shared home is `@objectstack/core`'s `json-column-operator-refusal.ts`, and both names are read. Before this change `driver-memory` had NO withheld-diagnostic seam: every refusal it raises (the `$null` / `$exists` non-boolean refusals included) names the field in the message, and nothing in the package logged a diagnostic. This change keeps the shared posture: the message names neither field nor operator, and the diagnostic goes to the server log. - **H3.** `driver-sql` decides a JSON column from `jsonFields`, filled from `JSON_COLUMN_TYPES.has(type) or isMultiValueField(field)`. `JSON_COLUMN_TYPES` is `STRUCTURED_JSON_TYPES` plus `MULTI_OPTION_TYPES` plus the driver-internal `object` / `array` aliases. Memory's population is the same predicate less those aliases and less a single-value media field on an unmoved deployment (both recorded on `isJsonStoredField`). On a schemaless direct call (an object never passed through `syncSchema`), nothing is judged. Every operator answers per element as before, as `SqlDriver.isJsonColumn` answers `false` for a table it was never told about. Pinned. A field declared SCALAR (`text`) that holds an array is not judged either. - **H4.** `@objectstack/formula`'s `ORDERING_OPERATORS` docblock does NOT declare a per-element reading for the query plane. It records a non-alignment ("driver-memory's read, a frozen test driver, compares a stored list element by element and keeps returning those rows ... declared on objectstack-ai#15104"). objectstack-ai#15104 is the `$field` cross-field reference card, shut as `not_planned` under the driver-memory investment freeze. It rules nothing about the equality or ordering family on a stored list. So this is a formula-plane record of observed behaviour, not a query-plane contract, and no contract conflict stops the card. That docblock sentence goes stale on declared fields once this lands (see Acceptance notes). - **H5.** objectstack-ai#21009 widens the same shared set to the text operators. Both gates here read the set live, and the new suite iterates `JSON_COLUMN_INCOMPATIBLE_OPERATORS` intersected with this driver's vocabulary, with a floor of the nine `$`-spellings. So once both land, memory refuses `$startsWith` / `$endsWith` / `$icontains` on these fields with no edit here, and the suite pins them. Whichever of the two lands second merges `main` and checks the other's members on its face. The suite's `$contains` control is outside objectstack-ai#21009's scope. ## Pin sweep - The ONE per-element pin the package carried on a declared field flipped: `memory-20444-empty-operator.test.ts` had `{ tags: { $empty: true, $ne: null } }` giving `r2`. It is now a refusal pin (`code` + `status` + the shared message). The composition (`$empty` beside a has-a-value sibling on one multi-value field) is kept through `$null: false`, which answers `r2`. `driver-sql`/SQLite answers that row too, and refuses the `$ne: null` spelling with the same body (measured on the built driver). - `memory-matcher-scalar-comparand-array-value.test.ts` pins per-element answers on a column declared `text`. Those cells still hold, and a header note now says the population there is a scalar-declared column. - Repo-wide: only two tests outside this package bind the real driver (`packages/runtime`'s two ruled consumers). Neither filters a JSON-stored field. No other `INVALID_FILTER` pin moves. ## Tests (final head `13407b76f`) - `pnpm --filter @objectstack/driver-memory exec vitest run --maxWorkers=2`: **70 files, 1703 passed**. The first run after the implementation, before any test edit: 69 files, 1 red of 1613, the per-element pin flipped above. - `pnpm --filter @objectstack/driver-memory run typecheck`: exit 0. `tsc --listFiles` includes both edited test files. - New `memory-21066-json-column-family-refusal.test.ts` (89 tests): - the card's five rows; - every family member on `owners` / `tags` / `meta` (`json`); - ten shapes per field (bare, bare null, `$eq null`, `$ne null`, the engine's `$ne` lowering, `$in []`, `$nin []`, under `$not`, an `$or` branch after a holding one, `$eq` beside `$contains`); - `count` / `findOne` / `updateMany` / `deleteMany` refusing with the table untouched; - the withheld message plus the logged `At filter.$or[1].owners.$gte:` diagnostic; - comparand-first ordering; - eleven answered controls; - the declaration boundary (undeclared object, scalar-declared column, the gate with and without declarations); - the analytics face, `query()` and `generateSql()`, including the `cube.member` spelling, plus its log line and a `$contains` control. - Ablations (`node scripts/ablation-replace.mjs`, wrap mode, run at `de1fef341`; the two later merges touched no file in this package; each restore proven blob == HEAD with `git diff HEAD` empty). The subjects are this package's `src`, imported relatively, so no `dist` is involved: - **A** `filterFieldDeclarations`' predicate forced to `() => false`: **73 red / 37 green** of 110 across the new file and 20444. The 17 green in the new file are exactly the answered controls, the declaration-boundary trio, the premise, the comparand-first case and the analytics `$contains` control. - **B** the implicit-equality arm disabled: **7 red**, exactly the six bare cases and the analytics bare case. - **C** the operator arm disabled: **67 red**, every operator-based refusal pin, the direct-gate test and the 20444 flip. - Gate union, derived with `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` (no paths; 7 changed paths, working tree clean) at `13407b76f`: **60 derived, 60 run, every one exit 0**. Reconciled with `--ran`: "60 derived famil(ies) accounted for — 60 run, 0 NOT-MEASURED (a DERIVED zero — all 60 recorded an exit code and none of them is 3)". The same 60 also ran all-zero at the previous merge head `5b75fe461`. At `de1fef341`, `check:dual-build-cjs-loads` exited 3 (PREREQUISITE NOT MET, no dist yet); it measured on both later heads. - Driver conformance ledger (`node scripts/check-driver-conformance.mjs`), before and after: byte-identical. 50 covered cells, 0 DEBT, 0 exempt. The shared matrix has no JSON-column or multi-value case-set, so this invariant is held by the per-package pins, not the matrix. ## Acceptance notes - **The class gains one method.** `InMemoryDriver.filterFieldDeclarations` is tagged `@internal`. It is not private only because the analytics face is another class. `FilterFieldDeclarations` is not exported from the package root, but the method does appear in the published `.d.ts`. objectstack-ai#20984 graded the analogous public `filterContainsTest` as a surface widening (`Clause-②: yes (widening)`). The seat graded it so (5929927010): the line is `yes (narrowing)`, with the semver (`minor`) and the ADR-0087 marker unchanged. - **Surface beyond the claim's list.** `memory-driver.ts` and `memory-analytics.ts` are edited. The gate cannot see a declaration on its own, so the plumbing is the minimum the direction needs, and the analytics face calls the same gate. No open PR touched either file when read before the first edit. - **The AST comparison-node door** (`{ type: 'comparison', field: 'owners', operator: '=', value: 'u1' }`) still answers per element on a declared field: `d1`, `d3`, measured on the built driver. No seam emits that form (the engine and the protocol hand a driver a FilterCondition), so it is reachable only by a direct driver call. Left alone. - **The shared sentence's mechanism clause** ("a field this driver stores as a JSON TEXT column", "$in/$eq matched nothing") is `driver-sql`'s, and is literally untrue of this driver and of the engine's per-aggregation face. The prescription (`$contains`, an `$or` of `$contains`) is right on all three. Inherited as objectstack-ai#21007 shipped it. objectstack-ai#21009 is the PR that next edits the shared home. - `@objectstack/formula`'s `ORDERING_OPERATORS` docblock ("driver-memory's read ... keeps returning those rows") is now true only of undeclared objects. It is a comment, and no claim holds that file. - **Tooling.** With the `turbo` 2.10.10 to 2.11.5 bump now on `main`, every repo-scoped turbo run in an agent session appends a managed "turborepo-agent-rules" block (an HTML-comment-delimited section) to `AGENTS.md`. These include `pnpm exec turbo run build`, `pnpm check:type-check-debt`, `check:query-options-erasure` and `check:slot-lookup`. It happened repeatedly in this worktree and was restored each time, and every gate derivation above was taken on a clean tree; this PR does not touch `AGENTS.md`. Tracked as objectstack-ai#21146 (PR objectstack-ai#21151). --- _Generated by [Claude Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Fixes #20873
Clause-②: no
What changes
The per-aggregation
filter(engine.aggregate({ aggregations: [{ …, filter }] }), and soPOST /api/v1/data/:object/query) is evaluated by the engine's own walker,matchesAggregationFilterinpackages/objectql/src/having-filter.ts. Its$containsarm failed every value that was not a string, so a stored array never matched, and its$notContainsarm passed every such value, members included.On a DECLARED JSON-stored field both arms now ask MEMBERSHIP, the reading
FILTER_OPERATORS'$containsdocblock (@objectstack/spec) declares andwherealready gives on every SQL dialect (SqlDriver.applyJsonMembership):$contains: vholds whenvnames an element of the stored array. A member stored as a JSON number or boolean is named by its text ('1'names1,'1.50'names1.5,'true'namestrue,'null'namesnull). That is the candidate setdriver-sql'sjsonMembershipCandidatesbinds on every dialect. Array-only, as the SQL constructs are.$notContains: vis its exact complement, and a row with no value still satisfies it (非否定路径上的$ne/$nin/$notContains:driver-sql 排除 NULL 行,driver-memory / formula 返回它们(#5146 只裁定了$not) #5298), ascol IS NULL OR NOT (…)does in SQL.having.Files:
having-filter.ts(the two arms,storedArrayHasMember,declaredJsonStoredFields, and an optionaljsonStoredset threaded throughmatchesHaving/matchesAggregationFilter), plusin-memory-aggregation.ts. That file is the one place the engine hands the object's declared field map to the per-aggregation filter, so it reads the declared set once per call besidedeclaredFieldClasses.having-filter.tsis not on objectql's published entry points.in-memory-aggregation.tsis (applyInMemoryAggregationandbucketDateValue, fromindex.tsandcore.ts), and neither exported signature changes; the declared set is threaded through the internalaggregateBucketonly.The card's table, through the REST door, before and after
Measured with a real
SqlDriveron SQLite and on a live PostgreSQL 16.14. Rows:d1 ['u1','u2'],d2 ['u2'],d3 ['u3','u1'],d4 [],d5 ['u10'],d6 null. Base212d613c, head90ba78d9.wheretwinm, basem, headowners $contains 'u1'(the card)owners $contains 'u10'owners $notContains 'u1'tags $contains 'red'(d3holds['redwood'])tags $notContains 'red'$orof$containsu1 / u3 (the any-of spelling #7398's refusal prescribes)$notoverowners $contains 'u1'title $contains 'u1'(text, the control)On the in-memory driver the per-aggregation
mis the same evaluator's answer, also 2 now. Memory's ownwhereanswers 3 for the card at this base (d5too, by a per-element substring). That face belongs to #20874 (in flight), whose branch (10656601) moves it to membership and pinsd1, d3.The fork: by the DECLARED column (Zone 2 H2)
The fork reads the declaration (
STRUCTURED_JSON_TYPESorisMultiValueField), never the row. That is the contract's sentence: "One operator, two questions, selected by the COLUMN rather than by the caller". It is alsoSqlDriver.isJsonColumn's population (built from the same two spec sets) and #20874'sisJsonStoredField, character for character.Measured: on every fixture reachable through the public doors, the declared reading and a value-shape reading select the same rows.
$contains/$notContainson a declared structured-JSON field is refused before any row is read: the engine's text-operator declared-type door,INVALID_FILTER400, inwhereand in the per-aggregation filter alike, on all three backends.find()value is an array ornullon memory, SQLite and PostgreSQL alike. The write door wraps a scalar:'u1'is stored as['u1']on memory and SQLite.The two readings differ only on rows a direct caller hands the walker: a declared multi-valued column holding a scalar string, or an undeclared column holding an array. There the declared reading gives what SQL
wheregives (no member; the substring reading), and a value-shape reading would not. Both cases are pinned. Noopen_questionsfork results.Zone 2 hypotheses, measured
m: 0).$in/$ninleft as they are.where: { owners: { $in: ['u1','u9'] } }is refusedINVALID_FILTER400 on SQLite and PostgreSQL (the driver-sql: a declared operator on amultiple: true(JSON array) column silently answers wrong —$in/$eqalways zero rows,$ninreturns the rows it was asked to EXCLUDE #7398 JSON-column gate). Memory'swhereanswersd1, d3.m: 0for$inandm: 6for$nin, a 200 wherewhereis a 400. That is reported as an out-of-scope finding, not pinned.having.groupByon a multi-valued field is refused 400 on all three backends, and on a structured-JSON field too.min/maxover a multi-valued field. That is answered three ways: an array on memory, the serialized TEXT on SQLite's native aggregate,DATABASE_ERROR500 on PostgreSQL. So there is no singlewhereanswer to holdhavingto, andhavingis not handed the declared set.having$containson agroupBytext projection keeps substring, on SQLite and PostgreSQL at the REST door and on the engine level.$notContains 'u1'counted 6 wherewherecounts 4.applyJsonMembership's complement. No other claim holdshaving-filter.ts: #5930 step 4 (domain:engine): the engine-fed faces delete their hand-copied filter meaning (driver-sql, turso remote, memory query, mongodb, formula,having); the memory reference matcher retires (D6) #20822 group 3 is unclaimed, and [finding] the aggregationfilterandhavingread a non-boolean$existsby truthiness and DROP a non-boolean$null, on every driver: the engine evaluates both in-process, and its gate refuses only$empty#20981 is filed bare. Same gate family.d6is counted, as SQL'scol IS NULL OR NOT (…)counts it.in-memory-aggregation.tsand the two test files.driver-sql'sjsonMembershipCandidatesanddriver-memory'scontainsMemberCandidates(landed by PR fix(driver-memory): $contains on a multi-valued or JSON-stored field is membership, on every face #20984 after this branch's merge base) are both module-private in driver packages, which objectql does not depend on. SostoredArrayHasMemberis the third copy of the rule onmain; see Acceptance notes.Compile-surface conclusions
driver-sqlapplyFilterCondition$contains/$notContainson a JSON column go throughapplyJsonMembership; thewheretwin numbers in the table above are this face, measured on SQLite and PostgreSQL 16.14.driver-sqlite-wasmanddriver-tursolocal inherit it (not measured separately).RemoteTransport.buildWhereSQL212d613c: its$contains/$notContainsarms gopushLike(substring over the stored text) with no JSON-column fork. Not measured (no remote libsql here). In the out-of-scope finding below.compileScopedFilterToSql212d613c:{ owners: { $contains: 'u1' } }compiles toinstr("t"."owners", ?) > 0on SQLite, which admits a row holding["u10"]. On PostgreSQL it compiles to"t"."owners" LIKE ? ESCAPE ?over a json column. In the out-of-scope finding below (an RLS read scope).lowerAnalyticsWherewhereto aFilterConditionand adds no$containsreading of its own. The ObjectQL strategy hands that to the driver (face 1). The native SQL strategy mapscontainsto the substring LIKE shape (native-sql-strategy.ts, read, not measured), in the same finding as face 3.formulamatchesFilterConditiontypeof actual === 'string' && typeof v === 'string' && actual.includes(v)(unchanged by #20972). Measured:['u1','u2']→ false,['u10']→ false,'u1 memo'→ true. In the out-of-scope finding below.having-filterapplyHaving/matchesHavingwithout a declared set unchanged (H4).driver-memory/driver-mongodbtranslateFieldOperatorscompiles$containsto a bare$regex, which MongoDB applies per array element (per-element substring). Read, not measured.Tests
pnpm --filter @objectstack/objectql exec vitest run --project local --maxWorkers=2 src/engine-aggregate-filter-array-membership.test.ts— 30 passed. The card's rows with the rows themselves, empty table, per group,havingcontrol, the member-text reading (number / exponent / boolean / null / non-JSON-number spellings / nested / object / scalar), the declared fork, the declared population.OS_TEST_POSTGRES_URL=… pnpm --filter @objectstack/rest exec vitest run --project local --maxWorkers=2 src/aggregation-filter-array-membership.test.ts— 18 passed (9 SQLite, 9 live PostgreSQL 16.14), 9 named skips (MySQL). Each row runs beside its livewheretwin, populated and empty.localproject on the merged head90ba78d9: 349 files, 6851 tests passed;repoproject 1 file / 5 passed.90ba78d9with the PostgreSQL cell live: 9 files, 106 passed, 34 skipped.pnpm --filter @objectstack/objectql typecheckandpnpm --filter @objectstack/rest typecheck: green. Both new test files are in their package's test program (tsc -p tsconfig.test.json --listFiles).Reverse verification
Each leg ran through
scripts/ablation-replace.mjs(anchor hits proven on disk; restore proven blob == HEAD andgit diff HEADempty), from the committed change.$containsarm put back to the substring test: 15 of 30 red (every membership$containsrow, the member-text rows, the declared-fork row); the$notContainsrows and controls green, as predicted.$notContainsarm put back: 3 red (its two rows and the complement row).dist/.declaredJsonStoredFieldswas emptied, objectql rebuilt, andablation-dist-preflight.mjsfound the marker in 4 built files. 12 red: the 6 membership rows on each of SQLite and PostgreSQL. Text control,havingand empty-table rows stayed green. Restore leg: rebuilt, marker absent from all 14 built files, tree clean, 18 passed.Driver conformance ledger
node scripts/check-driver-conformance.mjs: before (212d613c) "50 covered cell(s), 0 in the DEBT ledger, 0 exempt"; after (90ba78d9) the same.Gates
node scripts/pm/dispatch-gates.mjs --commandsre-derived with no paths at90ba78d9gives 63 commands, all run, exit codes recorded to disk.--ranreconciliation: 63 derived, 61 run (all exit 0), 2 NOT MEASURED, 0 unrun.check:dual-build-cjs-loadsandcheck:type-check-debt. Both are PREREQUISITE NOT MET (exit 3): they need the whole-workspace buildlint.ymlperforms first, and 42 / 5 packages have nodist/in this worktree.check-engine-split-ratiofirst refused on the shallow checkout. It was green after a deepen to its window (git fetch --shallow-since=2026-06-26 origin main).90ba78d9:eslint --no-inline-config --format jsonreports 4 files, 0 errors, 0 warnings.--print-config).eslint.config.mjsnever enables type-aware linting (noparserOptions.project, no typed rules; its own lines 327-328 say so), so this diff cannot move an untouched file's verdict.Acceptance notes
main.storedArrayHasMemberrestatesdriver-sql'sjsonMembershipCandidatesas a predicate.driver-memory'scontainsMemberCandidates(PR fix(driver-memory): $contains on a multi-valued or JSON-stored field is membership, on every face #20984) is the other JS copy.@objectstack/spec/data, besideasciiCaseInsensitiveContainsandisEmptyFilterValue, the value-level filter rules every JS face already reads from there.$contains/$notContainson a declared multi-valued or JSON-stored field still answer SUBSTRING on five faces, the analytics RLS read scope among them (u1admits a row storingu10) #20987.FILTER_TEXT_CASEShas no array rows. The membership fixtures are literal per package:sql-driver-17590-json-column-membership.test.ts, [finding] driver-memory answers$containson a stored array by substring per element (u1matches a row storingu10), where the SQL drivers answer membership; the spec docblock records the gap against a card that answers 404 #20874'smemory-20874-contains-membership.test.ts, and the two files here. These use the sameu1/u10/redwooddisagreement rows.*_CASESkit besideFILTER_TEXT_CASESwould let all three faces be driven by one table.wheretwin.find()presents (measured identical on all three backends). A realInMemoryDriverconsumer would need a ruled entry in thecheck:driver-memory-censusledger.OS_TEST_POSTGRES_URL/OS_TEST_MYSQL_URLforpackages/rest, the same notedata-group-by-json-door.test.tscarries. The PostgreSQL cell's local run is above.$in/$ninon a multi-valued field answer 200 (m: 0/m: 6, the latter counting the rows it was asked to exclude) wherewhereis a 400 on the SQL family, and memory'swhereanswers membership;$containsmembership contract is not answered on the turso remote transport, the service-analytics SQL compilers (an RLS read scope over-reaches),driver-mongodbandformula;min/maxover a multi-valued field: three answers (array / serialized text / PostgreSQL 500);$startsWith/$icontainson a multi-valued field: PostgreSQLwhere500, SQLite over the serialized text, memory per element.Generated by Claude Code