Skip to content

fix(driver-memory): $contains on a multi-valued or JSON-stored field is membership, on every face - #20984

Merged
objectstack-fleet[bot] merged 3 commits into
mainfrom
claude/issue-20874-memory-contains-membership
Oct 1, 2026
Merged

objectstack-fleet[bot] merged 3 commits into
mainfrom
claude/issue-20874-memory-contains-membership

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #20874
Clause-②: yes (widening)

What changes

driver-memory now answers $contains and $notContains on a declared JSON-stored field by whole-element membership, the reading driver-sql compiles on all three dialects. A field counts as JSON-stored when it is multiple: true, a multiselect / checkboxes / tags field, or a STRUCTURED_JSON_TYPES member such as json. A scalar text column keeps the case-exact substring test. The change covers every face of the package:

  • the query path's $-spelling (normalizeFieldOperators) and its AST spelling (convertConditionToMongo): find, count, and every verb that goes through convertToMongoQuery;
  • the analytics (cube) face's mingo $match;
  • the analytics face's SQL echo (generateSql), which now renders SQLite's json_each membership construct for such a column, so the echoed statement still returns the rows the chart was drawn from.

One rule backs all of them: InMemoryDriver.filterContainsTest(object, field, value). On a JSON-stored field it returns { $elemMatch: { $in: members, $not: { $type: 'array' } } }, and on any other field { $regex: filterSubstringPattern(value) }. $notContains is $not over that test, on both faces. The members come from the comparand's text, read the way driver-sql's jsonMembershipCandidates reads it: '1' names the string or the number 1, '1.50' names 1.5, and 'true' / 'false' / 'null' name the string or the JSON literal.

The $contains docblock in packages/spec/src/data/filter.zod.ts lost its stale "driver-memory DOES NOT ANSWER IT YET" bullet. Its dead tracker link went with it (that card answers 404). The replacement text states the measured status, and states that the other text operators over a stored array are not ruled by that section.

Measured on origin/main f6ccca4a before the change, and after (HEAD 10656601)

Fixture: driver-sql's #17590 fixture plus a multi-valued lookup owners (['u1','u2'], ['u10'], ['u3','u1'], []).

where memory before SQLite (driver-sql) memory after
{ owners: { $contains: 'u1' } } 1, 2, 3 1, 3 1, 3
{ owners: { $notContains: 'u1' } } 4 (2 dropped) 2, 4 2, 4
{ tags_: { $contains: 'red' } } 1, 2 1 1
{ nums: { $contains: '1' } } (members are numbers) none 1 1
{ label: { $contains: 'red' } } (scalar control) 1, 2 1, 2 1, 2
the cube face, the same filters the query path's rows n/a the query path's rows

engine.find on a real InMemoryDriver with the nested-relation filter { owners: { region: 'NA' } } (owner u1 NA, u10 EU; this is PR #20872's lowering): d1, d3, d5 before and d1, d3 after. Its $not gave d2, d4 before and d2, d4, d5 after, which matches the rest suite's SQLite rows. That was a one-off run (a throwaway probe file, not committed): this package may not import the engine and the engine's packages may not import this driver (check:driver-memory-census). What is pinned instead is the driver input the engine sends, an $or of one $contains per related id, which objectql's engine-nested-relation-lowering.test.ts asserts.

Mechanism hypotheses (order Zone 2)

  • H1 confirmed. The $contains arm lowered to an escaped $regex, and mingo applies a $regex to each element of an array value. Reproduced on the fixture above before any edit.

  • H2: the fork is by the DECLARED field, not the row's runtime shape. Memory has the declaration (valueShapes, recorded by syncSchema since the $empty work), and SQL forks on its JSON-column registry, which is also filled from the declaration. The population is the spec's JSON-stored classes: STRUCTURED_JSON_TYPES plus isMultiValueField, the two halves driver-sql's registry is built from. The two readings DO give different rows on one fixture, so the choice goes to the PM in the report's open_questions: a declared json field holding the scalar string 'u1'. SQLite answers no member, because its constructs are array-only. A per-row-shape fork would answer it by substring. The declared fork matches SQLite, and the contract text says the question is "selected by the COLUMN", declared metadata. Undeclared fields (an object never passed through syncSchema) keep the substring reading, as SqlDriver.isJsonColumn answers false for a table it was never told about.

  • H3 confirmed and pinned. Number members answered nothing; '1', '2', '10', '0' and '1.50' now give SQLite's rows.

  • H4 confirmed: $notContains diverged. It is the mirror arm of the same defect, changed under os-dev rule 3's bounded in-place exemption. All four conditions hold:

    1. Same defect class.
    2. Mechanical, with its shape pinned by SQL's col IS NULL OR NOT (…): $not over the test admits null and missing rows, pinned on both fixtures.
    3. memory-driver.ts is held by no other claim; the sibling $exists card is kept off it by its own claim.
    4. Same tests, no new gate.

    The claim's surface should be amended to include it.

  • H5 confirmed. The cube face borrowed filterSubstringPattern and wrapped it in its own $regex, so it had the same defect. It now takes the driver's whole test. The echo reads its member set off that same test, so the chart and its echo cannot name different sets.

  • H6 confirmed. See the engine.find run above.

Compile-surface conclusions

# face conclusion
1 driver-sql applyFilterCondition (driver-sqlite-wasm, driver-turso local inherit it) already compliant: applyJsonMembership emits membership on JSON columns. Evidence: the #17590 suite now carries a multi-valued lookup column and the u1/u10 case, green on SQLite here. Its live PostgreSQL/MySQL cells are unprovisioned locally; the required live job runs the whole driver-sql suite. The two inheriting drivers were not separately run.
2 turso RemoteTransport.buildWhereSQL out of scope (another package). By reading, its $contains arm emits pushLike (a GLOB substring) on every column, JSON columns included, so remotely u1 would match ["u10"] while the local transport answers membership. Not measured. Reported as a finding.
3 service-analytics read-scope-sql compileScopedFilterToSql out of scope (another package). Measured: { owners: { $contains: 'u1' } } on a declared lookup + multiple: true field compiles to instr("t"."owners", ?) > 0 on SQLite and LIKE '%u1%' on PostgreSQL/MySQL, a substring over the stored JSON text. This is an RLS read-scope face, so it is reported as a security-relevant finding.
4 service-analytics filter-normalizer lowerAnalyticsWhere out of scope. By reading, it lowers $contains to the cube contains operator, which the native SQL strategy renders as LIKE, a substring. Reported.
5 formula matchesFilterCondition out of scope. By reading, the arm is typeof actual === 'string' && actual.includes(v), so a stored array never matches (fail-closed). Reported.
half objectql having-filter out of scope by the claim (serial behind another card). Untouched.
unfrozen driver-memory query path and cube face (memory-analytics.ts), behind filter-refusal.ts changed (this PR). filter-refusal.ts and the $exists arm are untouched (sibling card).
unfrozen driver-mongodb translateFieldOperators out of scope (another package). By reading, it lowers $contains to a native $regex, which MongoDB applies per array element: the same defect. Not measured. Reported.

Why the shared pin is a mirrored literal table, not FILTER_TEXT_CASES (order Zone 3, not taken)

FILTER_TEXT_CASES has no array column, so adding one changes the fixture of all five enrolled drivers. driver-mongodb imports every row of it and still carries the per-element defect, so its suite would go red. The table's own rule 2 says rows join a driver's suite in the PR that ends that driver's gap. Doing it here would also widen the spec touch beyond the one declared docblock. A new sibling case-set would add DEBT rows to a ledger that only goes down. So the new memory file mirrors driver-sql's #17590 fixture row for row and asserts the same literal row sets. That is how the #17590 file already holds its three dialect cells to one answer.

Files

  • packages/drivers/driver-memory/src/memory-driver.ts: the population (isJsonStoredField), the members, the one test (filterContainsTest), both query-path spellings.
  • packages/drivers/driver-memory/src/memory-analytics.ts: the $match rows take the driver's test; the echo renders membership (sqliteMembershipPredicate).
  • packages/drivers/driver-memory/src/memory-20874-contains-membership.test.ts (new): two fixtures. The first is the driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 fixture plus owners. The second holds the stored shapes SQLite decides: a scalar, an object, a nested array, [null], [[null]], [true, 1.5], [''] and a NULL row, each answer measured on driver-sql/SQLite first. Every case runs on find in both spellings, on count, and on the cube's $match. The cube's echo is EXECUTED on sql.js over the same rows. The file also covers the nested-relation driver input and the fork.
  • packages/drivers/driver-sql/src/sql-driver-17590-json-column-membership.test.ts: an owners multi-valued lookup column and the u1/u10 case (the claim's SQLite pin).
  • packages/spec/src/data/filter.zod.ts: the one declared docblock.
  • .changeset/20874-memory-contains-membership.md: minor, not the patch the order suggested. filterContainsTest is a new public method on the exported InMemoryDriver. It ships in dist/index.d.ts (the existing filterSubstringPattern appears there as the positive control), and the level ruling quoted in pr-automation.yml (WHICH LEVEL) grades an additive widening of a published package's public surface at least minor. Clause-②: yes (widening): the new public method widens the published surface; no filter key or operator is added.
  • Sweep, beyond the claim's listed surface. The order's pin sweep asks for every same-semantics statement to be flipped in one round. Each item below is an edit to a statement this change makes false:
    • packages/rest/src/data-nested-object-door.test.ts (a header comment that cited the gap and the removed docblock clause);
    • .changeset/20802-nested-relation-filter-served.md, one sentence of a pending release note that said the in-memory driver still matches per element.

Confirmation requested: a pending release note is corrected

This PR changes .changeset/20802-nested-relation-filter-served.md, a pending release note this PR did not add. This is the DELIBERATE CORRECTION class check-empty-changeset.mjs names, so Check Changeset stays red on purpose. It is not a required context. skip-changeset must not be applied. The note's last sentence said:

On the in-memory driver, a multi-valued relation's $contains still matches a stored id by substring per element, so there an id that is a substring of another stored id (u1 inside u10) also matches; SQLite and PostgreSQL match the element.

This PR makes that false, so it now reads:

SQLite, PostgreSQL and the in-memory driver match the element of a multi-valued relation, so an id that is a substring of another stored id (u1 inside u10) does not match it.

Please confirm the correction on this PR. If the release that consumes that note ships before this PR lands, the edit should be dropped on rebase.

Tests (HEAD 10656601)

  • pnpm --filter @objectstack/driver-memory typecheck passes, and vitest run gives 68 files / 1553 tests passed. Both typecheck programs include the new test file (--listFiles).
  • The new file alone gives 113 passed.
  • pnpm --filter @objectstack/driver-sql typecheck passes (its program includes the edited test). sql-driver-17590-json-column-membership.test.ts gives 22 passed, 2 skipped; the skips are the live PostgreSQL/MySQL cells, unprovisioned here, NOT MEASURED locally.
  • pnpm --filter @objectstack/spec typecheck passes. spec build and then check:generated report "All 15 generated artifacts are up to date".

Reverse verification. The code change was committed first. Then memory-driver.ts and memory-analytics.ts were checked out at BASE under an EXIT/INT/TERM restore trap. The landed mutation was verified on disk before the run: filterContainsTest count 0, two escapeRegex(val) arms back, and both files byte-equal to BASE.

The predicted direction was that the cases where per-element substring and membership disagree go red and the agreeing ones stay green. Measured: 69 failed / 44 passed. The failures were every disagreeing case, on all four faces, plus the executed echo, the relation-lowering input and the fork pin. The label controls and the redwood, ab, u10 and '0' cases stayed green. The restore was proven by blob hashes equal to HEAD (d376dadb… / 9de9198b…), an empty git diff HEAD and clean porcelain.

Guard ablation (scripts/ablation-replace.mjs, anchor 1 to 0). Dropping $not: { $type: 'array' } turned exactly the nested-array cases red: 12, the json 'u1', its complement and 'null', on all four faces. The executed echo stayed green, since SQLite is the oracle there. The file was restored to the HEAD blob with an empty git diff HEAD.

Lint (narrowed, a measurement).

  1. Population: eslint.config.mjs lints **/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs} minus NEVER_LINTED. The six changed TS files are all in it; the two changesets are not.
  2. eslint --no-inline-config --format json over those six files reports 6 files, 0 errors, 0 warnings at 10656601.
  3. The config never enables type-aware linting (no parserOptions.project, no typed rules; it says so itself), so this diff cannot move any untouched file's verdict.

Driver conformance ledger: node scripts/check-driver-conformance.mjs reads "50 covered cell(s), 0 in the DEBT ledger, 0 exempt" both at f6ccca4a (before) and at 10656601 (after).

Gates: dispatch-gates --commands was re-derived at 10656601. That gives 84 commands, six more than the dispatch list: engine-double-contract, objectql-double-limit, query-options-erasure, type-check-coverage, type-check-debt and where-matcher. The run also covered the eight roster gates flagged as sharing a directory with these paths. All exit 0 except:

  • check-empty-changeset: exit 1, the deliberate correction above.
  • check:dual-build-cjs-loads and check:type-check-debt: exit 3, PREREQUISITE NOT MET (they need the whole workspace built). NOT MEASURED; CI's Build/Lint jobs own those.
  • check:lean-entry-closure: exit 3 at first. It passed after building objectql.

Acceptance notes (observed, not filed by this PR)


Generated by Claude Code

…ership, on every face

A multi-valued or JSON-stored field now answers `$contains` / `$notContains`
by whole-element membership (the SQL family's reading), on the query path's
two spellings and on the analytics face's `$match` and its SQLite echo. A
scalar column keeps the case-exact substring test. The fork is the field's
declared storage shape, read from the declaration `syncSchema` recorded.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
…ory face; one seam; docblock and sweep

- memory-20874-contains-membership.test.ts: driver-sql's #17590 fixture row
  for row, plus the stored shapes SQLite decides (scalar, object, nested
  array, null/boolean/fractional members), through find() in both spellings,
  count(), the analytics query and its SQL echo executed on sql.js, and the
  nested-relation lowering's driver input.
- driver-sql #17590 suite: a multi-valued lookup column and the u1/u10 case.
- driver-memory: one public seam, filterContainsTest; the analytics echo
  reads its member set off the test the $match exit runs.
- spec: the $contains docblock's stale driver-memory status and dead tracker
  link are replaced with the measured status.
- the rest suite header and the pending #20802 changeset no longer state the
  in-memory gap.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/driver-memory, @objectstack/spec, touching 18 documentable anchor(s). ⚠️ 1 changed file(s) yielded no anchor (packages/spec/src/data/filter.zod.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/data-modeling/drivers.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/permissions/authentication.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/plugins/packages.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/protocol/objectql/query-syntax.mdx (via InMemoryDriver (symbol, a top-level class))

⛔ 4 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/implementation-status.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v16.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v17/17-0.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v17/17-3.mdx (via InMemoryDriver (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/spec/src/data/filter.zod.ts) — pages documenting those are invisible to this run
  • 4 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 138 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 05be3525961a4977ea49eda507e44a4482f87681 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from d5b60fc179b36ee6d41982a8c9bd69b30dc48485 — the merge of head 1bf8dbb609ca103f2e7b7cd062dfad72a0e2ba17 into base 05be3525961a4977ea49eda507e44a4482f87681, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin d5b60fc179b36ee6d41982a8c9bd69b30dc48485 && git checkout d5b60fc179b36ee6d41982a8c9bd69b30dc48485
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 05be3525961a4977ea49eda507e44a4482f87681 1bf8dbb609ca103f2e7b7cd062dfad72a0e2ba17 && git checkout -B drift-repro 05be3525961a4977ea49eda507e44a4482f87681 && git merge --no-ff 1bf8dbb609ca103f2e7b7cd062dfad72a0e2ba17

node scripts/docs-audit/affected-docs.mjs --json 05be3525961a4977ea49eda507e44a4482f87681

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 05be3525961a4977ea49eda507e44a4482f87681 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

…ing)

filterContainsTest is a new public method on the exported InMemoryDriver
class, so the published surface widens; the entry stays minor.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 1bf8dbb609ca103f2e7b7cd062dfad72a0e2ba17
Local-runs: none

Inputs: card #20874 (body and all five comments: triage 5914978120, claim 5921357350, report 5922095993, claim amendment 5922144972, patch-round report 5922235042), PR #20984 (body, its 8-file list, the net diff of refs/review/pr20984 against merge-base f6ccca4a), and the 46 check-runs on this head. Nothing was built, run or re-run; every claim below is judged by reading the diff against the base tree.

① Derived judgments

  1. Population (which fields ask membership) — right. isJsonStoredField = STRUCTURED_JSON_TYPES.has(type) || isMultiValueField(shape) over the valueShapes map syncSchema records (indexValueShapes: type plus multiple === true). driver-sql's registry is JSON_COLUMN_TYPES.has(type) || isMultiValueField(field) with JSON_COLUMN_TYPES = STRUCTURED_JSON_TYPES plus MULTI_OPTION_TYPES plus the driver-internal object/array aliases; isMultiValueField already covers MULTI_OPTION_TYPES, so the two populations agree on every authorable declaration and differ only on the two introspection aliases the memory driver never produces and on single-value media (per-deployment on SQL, bare id on memory). The changeset's type list (multiple: true on select/radio/lookup/user/file/image; multiselect/checkboxes/tags; structured types such as json) is exactly MULTI_CAPABLE_TYPES, MULTI_OPTION_TYPES, STRUCTURED_JSON_TYPES.
  2. The fork is the DECLARED field, not the row (H2 = A) — right. The spec's $contains docblock selects the question by the column (declared metadata), and driver-sql forks on isJsonColumn, a declaration registry. A per-row fork would answer a declared json field holding the scalar 'u1' by substring where every SQL dialect answers no member, and would read the answer off data. Triage's wording "by the stored value's shape" is subordinate to its own clause "matching the SQL drivers' existing rule", and that rule is declared. An undeclared object keeps substring, as isJsonColumn answers false for an unknown table; the test pins both halves.
  3. Member reading — right, and in parity. containsMemberCandidates (text; true/false/null literal; a finite number under the spelled-out JSON number grammar) is the same reading as driver-sql's jsonMembershipCandidates (same regex, same String(value) rendering, '1.50' names 1.5). Each candidate's JSON.stringify is the text the SQL construct binds.
  4. The test shape — right by reading; the CI verdict is pending (③). { $elemMatch: { $in: members, $not: { $type: 'array' } } } is array-only (a scalar, object or null stored value has no member) and excludes a nested array, the twin of SQLite's typeof(os_member.key) = 'integer' plus JSON-text equality. $notContains is $not over that test, which admits null and missing rows, the twin of col IS NULL OR NOT (...). The dev's reverse verification (69 red / 44 green at base) and guard ablation (exactly the 12 nested-array cases red) are the dev's own evidence, not a gate verdict.
  5. Every verb, both spellings — right. convertToMongoQuery(query.where, object) is the one entry for find/findOne/count/aggregate/update/delete (six call sites), it threads object into convertConditionToMongo (already took it; recursion passes it) and now into the single normalizeFieldOperators call. $elemMatch is a new lowered key that assembleLoweredWrites merges generically; no other operator writes it; the substring arm still joins regexConditions so composition with $startsWith/$endsWith is unchanged.
  6. No accept-set change — right. filter-refusal.ts and SUPPORTED_FIELD_OPERATORS are untouched, so an author-written $elemMatch is still refused; $contains/$notContains keep their declared string comparand. No key, operator, error code or refusal is added or removed on any face.
  7. Analytics $match — right. The builder input's substring became containment, resolved with extractTableName(cube.sql) and resolveFieldPath, the same resolution storageFormFor already uses for filterComparandStorageForm; a cube whose sql is not a bare object name falls to substring, which is this face's pre-existing resolution rule, not a new defect.
  8. Analytics SQL echo — right. sqliteMembershipPredicate is driver-sql's SQLite construct with literals where it binds (json_valid guard, integer key, CASE for the three literals else json_quote); notContains stays (col IS NULL OR NOT EXISTS (...)). The member set is read off the same filterContainsTest, so the chart and its echo cannot name two sets. The test executes the echo on sql.js, already a devDependency; package.json is unchanged.
  9. Public surface — right, declared. filterContainsTest is a new public method on InMemoryDriver, which index.ts exports and exports["."] maps to dist/index.d.ts: reachable from the entry type graph, so a widening. MemoryContainsTest is export type in memory-driver.ts and not re-exported from index.ts, as the changeset states. PR body line 2 and the changeset both read Clause-②: yes (widening), matching the amended claim 5922144972.
  10. Spec docblock (packages/spec/src/data/filter.zod.ts) — right, docblock only. The stale "DOES NOT ANSWER IT YET" bullet and the dead driver-memory: the stored-ARRAY value axis is still unrepaired outside the equality arm — $in/$nin, the text family and the ordering family answer one filter two ways, and the two exclusion arms answer it in the WIDENING direction #17286 link are gone; the replacement states the declared-field fork and the measured rows. The added sentence that $startsWith still differs (memory per element, SQLite over the serialized text) is consistent with driver-sql: $startsWith is not in JSON_COLUMN_INCOMPATIBLE_OPERATORS, so it reaches applyLike over the JSON text. No schema, key or generated artifact moves; Spec property liveness and Governed Surface Queue Guard are green on this head.
  11. The two test-side touches — right. driver-sql's driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 fixture gains an owners multi-valued lookup on every row and the u1/u10 case (['1','3'], ['2'], $notContains ['2','4']); no existing assertion changes. packages/rest/src/data-nested-object-door.test.ts is a header comment flipped to the new status, and its pointer to the memory pin is accurate (the memory file pins { $or: [{ owners: { $contains: 'u1' } }] } and its $not, the driver input objectql's engine-nested-relation-lowering.test.ts asserts the engine sends).
  12. Sweep residue — one stale statement remains, outside the claim and outside the review faces (③ item 7). On this head, git grep finds no other "DOES NOT ANSWER IT YET" or driver-memory: the stored-ARRAY value axis is still unrepaired outside the equality arm — $in/$nin, the text family and the ordering family answer one filter two ways, and the two exclusion arms answer it in the WIDENING direction #17286 reference, and no content/docs or apps/docs page states the per-element behaviour (query-syntax.mdx line 347 is about drivers(memory, mongodb): the $contains family still folds case — the last two backends left on the wrong side of #4706 Q2 = A #6682 case folding). packages/objectql/src/engine.ts line 15427 still says driver-memory matches per element and every backend answers a substring superset.

② Semver level

  • @objectstack/driver-memory 17.5.0, changeset minor, Clause-②: yes (widening) — right. The act is a new public method on an exported class (① item 9); WHICH LEVEL grades an additive widening of a published surface at least minor and the commit type fix( cannot lower it; AGENTS.md: yes takes at least minor. The row-set change itself is a fix toward the contract the spec docblock already declared, refuses no input, and the changeset carries the migration paragraph ("If your tests relied on the old answer"). (narrowing) does not apply.
  • No other package publishes: driver-sql and rest move test files only; spec moves one docblock (no key, no runtime, no generated artifact). No second changeset is owed and skip-changeset must not be applied.
  • Check Changeset on this head is red at step 12 ("Reject an empty-frontmatter changeset added by this PR") because .changeset/20802-nested-relation-filter-served.md exists at the merge base and is edited here. This is a DELIBERATE CORRECTION, not a collision: this PR's own note is .changeset/20874-memory-contains-membership.md, a different filename; the 20802 file is byte-identical at the merge base and on current origin/main (still pending, consumed by no release); exactly one sentence of it changes and the rest of the file is untouched. The gate's own annotation names this class and its remedy ("say so on the PR and get it confirmed"); this record is that confirmation. The one rewritten sentence, judged against this head: "SQLite, PostgreSQL and the in-memory driver match the element of a multi-valued relation, so an id that is a substring of another stored id (u1 inside u10) does not match it." — right. The engine lowers a multi-valued relation condition to an $or of one $contains per related id (objectql's lowering test); on this head a declared lookup with multiple: true is in the membership population (① items 1, 2), so ['u10'] does not answer 'u1', pinned on the mirrored fixture; the SQLite and PostgreSQL halves are the original sentence's own clause, unchanged. The removed clause ("still matches a stored id by substring per element") is exactly the statement this diff makes false, so restoring it from base would put a false sentence into objectql's CHANGELOG. If the release that consumes the 20802 note ships before this PR lands, the edit drops on rebase and nothing else changes.
  • Because step 12 is red, steps 13 to 15 of Check Changeset were skipped, so check-changeset-no-major's LEVEL AXIS and the ADR-0087 step have no CI verdict on this head. By the rule text ("declared yes must grade at least one package whose published source it moves minor or above"), driver-memory at minor satisfies it; no breaking changeset exists, so ADR-0087 is not applicable. The dev's offline --event run is the dev's evidence, not a gate verdict.

Clause-②: yes (widening) — one new public method InMemoryDriver.filterContainsTest on the exported class; no accepted filter key, operator, comparand shape or error code is added or removed.

③ Boundary flags

  1. open_questions[0] (H2, declared field vs row shape) — answered: A, the declared field. Reasoning in ① item 2. The in-seat answer in 5922235042 is not adopted; it is re-derived here and agrees.
  2. open_questions[1] (correct the pending 20802 note in this PR) — answered: A, keep the correction. The note is named and its one rewritten sentence judged in ②. This record is the confirmation the gate asks for.
  3. H4, $notContains under the bounded in-place exemption — accepted. Same operator family, same seam (filterContainsTest), a mechanical $not mirror with its null rule pinned on both fixtures, no new gate, and "No other open PR may claim the same single-writer path" is green on this head; the claim was amended (5922144972) to carry it.
  4. Deviation, minor not the order's patch — right (②).
  5. Deviation, FILTER_TEXT_CASES not used as the carrier — right. That table has no array column; driver-mongodb imports every row and still lowers $contains to a per-element $regex, so a shared row would turn a suite outside this claim red; the literal mirrored fixture holds the two packages to one answer the way the driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 file already holds its three dialect cells.
  6. Deviation, no permanent engine.find pin on memory — accepted. check:driver-memory-census forbids a new consumer; the driver-input pin (memory) plus the lowering pin (objectql) cover the seam from both sides.
  7. Stale docblock left in packages/objectql/src/engine.ts (delete-probe, line 15427) — escalated to the seat, not a FAIL. It says driver-memory matches per element and every backend answers a substring superset; false on this head for memory (and already for driver-sql since driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590). The code stays correct because membership is a subset of substring and the exact narrowing through storedReferenceIncludes is unchanged. It is a code comment, outside the five review faces and outside the claim; it needs a carrier (a comment-only rider on the next objectql touch, or a card).
  8. out_of_scope_findings (five class-b, two carrier-none) — escalated to the seat for carriers; none blocks this PR. The service-analytics read-scope-sql finding (an RLS read-scope compiler answering $contains on a multi-valued field by substring, measured) is security-relevant and should not wait for the family card; turso remote transport, mongodb, formula and the analytics filter-normalizer are the same family by reading.
  9. Docs drift (PR comment 5922055209) — no docs edit owed. The four hand-written pages name InMemoryDriver only; none states the per-element behaviour (① item 12). Release-owned pages are untouched.
  10. Check-runs on this head. Red by design: Check Changeset (step 12, the ruled correction class, ②). Green: Build Core, Type Check · source gates, Type Check · debt ledger, filter, Governed Surface Queue Guard, Spec property liveness, Flag docs affected by code changes, Check Documentation Links, the four card/claim guards, Auto Label, Check PR Size. Skipped: Build Docs, Console Pin Gate, Packed-tarball smoke. Not verdicts, still in_progress at review time (not awaited): Test Core 1/6 to 6/6, Dogfood Regression Gate 1/3 to 3/3, Dogfood Verify CLI, Temporal Conformance (live PG + MySQL), Type Check · workspace, Type Check · consumer gates, Lint & Repo Gates. The new memory test file (113 cases), the edited driver-sql suite, and the live PostgreSQL/MySQL cells of the driver-sql: the $contains MEMBERSHIP spelling on any multi-valued / JSON column is a DATABASE_ERROR 500 on live PostgreSQL (SQLSTATE 42883, operator does not exist: json ~~ text) — it has only ever been executed on SQLite #17590 file (the only measurement of the new owners rows on those two dialects; the changeset's "SQLite, PostgreSQL and MySQL" sentence rests on it for those rows) are answered only when those jobs conclude; the dev's local counts are not adopted as gate verdicts. Landing waits on them; this record's verdict is on the contract.
  11. PR is a draft (draft: true); the seat marks it ready when the checks above conclude.

Implemented-by: claude/issue-20874-memory-contains-membership
Reviewed-by: session_01Ujdtvqs7ree7WyQmEDwEnG

VERDICT: PASS

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Check Changeset stays red on this PR by design, and it is carried into the queue

domain:engine#2 (seat post #20966) · session_01Ujdtvqs7ree7WyQmEDwEnG · 2026-10-01T00:57Z.

  • The gate and step: Check Changeset (pr-automation.yml), step 12, "Reject an empty-frontmatter changeset added by this PR". It is red on head 1bf8dbb6 because this PR edits .changeset/20802-nested-relation-filter-served.md, a pending note it did not add.
  • The reason: a DELIBERATE CORRECTION. The note's last sentence said the in-memory driver matches u1 inside u10; this PR makes that false. The gate's own text says it "stays red either way" and asks for confirmation on the PR. The at-tier contract review 5922367217 is that confirmation: it names the note and judges the one rewritten sentence right.
  • Why it may be carried: the gate is red by design on a pushed branch. pr-automation.yml has no merge_group trigger, so the queue never runs it. And Check Changeset is not one of the queue's seven required contexts. Every other check on this head is green or a rostered skip (check-expected-skips --pr 20984: OK).

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 1, 2026 00:58
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 1, 2026 00:59
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit f8178ff Oct 1, 2026
48 of 51 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-20874-memory-contains-membership branch October 1, 2026 01:23
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Oct 7, 2026
…alued field by membership, as its where twin does (objectstack-ai#21004)

Fixes objectstack-ai#20873
Clause-②: no

## What changes

The per-aggregation `filter` (`engine.aggregate({ aggregations: [{ …,
filter }] })`, and so `POST /api/v1/data/:object/query`) is evaluated by
the engine's own walker, `matchesAggregationFilter` in
`packages/objectql/src/having-filter.ts`. Its `$contains` arm failed
every value that was not a string, so a stored array never matched, and
its `$notContains` arm passed every such value, members included.

On a DECLARED JSON-stored field both arms now ask MEMBERSHIP, the
reading `FILTER_OPERATORS`' `$contains` docblock (`@objectstack/spec`)
declares and `where` already gives on every SQL dialect
(`SqlDriver.applyJsonMembership`):

- `$contains: v` holds when `v` names an element of the stored array. A
member stored as a JSON number or boolean is named by its text (`'1'`
names `1`, `'1.50'` names `1.5`, `'true'` names `true`, `'null'` names
`null`). That is the candidate set `driver-sql`'s
`jsonMembershipCandidates` binds on every dialect. Array-only, as the
SQL constructs are.
- `$notContains: v` is its exact complement, and a row with no value
still satisfies it (objectstack-ai#5298), as `col IS NULL OR NOT (…)` does in SQL.
- A scalar text column keeps the substring test, unchanged. So does
`having`.

Files: `having-filter.ts` (the two arms, `storedArrayHasMember`,
`declaredJsonStoredFields`, and an optional `jsonStored` set threaded
through `matchesHaving` / `matchesAggregationFilter`), plus
`in-memory-aggregation.ts`. That file is the one place the engine hands
the object's declared field map to the per-aggregation filter, so it
reads the declared set once per call beside `declaredFieldClasses`.
`having-filter.ts` is not on objectql's published entry points.
`in-memory-aggregation.ts` is (`applyInMemoryAggregation` and
`bucketDateValue`, from `index.ts` and `core.ts`), and neither exported
signature changes; the declared set is threaded through the internal
`aggregateBucket` only.

### The card's table, through the REST door, before and after

Measured with a real `SqlDriver` on SQLite and on a live PostgreSQL
16.14. Rows: `d1 ['u1','u2']`, `d2 ['u2']`, `d3 ['u3','u1']`, `d4 []`,
`d5 ['u10']`, `d6 null`. Base `212d613c`, head `90ba78d9`.

| query | `where` twin | per-aggregation `m`, base | `m`, head |
|:--|:--|:--|:--|
| `owners $contains 'u1'` (the card) | 2 | 0 | 2 |
| `owners $contains 'u10'` | 1 | 0 | 1 |
| `owners $notContains 'u1'` | 4 | 6 | 4 |
| `tags $contains 'red'` (`d3` holds `['redwood']`) | 2 | 0 | 2 |
| `tags $notContains 'red'` | 4 | 6 | 4 |
| `$or` of `$contains` u1 / u3 (the any-of spelling objectstack-ai#7398's refusal
prescribes) | 2 | 0 | 2 |
| `$not` over `owners $contains 'u1'` | 4 | 6 | 4 |
| `title $contains 'u1'` (text, the control) | 3 | 3 | 3 |

On the in-memory driver the per-aggregation `m` is the same evaluator's
answer, also 2 now. Memory's own `where` answers 3 for the card at this
base (`d5` too, by a per-element substring). That face belongs to objectstack-ai#20874
(in flight), whose branch (`10656601`) moves it to membership and pins
`d1, d3`.

## The fork: by the DECLARED column (Zone 2 H2)

The fork reads the declaration (`STRUCTURED_JSON_TYPES` or
`isMultiValueField`), never the row. That is the contract's sentence:
"One operator, two questions, selected by the COLUMN rather than by the
caller". It is also `SqlDriver.isJsonColumn`'s population (built from
the same two spec sets) and objectstack-ai#20874's `isJsonStoredField`, character for
character.

Measured: on every fixture reachable through the public doors, the
declared reading and a value-shape reading select the same rows.
- `$contains` / `$notContains` on a declared structured-JSON field is
refused before any row is read: the engine's text-operator declared-type
door, `INVALID_FILTER` 400, in `where` and in the per-aggregation filter
alike, on all three backends.
- A multi-valued field's `find()` value is an array or `null` on memory,
SQLite and PostgreSQL alike. The write door wraps a scalar: `'u1'` is
stored as `['u1']` on memory and SQLite.

The two readings differ only on rows a direct caller hands the walker: a
declared multi-valued column holding a scalar string, or an undeclared
column holding an array. There the declared reading gives what SQL
`where` gives (no member; the substring reading), and a value-shape
reading would not. Both cases are pinned. No `open_questions` fork
results.

## Zone 2 hypotheses, measured

- **H1 — confirmed.** The arms were as hypothesised; the card's table
reproduced on all three backends (`m: 0`).
- **H2 — declared column**, above.
- **H3 — `$in` / `$nin` left as they are.**
- `where: { owners: { $in: ['u1','u9'] } }` is refused `INVALID_FILTER`
400 on SQLite and PostgreSQL (the objectstack-ai#7398 JSON-column gate). Memory's
`where` answers `d1, d3`.
- The drivers disagree and SQL refuses, so no membership semantics are
invented here.
- The per-aggregation answer stays `m: 0` for `$in` and `m: 6` for
`$nin`, a 200 where `where` is a 400. That is reported as an
out-of-scope finding, not pinned.
- **H4 — `having`.**
- A `groupBy` on a multi-valued field is refused 400 on all three
backends, and on a structured-JSON field too.
- The one aggregated column that can still hold a stored array is a
`min` / `max` over a multi-valued field. That is answered three ways: an
array on memory, the serialized TEXT on SQLite's native aggregate,
`DATABASE_ERROR` 500 on PostgreSQL. So there is no single `where` answer
to hold `having` to, and `having` is not handed the declared set.
- What is pinned: `having` `$contains` on a `groupBy` text projection
keeps substring, on SQLite and PostgreSQL at the REST door and on the
engine level.
- **H5 — confirmed, so the mirror arm moved under os-dev rule 3's
bounded in-place exemption.** Base per-aggregation `$notContains 'u1'`
counted 6 where `where` counts 4.
- All four conditions hold. Same defect class, same arm pair. The shape
is pinned by `applyJsonMembership`'s complement. No other claim holds
`having-filter.ts`: objectstack-ai#20822 group 3 is unclaimed, and objectstack-ai#20981 is filed
bare. Same gate family.
- The NULL row `d6` is counted, as SQL's `col IS NULL OR NOT (…)` counts
it.
- The claim's file surface does not name this arm. PM: please amend it,
together with `in-memory-aggregation.ts` and the two test files.
- **H6 — no importable predicate.** `driver-sql`'s
`jsonMembershipCandidates` and `driver-memory`'s
`containsMemberCandidates` (landed by PR objectstack-ai#20984 after this branch's
merge base) are both module-private in driver packages, which objectql
does not depend on. So `storedArrayHasMember` is the third copy of the
rule on `main`; see Acceptance notes.

## Compile-surface conclusions

| # | face | conclusion |
|:--|:--|:--|
| 1 | `driver-sql` `applyFilterCondition` | **already compliant
(evidence)** — `$contains` / `$notContains` on a JSON column go through
`applyJsonMembership`; the `where` twin numbers in the table above are
this face, measured on SQLite and PostgreSQL 16.14. `driver-sqlite-wasm`
and `driver-turso` local inherit it (not measured separately). |
| 2 | turso `RemoteTransport.buildWhereSQL` | **out of scope (reason)**
— an independent compiler this card does not touch. Read at `212d613c`:
its `$contains` / `$notContains` arms go `pushLike` (substring over the
stored text) with no JSON-column fork. Not measured (no remote libsql
here). In the out-of-scope finding below. |
| 3 | service-analytics `compileScopedFilterToSql` | **out of scope
(reason)** — a different compiler. Measured function-level at
`212d613c`: `{ owners: { $contains: 'u1' } }` compiles to
`instr("t"."owners", ?) > 0` on SQLite, which admits a row holding
`["u10"]`. On PostgreSQL it compiles to `"t"."owners" LIKE ? ESCAPE ?`
over a json column. In the out-of-scope finding below (an RLS read
scope). |
| 4 | service-analytics `lowerAnalyticsWhere` | **out of scope
(reason)** — it lowers the analytics `where` to a `FilterCondition` and
adds no `$contains` reading of its own. The ObjectQL strategy hands that
to the driver (face 1). The native SQL strategy maps `contains` to the
substring LIKE shape (`native-sql-strategy.ts`, read, not measured), in
the same finding as face 3. |
| 5 | `formula` `matchesFilterCondition` | **out of scope (reason:
fenced; PR objectstack-ai#20972 landed on this file during this run)** — its arm is
`typeof actual === 'string' && typeof v === 'string' &&
actual.includes(v)` (unchanged by objectstack-ai#20972). Measured: `['u1','u2']` →
false, `['u10']` → false, `'u1 memo'` → true. In the out-of-scope
finding below. |
| half | objectql `having-filter` | **changed** — the per-aggregation
filter, as above; `applyHaving` / `matchesHaving` without a declared set
unchanged (H4). |
| unfrozen | `driver-memory` / `driver-mongodb` | **out of scope
(reason: fenced, objectstack-ai#20874 / objectstack-ai#20897 in flight).** Memory measured above,
and objectstack-ai#20874's fork matches this one. Mongo's `translateFieldOperators`
compiles `$contains` to a bare `$regex`, which MongoDB applies per array
element (per-element substring). Read, not measured. |

## Tests

- `pnpm --filter @objectstack/objectql exec vitest run --project local
--maxWorkers=2 src/engine-aggregate-filter-array-membership.test.ts` —
30 passed. The card's rows with the rows themselves, empty table, per
group, `having` control, the member-text reading (number / exponent /
boolean / null / non-JSON-number spellings / nested / object / scalar),
the declared fork, the declared population.
- `OS_TEST_POSTGRES_URL=… pnpm --filter @objectstack/rest exec vitest
run --project local --maxWorkers=2
src/aggregation-filter-array-membership.test.ts` — 18 passed (9 SQLite,
9 live PostgreSQL 16.14), 9 named skips (MySQL). Each row runs beside
its live `where` twin, populated and empty.
- objectql whole `local` project on the merged head `90ba78d9`: 349
files, 6851 tests passed; `repo` project 1 file / 5 passed.
- REST aggregation-adjacent files on `90ba78d9` with the PostgreSQL cell
live: 9 files, 106 passed, 34 skipped.
- `pnpm --filter @objectstack/objectql typecheck` and `pnpm --filter
@objectstack/rest typecheck`: green. Both new test files are in their
package's test program (`tsc -p tsconfig.test.json --listFiles`).

### Reverse verification

Each leg ran through `scripts/ablation-replace.mjs` (anchor hits proven
on disk; restore proven blob == HEAD and `git diff HEAD` empty), from
the committed change.

- **A1** — the `$contains` arm put back to the substring test: 15 of 30
red (every membership `$contains` row, the member-text rows, the
declared-fork row); the `$notContains` rows and controls green, as
predicted.
- **A2** — the `$notContains` arm put back: 3 red (its two rows and the
complement row).
- **B** — the REST suite reads objectql through `dist/`.
`declaredJsonStoredFields` was emptied, objectql rebuilt, and
`ablation-dist-preflight.mjs` found the marker in 4 built files. 12 red:
the 6 membership rows on each of SQLite and PostgreSQL. Text control,
`having` and empty-table rows stayed green. Restore leg: rebuilt, marker
absent from all 14 built files, tree clean, 18 passed.

### Driver conformance ledger

`node scripts/check-driver-conformance.mjs`: before (`212d613c`) "50
covered cell(s), 0 in the DEBT ledger, 0 exempt"; after (`90ba78d9`) the
same.

### Gates

- `node scripts/pm/dispatch-gates.mjs --commands` re-derived with no
paths at `90ba78d9` gives 63 commands, all run, exit codes recorded to
disk. `--ran` reconciliation: 63 derived, 61 run (all exit 0), 2 NOT
MEASURED, 0 unrun.
- NOT MEASURED: `check:dual-build-cjs-loads` and
`check:type-check-debt`. Both are PREREQUISITE NOT MET (exit 3): they
need the whole-workspace build `lint.yml` performs first, and 42 / 5
packages have no `dist/` in this worktree.
- `check-engine-split-ratio` first refused on the shallow checkout. It
was green after a deepen to its window (`git fetch
--shallow-since=2026-06-26 origin main`).
- Lint, a declared narrowing over the four touched TS files at
`90ba78d9`: `eslint --no-inline-config --format json` reports 4 files, 0
errors, 0 warnings.
  - Each file maps to a config (`--print-config`).
- `eslint.config.mjs` never enables type-aware linting (no
`parserOptions.project`, no typed rules; its own lines 327-328 say so),
so this diff cannot move an untouched file's verdict.

## Acceptance notes

- **The member rule now has three copies on `main`.**
- `storedArrayHasMember` restates `driver-sql`'s
`jsonMembershipCandidates` as a predicate. `driver-memory`'s
`containsMemberCandidates` (PR objectstack-ai#20984) is the other JS copy.
- None can be imported by the others. The one shared home would be
`@objectstack/spec/data`, beside `asciiCaseInsensitiveContains` and
`isEmptyFilterValue`, the value-level filter rules every JS face already
reads from there.
  - Carrier: the convergence item recorded on objectstack-ai#20987.
- **No shared conformance kit for stored-array membership.**
- `FILTER_TEXT_CASES` has no array rows. The membership fixtures are
literal per package: `sql-driver-17590-json-column-membership.test.ts`,
objectstack-ai#20874's `memory-20874-contains-membership.test.ts`, and the two files
here. These use the same `u1` / `u10` / `redwood` disagreement rows.
- A spec `*_CASES` kit beside `FILTER_TEXT_CASES` would let all three
faces be driven by one table.
- Here the cross-face invariant is held by running each row beside its
live `where` twin.
- **The memory cell is engine-level.** It runs over the read shape
`find()` presents (measured identical on all three backends). A real
`InMemoryDriver` consumer would need a ruled entry in the
`check:driver-memory-census` ledger.
- **The PostgreSQL / MySQL REST cells are a named skip in CI.** No job
provisions `OS_TEST_POSTGRES_URL` / `OS_TEST_MYSQL_URL` for
`packages/rest`, the same note `data-group-by-json-door.test.ts`
carries. The PostgreSQL cell's local run is above.
- **Out-of-scope findings, for the seat to route** (full evidence in the
report on the card):
- per-aggregation `$in` / `$nin` on a multi-valued field answer 200 (`m:
0` / `m: 6`, the latter counting the rows it was asked to exclude) where
`where` is a 400 on the SQL family, and memory's `where` answers
membership;
- the `$contains` membership contract is not answered on the turso
remote transport, the service-analytics SQL compilers (an RLS read scope
over-reaches), `driver-mongodb` and `formula`;
- `min` / `max` over a multi-valued field: three answers (array /
serialized text / PostgreSQL 500);
- `$startsWith` / `$icontains` on a multi-valued field: PostgreSQL
`where` 500, SQLite over the serialized text, memory per element.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Oct 7, 2026
…lared JSON-stored field, in the SQL family's words (objectstack-ai#21066) (objectstack-ai#21159)

Fixes objectstack-ai#21066
Clause-②: yes (narrowing)

On a field the object declares JSON-stored (a `multiple: true` field,
`tags` / `multiselect` / `checkboxes`, or a structured-JSON type such as
`json`), `driver-memory` now refuses the scalar-comparison family that
`driver-sql`'s `where` refuses: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`,
`$lte`, `$between`, `$in`, `$nin` and implicit equality, whatever the
comparand, at any depth. The answer is `INVALID_FILTER` / 400 with the
same message. The operator set and the sentence are read from
`@objectstack/core` (`JSON_COLUMN_INCOMPATIBLE_OPERATORS`,
`jsonColumnOperatorRefusalText`, homed by PR objectstack-ai#21097). There is no third
copy. `$contains` / `$notContains` (membership), `$null`, `$exists` and
`$empty` keep answering.

## What was wrong (H1, measured at `origin/main` `670680e93` through
`engine.find`)

A real `ObjectQL` over `InMemoryDriver`, objectstack-ai#21004's six rows (`owners` is
a `multiple: true` lookup, `tags` is a `tags` field). Every row
reproduces the card:

| `where` | before | now |
|:--|:--|:--|
| `owners` `$eq 'u1'` | `d1`, `d3` (per element) | 400 `INVALID_FILTER`
|
| `owners` `$in ['u1','u9']` | `d1`, `d3` | 400 |
| `owners` `$nin ['u1','u9']` | `d2`, `d4`, `d5`, `d6` | 400 |
| `owners` `$gt 'u1'` | `d1`, `d2`, `d3`, `d5` | 400 |
| `tags` `$gt 'red'` | `d3` | 400 |
| also: bare `{ owners: 'u1' }`, `$ne`, `$gte`, `$lt`, `$lte`,
`$between`, `tags $eq`, `{ owners: null }`, `$eq null`, `$ne null`, `$in
[]`, `$nin []` | rows, per element | 400 |
| controls: `owners $contains 'u1'` / `$notContains` / `$null` /
`$exists` / `$empty`, `title $in` | rows | unchanged rows |

The engine hands the driver the operators as written, except `$ne` /
`$nin`. Those arrive inside the spec's null-safe lowering (`$and` of
`$or` of `$null: true` and the operator). The gate walks `$and` / `$or`
/ `$not`, so that shape is refused too.

The analytics face (`MemoryAnalyticsService`) answered the same
per-element rows. Its SQL echo rendered `owners = 'u1'`, which matches
no row over the JSON text the SQL family stores. It now refuses in
`query()` and `generateSql()` alike.

## What changed

- `filter-refusal.ts`: the shape gate (`assertFilterConditionShape`)
takes an optional `FilterFieldDeclarations` (`isJsonStoredField`,
`reportWithheld`). It has two arms. Implicit equality on a declared
JSON-stored field is refused as `=`, bare. Any operator in the shared
set is refused AFTER the existing comparand-shape rules, which is
`driver-sql`'s order (comparand gate, then column-type gate). So an
array under `$eq` or a one-element `$between` still gets its own refusal
first. `jsonStoredFieldOperatorError` builds the error from the shared
text: this package's `unsupportedFilterError` envelope, with the
withheld diagnostic handed to `reportWithheld` (prefixed `At PATH:`)
before the throw.
- `memory-driver.ts`: `convertToMongoQuery` passes
`this.filterFieldDeclarations(object)`. The population is
`isJsonStoredField`, the predicate `$contains` already forks on
(`STRUCTURED_JSON_TYPES` or `isMultiValueField`). So the fields where
`$contains` asks membership are exactly the fields where the family is
refused. The diagnostic goes to the driver's logger at `warn`, the level
`driver-sql` uses for its withheld filter diagnostics. That keeps the
message's "the full diagnostic is in the server log" true here.
- `memory-analytics.ts`: `normalizeFilters` takes the cube. It judges a
`where` key (a cube member) by the field it maps to on the cube's table,
the same (table, field path) pair `filterContainsTest` reads. Its
diagnostic goes to the analytics service's own logger.
- `.changeset/21066-memory-json-column-family-refusal.md`:
`@objectstack/driver-memory` `minor`, BREAKING banner, `Clause-②: yes
(narrowing)`, one ADR-0087 marker `not-required
(no-migration-prescription)`. No registered id covers a filter operator
on a JSON-stored column. The one migration-registry entry that mentions
json columns (`cel-predicate-one-value-comparand-refused`) is the CEL
list-comparand surface, not this one.

## Hypotheses, measured

- **H1** holds: the table above.
- **H2.** The shared home is `@objectstack/core`'s
`json-column-operator-refusal.ts`, and both names are read. Before this
change `driver-memory` had NO withheld-diagnostic seam: every refusal it
raises (the `$null` / `$exists` non-boolean refusals included) names the
field in the message, and nothing in the package logged a diagnostic.
This change keeps the shared posture: the message names neither field
nor operator, and the diagnostic goes to the server log.
- **H3.** `driver-sql` decides a JSON column from `jsonFields`, filled
from `JSON_COLUMN_TYPES.has(type) or isMultiValueField(field)`.
`JSON_COLUMN_TYPES` is `STRUCTURED_JSON_TYPES` plus `MULTI_OPTION_TYPES`
plus the driver-internal `object` / `array` aliases. Memory's population
is the same predicate less those aliases and less a single-value media
field on an unmoved deployment (both recorded on `isJsonStoredField`).
On a schemaless direct call (an object never passed through
`syncSchema`), nothing is judged. Every operator answers per element as
before, as `SqlDriver.isJsonColumn` answers `false` for a table it was
never told about. Pinned. A field declared SCALAR (`text`) that holds an
array is not judged either.
- **H4.** `@objectstack/formula`'s `ORDERING_OPERATORS` docblock does
NOT declare a per-element reading for the query plane. It records a
non-alignment ("driver-memory's read, a frozen test driver, compares a
stored list element by element and keeps returning those rows ...
declared on objectstack-ai#15104"). objectstack-ai#15104 is the `$field` cross-field reference card,
shut as `not_planned` under the driver-memory investment freeze. It
rules nothing about the equality or ordering family on a stored list. So
this is a formula-plane record of observed behaviour, not a query-plane
contract, and no contract conflict stops the card. That docblock
sentence goes stale on declared fields once this lands (see Acceptance
notes).
- **H5.** objectstack-ai#21009 widens the same shared set to the text operators. Both
gates here read the set live, and the new suite iterates
`JSON_COLUMN_INCOMPATIBLE_OPERATORS` intersected with this driver's
vocabulary, with a floor of the nine `$`-spellings. So once both land,
memory refuses `$startsWith` / `$endsWith` / `$icontains` on these
fields with no edit here, and the suite pins them. Whichever of the two
lands second merges `main` and checks the other's members on its face.
The suite's `$contains` control is outside objectstack-ai#21009's scope.

## Pin sweep

- The ONE per-element pin the package carried on a declared field
flipped: `memory-20444-empty-operator.test.ts` had `{ tags: { $empty:
true, $ne: null } }` giving `r2`. It is now a refusal pin (`code` +
`status` + the shared message). The composition (`$empty` beside a
has-a-value sibling on one multi-value field) is kept through `$null:
false`, which answers `r2`. `driver-sql`/SQLite answers that row too,
and refuses the `$ne: null` spelling with the same body (measured on the
built driver).
- `memory-matcher-scalar-comparand-array-value.test.ts` pins per-element
answers on a column declared `text`. Those cells still hold, and a
header note now says the population there is a scalar-declared column.
- Repo-wide: only two tests outside this package bind the real driver
(`packages/runtime`'s two ruled consumers). Neither filters a
JSON-stored field. No other `INVALID_FILTER` pin moves.

## Tests (final head `13407b76f`)

- `pnpm --filter @objectstack/driver-memory exec vitest run
--maxWorkers=2`: **70 files, 1703 passed**. The first run after the
implementation, before any test edit: 69 files, 1 red of 1613, the
per-element pin flipped above.
- `pnpm --filter @objectstack/driver-memory run typecheck`: exit 0. `tsc
--listFiles` includes both edited test files.
- New `memory-21066-json-column-family-refusal.test.ts` (89 tests):
  - the card's five rows;
  - every family member on `owners` / `tags` / `meta` (`json`);
- ten shapes per field (bare, bare null, `$eq null`, `$ne null`, the
engine's `$ne` lowering, `$in []`, `$nin []`, under `$not`, an `$or`
branch after a holding one, `$eq` beside `$contains`);
- `count` / `findOne` / `updateMany` / `deleteMany` refusing with the
table untouched;
- the withheld message plus the logged `At filter.$or[1].owners.$gte:`
diagnostic;
  - comparand-first ordering;
  - eleven answered controls;
- the declaration boundary (undeclared object, scalar-declared column,
the gate with and without declarations);
- the analytics face, `query()` and `generateSql()`, including the
`cube.member` spelling, plus its log line and a `$contains` control.
- Ablations (`node scripts/ablation-replace.mjs`, wrap mode, run at
`de1fef341`; the two later merges touched no file in this package; each
restore proven blob == HEAD with `git diff HEAD` empty). The subjects
are this package's `src`, imported relatively, so no `dist` is involved:
- **A** `filterFieldDeclarations`' predicate forced to `() => false`:
**73 red / 37 green** of 110 across the new file and 20444. The 17 green
in the new file are exactly the answered controls, the
declaration-boundary trio, the premise, the comparand-first case and the
analytics `$contains` control.
- **B** the implicit-equality arm disabled: **7 red**, exactly the six
bare cases and the analytics bare case.
- **C** the operator arm disabled: **67 red**, every operator-based
refusal pin, the direct-gate test and the 20444 flip.
- Gate union, derived with `node scripts/pm/dispatch-gates.mjs
--commands --repo objectstack-ai/objectstack` (no paths; 7 changed
paths, working tree clean) at `13407b76f`: **60 derived, 60 run, every
one exit 0**. Reconciled with `--ran`: "60 derived famil(ies) accounted
for — 60 run, 0 NOT-MEASURED (a DERIVED zero — all 60 recorded an exit
code and none of them is 3)". The same 60 also ran all-zero at the
previous merge head `5b75fe461`. At `de1fef341`,
`check:dual-build-cjs-loads` exited 3 (PREREQUISITE NOT MET, no dist
yet); it measured on both later heads.
- Driver conformance ledger (`node
scripts/check-driver-conformance.mjs`), before and after:
byte-identical. 50 covered cells, 0 DEBT, 0 exempt. The shared matrix
has no JSON-column or multi-value case-set, so this invariant is held by
the per-package pins, not the matrix.

## Acceptance notes

- **The class gains one method.**
`InMemoryDriver.filterFieldDeclarations` is tagged `@internal`. It is
not private only because the analytics face is another class.
`FilterFieldDeclarations` is not exported from the package root, but the
method does appear in the published `.d.ts`. objectstack-ai#20984 graded the analogous
public `filterContainsTest` as a surface widening (`Clause-②: yes
(widening)`). The seat graded it so (5929927010): the line is `yes
(narrowing)`, with the semver (`minor`) and the ADR-0087 marker
unchanged.
- **Surface beyond the claim's list.** `memory-driver.ts` and
`memory-analytics.ts` are edited. The gate cannot see a declaration on
its own, so the plumbing is the minimum the direction needs, and the
analytics face calls the same gate. No open PR touched either file when
read before the first edit.
- **The AST comparison-node door** (`{ type: 'comparison', field:
'owners', operator: '=', value: 'u1' }`) still answers per element on a
declared field: `d1`, `d3`, measured on the built driver. No seam emits
that form (the engine and the protocol hand a driver a FilterCondition),
so it is reachable only by a direct driver call. Left alone.
- **The shared sentence's mechanism clause** ("a field this driver
stores as a JSON TEXT column", "$in/$eq matched nothing") is
`driver-sql`'s, and is literally untrue of this driver and of the
engine's per-aggregation face. The prescription (`$contains`, an `$or`
of `$contains`) is right on all three. Inherited as objectstack-ai#21007 shipped it.
objectstack-ai#21009 is the PR that next edits the shared home.
- `@objectstack/formula`'s `ORDERING_OPERATORS` docblock
("driver-memory's read ... keeps returning those rows") is now true only
of undeclared objects. It is a comment, and no claim holds that file.
- **Tooling.** With the `turbo` 2.10.10 to 2.11.5 bump now on `main`,
every repo-scoped turbo run in an agent session appends a managed
"turborepo-agent-rules" block (an HTML-comment-delimited section) to
`AGENTS.md`. These include `pnpm exec turbo run build`, `pnpm
check:type-check-debt`, `check:query-options-erasure` and
`check:slot-lookup`. It happened repeatedly in this worktree and was
restored each time, and every gate derivation above was taken on a clean
tree; this PR does not touch `AGENTS.md`. Tracked as objectstack-ai#21146 (PR objectstack-ai#21151).

---
_Generated by [Claude
Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation protocol:data size/l tests tooling

Projects

None yet

2 participants