Skip to content

fix(driver-memory)!: refuse the equality and ordering family on a declared JSON-stored field, in the SQL family's words (#21066) - #21159

Merged
objectstack-fleet[bot] merged 7 commits into
mainfrom
claude/issue-21066-memory-json-column-family
Oct 1, 2026
Merged

objectstack-fleet[bot] merged 7 commits into
mainfrom
claude/issue-21066-memory-json-column-family

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #21066
Clause-②: yes (narrowing)

On a field the object declares JSON-stored (a multiple: true field, tags / multiselect / checkboxes, or a structured-JSON type such as json), driver-memory now refuses the scalar-comparison family that driver-sql's where refuses: $eq, $ne, $gt, $gte, $lt, $lte, $between, $in, $nin and implicit equality, whatever the comparand, at any depth. The answer is INVALID_FILTER / 400 with the same message. The operator set and the sentence are read from @objectstack/core (JSON_COLUMN_INCOMPATIBLE_OPERATORS, jsonColumnOperatorRefusalText, homed by PR #21097). There is no third copy. $contains / $notContains (membership), $null, $exists and $empty keep answering.

What was wrong (H1, measured at origin/main 670680e93 through engine.find)

A real ObjectQL over InMemoryDriver, #21004's six rows (owners is a multiple: true lookup, tags is a tags field). Every row reproduces the card:

where before now
owners $eq 'u1' d1, d3 (per element) 400 INVALID_FILTER
owners $in ['u1','u9'] d1, d3 400
owners $nin ['u1','u9'] d2, d4, d5, d6 400
owners $gt 'u1' d1, d2, d3, d5 400
tags $gt 'red' d3 400
also: bare { owners: 'u1' }, $ne, $gte, $lt, $lte, $between, tags $eq, { owners: null }, $eq null, $ne null, $in [], $nin [] rows, per element 400
controls: owners $contains 'u1' / $notContains / $null / $exists / $empty, title $in rows unchanged rows

The engine hands the driver the operators as written, except $ne / $nin. Those arrive inside the spec's null-safe lowering ($and of $or of $null: true and the operator). The gate walks $and / $or / $not, so that shape is refused too.

The analytics face (MemoryAnalyticsService) answered the same per-element rows. Its SQL echo rendered owners = 'u1', which matches no row over the JSON text the SQL family stores. It now refuses in query() and generateSql() alike.

What changed

  • filter-refusal.ts: the shape gate (assertFilterConditionShape) takes an optional FilterFieldDeclarations (isJsonStoredField, reportWithheld). It has two arms. Implicit equality on a declared JSON-stored field is refused as =, bare. Any operator in the shared set is refused AFTER the existing comparand-shape rules, which is driver-sql's order (comparand gate, then column-type gate). So an array under $eq or a one-element $between still gets its own refusal first. jsonStoredFieldOperatorError builds the error from the shared text: this package's unsupportedFilterError envelope, with the withheld diagnostic handed to reportWithheld (prefixed At PATH:) before the throw.
  • memory-driver.ts: convertToMongoQuery passes this.filterFieldDeclarations(object). The population is isJsonStoredField, the predicate $contains already forks on (STRUCTURED_JSON_TYPES or isMultiValueField). So the fields where $contains asks membership are exactly the fields where the family is refused. The diagnostic goes to the driver's logger at warn, the level driver-sql uses for its withheld filter diagnostics. That keeps the message's "the full diagnostic is in the server log" true here.
  • memory-analytics.ts: normalizeFilters takes the cube. It judges a where key (a cube member) by the field it maps to on the cube's table, the same (table, field path) pair filterContainsTest reads. Its diagnostic goes to the analytics service's own logger.
  • .changeset/21066-memory-json-column-family-refusal.md: @objectstack/driver-memory minor, BREAKING banner, Clause-②: yes (narrowing), one ADR-0087 marker not-required (no-migration-prescription). No registered id covers a filter operator on a JSON-stored column. The one migration-registry entry that mentions json columns (cel-predicate-one-value-comparand-refused) is the CEL list-comparand surface, not this one.

Hypotheses, measured

Pin sweep

  • The ONE per-element pin the package carried on a declared field flipped: memory-20444-empty-operator.test.ts had { tags: { $empty: true, $ne: null } } giving r2. It is now a refusal pin (code + status + the shared message). The composition ($empty beside a has-a-value sibling on one multi-value field) is kept through $null: false, which answers r2. driver-sql/SQLite answers that row too, and refuses the $ne: null spelling with the same body (measured on the built driver).
  • memory-matcher-scalar-comparand-array-value.test.ts pins per-element answers on a column declared text. Those cells still hold, and a header note now says the population there is a scalar-declared column.
  • Repo-wide: only two tests outside this package bind the real driver (packages/runtime's two ruled consumers). Neither filters a JSON-stored field. No other INVALID_FILTER pin moves.

Tests (final head 13407b76f)

  • pnpm --filter @objectstack/driver-memory exec vitest run --maxWorkers=2: 70 files, 1703 passed. The first run after the implementation, before any test edit: 69 files, 1 red of 1613, the per-element pin flipped above.
  • pnpm --filter @objectstack/driver-memory run typecheck: exit 0. tsc --listFiles includes both edited test files.
  • New memory-21066-json-column-family-refusal.test.ts (89 tests):
    • the card's five rows;
    • every family member on owners / tags / meta (json);
    • ten shapes per field (bare, bare null, $eq null, $ne null, the engine's $ne lowering, $in [], $nin [], under $not, an $or branch after a holding one, $eq beside $contains);
    • count / findOne / updateMany / deleteMany refusing with the table untouched;
    • the withheld message plus the logged At filter.$or[1].owners.$gte: diagnostic;
    • comparand-first ordering;
    • eleven answered controls;
    • the declaration boundary (undeclared object, scalar-declared column, the gate with and without declarations);
    • the analytics face, query() and generateSql(), including the cube.member spelling, plus its log line and a $contains control.
  • Ablations (node scripts/ablation-replace.mjs, wrap mode, run at de1fef341; the two later merges touched no file in this package; each restore proven blob == HEAD with git diff HEAD empty). The subjects are this package's src, imported relatively, so no dist is involved:
    • A filterFieldDeclarations' predicate forced to () => false: 73 red / 37 green of 110 across the new file and 20444. The 17 green in the new file are exactly the answered controls, the declaration-boundary trio, the premise, the comparand-first case and the analytics $contains control.
    • B the implicit-equality arm disabled: 7 red, exactly the six bare cases and the analytics bare case.
    • C the operator arm disabled: 67 red, every operator-based refusal pin, the direct-gate test and the 20444 flip.
  • Gate union, derived with node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack (no paths; 7 changed paths, working tree clean) at 13407b76f: 60 derived, 60 run, every one exit 0. Reconciled with --ran: "60 derived famil(ies) accounted for — 60 run, 0 NOT-MEASURED (a DERIVED zero — all 60 recorded an exit code and none of them is 3)". The same 60 also ran all-zero at the previous merge head 5b75fe461. At de1fef341, check:dual-build-cjs-loads exited 3 (PREREQUISITE NOT MET, no dist yet); it measured on both later heads.
  • Driver conformance ledger (node scripts/check-driver-conformance.mjs), before and after: byte-identical. 50 covered cells, 0 DEBT, 0 exempt. The shared matrix has no JSON-column or multi-value case-set, so this invariant is held by the per-package pins, not the matrix.

Acceptance notes


Generated by Claude Code

claude added 6 commits October 1, 2026 09:02
…lared JSON-stored field

The shape gate in front of the query path and the analytics face now
refuses a scalar comparison (the shared JSON_COLUMN_INCOMPATIBLE_OPERATORS
set, and implicit equality) aimed at a field declared JSON-stored, with
INVALID_FILTER / 400 and the shared refusal text from @objectstack/core.
The withheld diagnostic goes to the face's logger at warn.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
…write doors and the analytics face

Flip the one per-element pin the suite carried ($ne beside $empty on a
tags field) to a refusal pin, keep its composition through $null: false,
and note the declared population on the scalar-column array suite.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
…family refusal

Also tag filterFieldDeclarations @internal: it is reachable from the
analytics face, not a consumer contract.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/driver-memory, touching 12 documentable anchor(s).

4 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/data-modeling/drivers.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/permissions/authentication.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/plugins/packages.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/protocol/objectql/query-syntax.mdx (via InMemoryDriver (symbol, a top-level class))

⛔ 6 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/implementation-status.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v14.mdx (via generateSql (symbol, a method of class MemoryAnalyticsService))
  • content/docs/releases/v16.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v17/17-0.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v17/17-3.mdx (via InMemoryDriver (symbol, a top-level class))
  • content/docs/releases/v17/17-5.mdx (via generateSql (symbol, a method of class MemoryAnalyticsService))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 9 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 2488b98b48f51e2a1bc5b5e50fc1c206c9565291 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from a53e51e3e00813bb5881a9194e1a82df6d53935b — the merge of head 2eab11e409f28a6d3be235dfabbd53e50bc124ef into base 2488b98b48f51e2a1bc5b5e50fc1c206c9565291, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin a53e51e3e00813bb5881a9194e1a82df6d53935b && git checkout a53e51e3e00813bb5881a9194e1a82df6d53935b
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2488b98b48f51e2a1bc5b5e50fc1c206c9565291 2eab11e409f28a6d3be235dfabbd53e50bc124ef && git checkout -B drift-repro 2488b98b48f51e2a1bc5b5e50fc1c206c9565291 && git merge --no-ff 2eab11e409f28a6d3be235dfabbd53e50bc124ef

node scripts/docs-audit/affected-docs.mjs --json 2488b98b48f51e2a1bc5b5e50fc1c206c9565291

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 2488b98b48f51e2a1bc5b5e50fc1c206c9565291 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

…narrowing)

The seat's answer on the card: InMemoryDriver.filterFieldDeclarations is in
the published .d.ts, so the declaration follows the precedent the analogous
filterContainsTest set. Level and ADR-0087 marker unchanged.

Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 2eab11e409f28a6d3be235dfabbd53e50bc124ef
Local-runs: none

PR #21159 on card #21066, judged against triage's direction 5925785069, the claim 5927990211, the report 5929893869 and the seat's answer 5929927010. Inputs: the card thread, the PR body and file list, the net diff against the merge-base with main (58a77dbde, 7 files, +565 / -10), and the check-runs on the head. Nothing built, run or re-run.

① Derived judgments

Every accept-set and public-surface change the diff implies, each named right or wrong:

  1. The query path narrows on a declared JSON-stored field — right. convertToMongoQuery hands this.filterFieldDeclarations(object) to the one shape gate, and that method is the filter entry of find, count, updateMany, deleteMany, aggregate and also distinct (six call sites on the head; findOne is pinned refusing in the new suite). The gate has two arms. Implicit equality (the !isFilterNode branch, so any scalar, null or Date comparand) is refused as bare =. An operator in the shared set is refused after every comparand-shape rule in the loop, with no continue ahead of it. The walk recurses $and / $or / $not, so the depth claim holds; the engine's null-safe lowering of $ne / $nin (an $or holding $null: true and the operator) reaches the operator arm on its second branch.
  2. The set and the sentence are read, not copied — right. filter-refusal.ts imports JSON_COLUMN_INCOMPATIBLE_OPERATORS and jsonColumnOperatorRefusalText from @objectstack/core; no literal of either exists in the package. Byte-identity with driver-sql: both drivers build the error as unsupportedFilterError(message) over the same core text (driver-sql's withheldFilterError is that call plus non-enumerable symbol carriers), so message, code INVALID_FILTER and status 400 are the same bytes. The new suite asserts the message equal to the imported text, and derives its FAMILY from the shared set intersected with SUPPORTED_FIELD_OPERATORS with a floor of the nine $-spellings — the forward pin for [finding] $startsWith / $icontains on a multi-valued lookup answer 500 on PostgreSQL and a wrong count on SQLite: the text operators other than $contains reach a JSON column unrefused and unruled #21009 the report describes.
  3. The predicate for "declared JSON-stored" — right, and the same reading as driver-sql's gate less two recorded members. Memory: isJsonStoredField is STRUCTURED_JSON_TYPES.has(type) || isMultiValueField(shape) over valueShapes, which only syncSchema fills (one write site). driver-sql: jsonFields is filled by JSON_COLUMN_TYPES.has(type) || isMultiValuedColumn(type, field), where JSON_COLUMN_TYPES is STRUCTURED_JSON_TYPES plus MULTI_OPTION_TYPES plus the driver-internal object / array aliases, and a single-value media column is asked per instance (mediaColumnIsJson). The two omissions are documented on the method and are not declarations an authored object can carry into this driver. An object never synced answers false and is not judged; SqlDriver.isJsonColumn answers false for a table with no jsonFields entry and assertOperatorAppliesToColumn returns on it. Same answer. A column declared text holding an array is not judged either; the memory-matcher-scalar-comparand-array-value pins hold on that population and its new header note says so.
  4. Order: comparand gate, then column-type gate — right, matching driver-sql at all three of its positions (the bare loop, the operator map and the reduction walk each run assertCompilableComparand before assertOperatorAppliesToColumn). Memory's loop refuses the array comparand, the malformed $between, the non-boolean $null / $exists / $empty, the array under a single-value operator and the $icontains / $like shapes first; pinned by the comparand-first test.
  5. The controls keep answering — right. $contains, $notContains, $startsWith, $endsWith, $icontains, $null, $exists and $empty are absent from the shared set by design; eleven answered controls are pinned on the query path and $contains on the analytics face. Spec FILTER_OPERATORS carries no equality or ordering alias beyond the nine $-spellings, and the set's bare infix spellings are refused by the vocabulary gate before this one (a refusal, in that gate's words; pre-existing).
  6. The analytics face narrows the same way — right. normalizeFilters is the face's only filter entry (no cube-style filters list is read anywhere in the file). The gate runs on the lowered where with declarations resolved through extractTableName(cube.sql) and resolveFieldPath(cube, member), the same (table, field path) pair filterContainsTest is asked, so the cube.member spelling is judged and pinned. Lowering keeps every family member reachable: rule 3 leaves $ne: null alone and wraps a non-null $ne / $nin in an $or; rule 1 splits $between into $gte / $lte, both in the set. query() and generateSql() share the entry, so the SQL echo refuses too. One corner, pre-existing and not of this diff: this face lowers type-blind, so rule 2 turns a lone $lte whose comparand is the last supported day (9999-12-31) into $null: false and answers it, where find() refuses $lte on the same field. A sentinel comparand on a list field; noted for the seat, not escalated.
  7. Disclosure — right, in the shared posture, with one recorded difference. The message withholds the field and the operator; the diagnostic, prefixed with the filter position, goes to the driver's or the service's own logger at warn, the level logWithheldFilterDiagnostic writes at in driver-sql, and the log line has the same shape. The seam is new to this package, as the report says. The difference: driver-sql resolves driver-sql: the #7929 withhold covers the cross-field family only — every other INVALID_FILTER refusal still names the target field, which is admin-authored on a read-scope predicate #8197 provenance and discloses the diagnostic on the wire for a subtree a boundary vouched author (plugin-security, plugin-sharing and service-analytics mark the caller's own where); this driver has no provenance read, withholds unconditionally, and carries no withheldFilterDiagnosticOf twin. The fail-closed direction, outside the direction's scope; noted.
  8. Public surface — one method, declared. InMemoryDriver.filterFieldDeclarations is public on the exported class and so in the published .d.ts; FilterFieldDeclarations is not re-exported from src/index.ts (checked). Declared in the changeset as yes (narrowing), as the seat ruled (5929927010) on [finding] driver-memory answers $contains on a stored array by substring per element (u1 matches a row storing u10), where the SQL drivers answer membership; the spec docblock records the gap against a card that answers 404 #20874's grading of filterContainsTest. Right.
  9. Edits outside the claim's file list — justified. memory-driver.ts (the declarations and the log sink) and memory-analytics.ts (the second gate caller) are the plumbing the gate needs to see a declaration; the single-writer path check on the head is success.
  10. The pin flip — flipped, not deleted. memory-20444-empty-operator.test.ts keeps the composition cell through $null: false giving r2, and adds a refusal pin for the $ne: null spelling asserting code, status and the shared message.
  11. Left alone, and rightly: the AST comparison-node door. No non-test emitter of a type: 'comparison' node exists outside driver-memory (swept on the head), so the per-element answer there is reachable by a direct driver call only, as the report says.
  12. A residual of the same class, found here. A no-operator object comparand on a declared JSON-stored field ({ meta: { k: 'a' } } on a json column, { owners: { k: 'a' } } on a multi lookup) is not judged by this gate: the bare arm sits inside !isFilterNode(spec) and the operator arm needs a $ key, so on a direct driver call this driver still answers it by deep equality, as it did before this PR, where driver-sql refuses the same input at its comparand gate. Through the engine, [finding] a no-operator object under a lookup, master_detail or json field answers per driver: the declared nested-relation filter returns no rows on memory and a 400 on SQL, and a json object comparand deep-equals on memory and is refused on SQL #20745's no-operator-object door refuses the json and relation kinds before any driver is asked, so public-door reach is none. Not in this card's family (the operators plus bare scalar equality); not a defect of this diff. For the seat to note beside the AST door, or file.

② Semver level

  • .changeset/21066-memory-json-column-family-refusal.md: @objectstack/driver-memory minor (from 17.5.0), a ! summary, a BREAKING banner naming every door that narrows, and exactly one ADR-0087 marker, not-required (no-migration-prescription), with its closing of the other categories. What the diff publishes is an accept-set narrowing on a released package plus one public method: (narrowing) is BREAKING under AGENTS.md's changeset rule, BREAKING ships minor in the launch window (check-changeset-no-major, the rule a dozen sibling changesets cite), and no authorable key, export or stored shape moves, so the marker's arm is right. Check Changeset, the job that runs check-adr-0087-registration against the merge-base, is success on the head.
  • Clause-②: yes (narrowing). The changeset's line and the PR body's line 2 both carry it, as the seat's answer prescribed. Right.
  • One defect, body-only: two sentences of the PR body still read no (narrowing) — the "What changed" bullet on the changeset, and the first "Acceptance notes" bullet ("carries the claim's no (narrowing) line verbatim and leaves the grading to the seat"). They contradict line 2 and the changeset. A PR-body edit, no new head; the seat makes it before the queue reads the body.

③ Boundary flags

Implemented-by: claude/issue-21066-memory-json-column-family
Reviewed-by: session_01Ujdtvqs7ree7WyQmEDwEnG

VERDICT: PASS


Generated by Claude Code

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review October 1, 2026 15:07
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge October 1, 2026 15:08
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Oct 1, 2026
Merged via the queue into main with commit 45ce12a Oct 1, 2026
50 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-21066-memory-json-column-family branch October 1, 2026 15:28
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Oct 7, 2026
…ntains and $like / $ilike as it refuses the equality family (objectstack-ai#21165)

Fixes objectstack-ai#21009
Clause-②: no (narrowing)

Patch round 1 executes the seat answer on objectstack-ai#21009 (5930243637):

- **Q1 → A.** The `$search` expander matches a multi-valued field by
membership, in this PR.
- **Q2 → A.** The shared sentence stays byte-identical; its rewrite
belongs to objectstack-ai#21067.
- **Q3 → A.** The ADR-0087 disposition is `not-required
(no-migration-prescription)`.

Head `143f4ccd8f` merges `main` at `d34aa58a2`.

## What changes

**`@objectstack/core`.** `JSON_COLUMN_INCOMPATIBLE_OPERATORS` is the one
set `driver-sql`'s `where` and objectql's per-aggregation `filter` both
read since objectstack-ai#21097. It gains the text operators other than the membership
pair:

- `$startsWith`, `$endsWith`, `$icontains`;
- the staged pattern pair `$like` / `$ilike`, which `driver-sql` answers
ahead of `FILTER_OPERATORS`.

On a JSON-stored column each now gets the `400` the equality family
already gets there:

- `INVALID_FILTER`, with the same withheld message, byte for byte;
- the operator and the field named in the server-log diagnostic, and in
the message for a filter marked as the caller's own.

Nothing else on that gate moves:

- `$contains` / `$notContains` (membership), `$exists`, `$null` and
`$empty` answer as before.
- No membership reading is invented for a prefix, suffix or case-folded
test.
- It is one edit to the shared set, with no second copy.
- `sql-driver.ts`, `having-filter.ts`, `remote-transport.ts` and
`driver-memory` are untouched.

**`@objectstack/objectql`.** The search expander (`search-filter.ts`,
`fieldClausesForTerm`) matches a field the object declares multi-valued
(`isMultiValueField`) by membership:

- a term matching option labels becomes one `$contains` per matched
option value, replacing the `$in` that is refused there;
- any other term, or any term on a field with no options (`tags`, a
multi-valued lookup), becomes `$contains` of the term.

The declaration is read from the field map the engine already passes in:
each entry is the object's whole field definition, `multiple` included.
No spec type moves. Scalar fields keep their clauses.

The visible cost: to hit a multi-valued field, a term must now equal one
of its members or match one of its option labels. SQLite used to match
substrings of the serialized array as well, so `wood` found a row tagged
`redwood`; it no longer does.

## Measured, before (`origin/main` `7a606a9a3`) and after

### The `where` / per-aggregation filter

Measured through `POST /api/v1/data/:object/query` on SQLite and a
private PostgreSQL 16.14. The fixture has six rows: `owners` is a
multi-value lookup (`d1` holds `u1, u2`; `d3` holds `u3, u1`; `d5` holds
only `u10`).

| filter on `owners` | SQLite `where`, before | PostgreSQL `where`,
before | per-aggregation `filter`, before | after |
|:--|:--|:--|:--|:--|
| `$startsWith: '['` | 5 rows, every row with a value | 500
`DATABASE_ERROR` | m = 0 | 400 on both faces and both dialects |
| `$startsWith: 'u1'` | 0 rows, though two rows hold `u1` | 500 | m = 0
| 400 |
| `$endsWith: ']'` | 5 rows | 500 | m = 0 | 400 |
| `$icontains: 'U1'` | `d1`, `d3`, `d5` (`d5` holds only `u10`) | 500 |
m = 0 | 400 |
| `$like` / `$ilike` | `d1`, `d3`, `d5` | 500 | 400, as an operator that
face does not evaluate | `where` 400 (this refusal); per-aggregation
unchanged |

- A `tags` field gave the same results.
- A `json` field was already refused all seven text operators at the
engine's declared-type door, which still answers first.
- The scalar `title` control answered the same rows before and after.

### `$search`

The search was measured through the same route, with `search`, on two
objects:

- the shape of `examples/app-todo`'s `todo_task.tags` (`select`,
`multiple: true`, in the auto-default set);
- a declared searchable `tags` field.

| search | `main`, SQLite / PostgreSQL (measured) | this branch before
the expander fix | now, SQLite and PostgreSQL (pinned) |
|:--|:--|:--|:--|
| task, a label term (`Important`, `quick`) | 400 (the `$in`) / 400 |
400 | 200, the rows holding the matched value |
| task, a term only the scalar subject holds (`meeting`) | 200 / 500 |
400 | 200, by the subject |
| note, a member (`red`) | 200, `n1` and `n2` / 500 | 400 | 200, `n1`
(member) and `n2` (scalar title) |
| task, a raw member (`quick_win`); note, a member (`redwood`) | not
measured on `main` | — | 200, the rows holding it |
| task, a non-member (`zebra`); note, a substring of a member (`wood`) |
not measured on `main` | — | 200, no rows |

No term answers 400 or 500 any more. The scalar controls (a `select`
label, a text fold) are unchanged.

### H2, H3, H4

**What a caller reads (H2).** For `$startsWith` on `owners`, the REST
body is cut at the envelope's 500 characters:

```text
A constraint in this filter WAS NOT APPLIED: it aims a scalar comparison operator at a field this driver stores as a JSON TEXT column (e.g. ["a","b"]), and such an operator compares that whole serialized text against a single value — it can never equal one member. Use "$contains" for membership ({ "FIELD": { "$contains": "a" } }), or an $or of "$contains" for any-of ({ "$or": [{ "FIELD": { "$contains": "a" } }, { "FIELD": { "$contains": "b" } }] }). Refused rather than compiled because the answ…
```

Per the seat answer, it stays byte-identical here, and objectstack-ai#21067 owns the
rewrite.

**Turso remote (H3).** `RemoteTransport.buildWhereSQL` compiles its own
filters and has no JSON-column gate at all, for the equality family
included. This PR leaves it alone, and it is reported for filing.

**driver-memory (H4).** It answers each text operator per element. It is
unchanged here; once objectstack-ai#21066's shape gate reads this set, it refuses them
too. The seat answer orders this PR ahead of PR objectstack-ai#21159.

## Pins

- `core` `json-column-operator-refusal.test.ts`:
  - the set, member for member;
  - every text operator except the membership pair is in it;
- each of the five reads the equality family's message (its SHA-256 and
length).
- `driver-sql`
`sql-driver-21009-json-column-text-operator-refusal.test.ts` (new) is a
dialect-cell suite: SQLite always, PostgreSQL and MySQL where
provisioned, and the Temporal Conformance job provisions both. On a
multi-value lookup and a `tags` column, each of the five gets:
  - `INVALID_FILTER` / `400` through `find` and `count`;
  - the operator and field named to an author;
  - for anyone else, the equality family's message, byte for byte.

The same file pins the scalar control (exact rows) and membership still
answering.
- `driver-sql` `sql-driver-json-column-operator-refusal.test.ts`: the
text family moves from the keep-working list to the refused list, on
every face.
- `driver-sql` `sql-driver-17590-…` and `sql-driver-17343-…` held the
text family "unmoved" or "compiling" on a JSON column. They now pin the
refusal on all three compilers, with the scalar column unmoved.
- `objectql` `engine-aggregate-filter-json-column-refusal.test.ts`: the
text family on the per-aggregation filter and its per-row floor; a
structured-JSON field still meets the declared-type door first.
- `objectql` `search-filter.test.ts`: membership clauses for a
multi-valued `select` (label, partial label, no label), for `tags` and
for a multi-valued lookup, with the scalar `select` and text controls.
- `rest` `data-search-multi-valued-membership.test.ts` (new) runs the
table above through `POST /api/v1/data/:object/query` with `search`, on
SQLite and PostgreSQL, with MySQL where provisioned.
- `rest` `aggregation-filter-json-column-refusal.test.ts`: the text
family on both faces, with the per-aggregation body equal to the `where`
twin's.
- The dogfood `search-conformance.ledger.ts` summary now names
membership for a multi-valued field. That half's HTTP proof is the REST
file above, because no showcase object carries one in its search set.

## Reverse verification

Both fixes were committed before each ablation. Each restore leg proved
the file's blob equal to HEAD and an empty `git diff HEAD`.

**The core set.** The ablation deleted the five new members (blob
`8799778c` to `cbf406f9`), rebuilt, and the preflight found the members
`--absent`.

| suite | with the members deleted | after the restore |
|:--|:--|:--|
| core | 2 failed of 8 | 8 passed |
| driver-sql | 82 failed of 220 | 219 passed, 1 skipped |
| objectql | 13 failed of 106 | 106 passed |
| REST | 20 failed of 195 | 130 passed, 65 skipped (MySQL) |

**The expander.** The membership branch was disabled (`&& term ===
'ablated-21009'`, blob `44a09d96` to `61401d11`), objectql rebuilt, and
the preflight found the marker present.

| suite | with the branch disabled | after the restore (rebuilt, marker
`--absent`) |
|:--|:--|:--|
| objectql `search-filter.test.ts` | 3 failed of 20 | 20 passed |
| REST search file | 20 failed of 22: every search case on SQLite and
PostgreSQL answered 400 `INVALID_FILTER` | 22 passed, 11 skipped (MySQL)
|

Both moved in the expected direction: the pins turned red.

## Tests and gates (head `143f4ccd8f`)

| suite | result |
|:--|:--|
| driver-sql, full | 4244 passed, 96 skipped (SQLite and PostgreSQL;
server at Asia/Shanghai, process at America/New_York) |
| driver-turso, full | 2218 passed, 33 skipped |
| core pins | 8 passed |
| objectql pins | 126 passed |
| REST pins | 152 passed, 76 skipped (SQLite and PostgreSQL) |
| ADR-0061 dogfood proof (`showcase-search.dogfood.test.ts`) and the
search-conformance ledger | 7 passed, exit 0 |

The full suites at `76d2fd5e8` (`main` `bafb8c949` merged; the last
merge brought only `sql-driver.ts`'s sequence region and `driver-turso`
into these packages) were:

| suite | result |
|:--|:--|
| objectql `local` | 7070 passed |
| REST (SQLite) | 4970 passed, 302 skipped |

- Typecheck passed for core, driver-sql, objectql, REST and dogfood.
**MySQL: NOT MEASURED locally** (no server); CI's temporal job runs it.
- Gates: `dispatch-gates --commands` was derived at `143f4ccd8f` with no
paths. It named 70 commands, and 69 exited 0.
`check:dual-build-cjs-loads` exited 3 (PREREQUISITE NOT MET, a
whole-workspace build): NOT MEASURED. `--ran` reconciled 70 derived, 69
run, 1 NOT MEASURED, 0 UNRUN. The derivation was stale by one `main`
commit, a production-dependency bump (`f3b16fc2f`) that changes
`package.json` only.
- Driver conformance: 50 / 0 / 0 before (`7a606a9a3`) and after
(`143f4ccd8f`).
- Lint was narrowed to the 12 changed `.ts` files. The proof has three
parts:
  1. each file resolves a config under `eslint --print-config`;
  2. `--format json` reports 12 files, 0 errors and 0 warnings;
3. `eslint.config.mjs` never enables type-aware linting, so no untouched
file's verdict can move.
- The changeset is `@objectstack/core` and `@objectstack/objectql`, both
`minor`, BREAKING. It states the search cost. Its ADR-0087 disposition
is `not-required (no-migration-prescription)`.

## Acceptance notes

- `SqlDriver.isNonTextColumn`'s docblock says "a text operator is legal
against a JSON column". That now holds for the membership pair only.
Carrier: none; it is outside this claim's surface.
- The registered migration entry
`filter-text-operator-declared-type-refused` names `multiselect` /
`checkboxes` / `tags` and lookup ids as fields that must keep answering
exactly as before. That over-claims once this lands. The seat records it
as a spec-lane wording finding, filed at landing.
- A view-filter builder offering "starts with" or "ends with" on a
multi-valued field now gets a loud 400. Carrier: the objectui filter
builder.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/l tests tooling

Projects

None yet

2 participants