Skip to content

Commit a11faee

Browse files
fix(objectql)!: a per-aggregation filter refuses a scalar comparison on a declared JSON-stored field, in where's words (#21097)
Fixes #21007 Clause-②: yes (widening) A per-aggregation `filter` now refuses a scalar comparison on a declared JSON-stored field (`$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$nin`, implicit equality) with `INVALID_FILTER` / 400, in the words `where` refuses the same filter in. It no longer counts rows the stored arrays cannot support. The operator set and the refusal text move from `driver-sql` to `@objectstack/core`, byte for byte, so both faces read one set and one sentence. **Clause-② has two halves.** It is `yes (widening)` because `@objectstack/core`'s root gains three exports (`JSON_COLUMN_INCOMPATIBLE_OPERATORS`, `jsonColumnOperatorRefusalText` and its return type `JsonColumnOperatorRefusalText`), and `applyInMemoryAggregation` gains an optional trailing `reportWithheld` parameter. It also narrows: `@objectstack/objectql` refuses queries it used to answer 200, at `engine.aggregate` and at the published `applyInMemoryAggregation` given a field map. The changeset therefore carries `minor` for objectql with a BREAKING banner and one ADR-0087 marker, `minor` for core, and `patch` for driver-sql, whose output is unchanged. The seat answer on the card (`## Seat answer — #21007`, comment 5924546829) amended the claim to this surface and this `Clause-②`. ## What was wrong (measured before this change, `d1f8ce865`) `POST /api/v1/data/:object/query` on SQLite and a live PostgreSQL 16.14, over the card's six rows (`owners` is a `multiple: true` lookup, and `d1` and `d3` hold `u1`). Both dialects answered identically: | filter | `where` twin | per-aggregation `m` before | now | |:--|:--|:--|:--| | `owners $in ['u1','u9']` (the card) | 400 `INVALID_FILTER` | 0 | 400, same body | | `owners $nin ['u1','u9']` (the card) | 400 | 6, with `d1` and `d3` counted | 400, same body | | `owners $eq 'u1'` / `{ owners: 'u1' }` | 400 | 0 | 400 | | `owners $ne` / `$gt` / `$lte` / `$between` | 400 | 6 / 4 / 1 / 5 | 400 | | `tags $eq 'red'` | 400 | 1 (`['red']` loosely `==` `'red'`) | 400 | | `meta` (json) `$eq` / `$in` | 400 | 0 / 0 | 400 | | `owners $contains 'u1'` (the prescribed spelling) | 2 | 2 | 2 (unchanged) | | `title $in` / `$nin` / `$eq` (controls) | 2 / 4 / 1 | 2 / 4 / 1 | unchanged | ## What changed - **`@objectstack/core`:** a new `src/utils/json-column-operator-refusal.ts`, exported from the root beside `temporal-storage-form.js`. The `export *` publishes three names: `JSON_COLUMN_INCOMPATIBLE_OPERATORS` (driver-sql's 22 spellings, member for member), `jsonColumnOperatorRefusalText(field, op, bare)`, and its return type `JsonColumnOperatorRefusalText` (`{ message, diagnostic }`). These are the two strings `jsonColumnOperatorError` built, and nothing else. Each face keeps its own error constructor: driver-sql keeps its #8220 provenance seam, and objectql keeps its ADR-0112 envelope. - **`driver-sql`** (`sql-driver.ts`): the module-private set and the two template strings are gone. `jsonColumnOperatorError` keeps its name and signature, and now calls the core builder (hunk at `:3298`). One import line (with its comment) sits at `:153`–`:156`, after the top import block. It is outside the declared `:3376`–`:3460` region on purpose, so as not to touch the `@objectstack/core` import block that #20988 and #20987 edit. `assertOperatorAppliesToColumn` (in #20988's former region) is untouched: it reads the imported set under the same name. - **`objectql`** (`having-filter.ts`): `assertAggregationFilterIsEvaluable` gains `assertAggregationFilterSparesJsonStoredFields`. It runs once on the filter after the reference rule, against `declaredJsonStoredFields(declared.fields)`, and before any driver is asked for a row, so an empty table refuses too (#20122's rule). It walks `$and` / `$or` / `$not` and refuses implicit equality (reported as `=`, bare, as driver-sql does) and every operator in the shared set, whatever the comparand (`null` and `[]` included). The withheld message is thrown, and the diagnostic, with the aggregation position, goes to `reportWithheld`, as #20148 does. `$contains` / `$notContains` / `$exists` / `$null` / `$empty` keep answering. `checkCondition` carries the same refusal above its no-value exit, but only as a backstop for a row that reaches the arm. Its reach is the row's: an empty row set never gets there, and spec lowering rule 3 puts a `$null` arm ahead of every negation that the walker's `$or` short-circuit takes first. The gate is the complete door, and the docblocks now say so (round 2, from the review's ①.5). - **`objectql`** (`in-memory-aggregation.ts`, round 2, the review's F10 (b)): the published `applyInMemoryAggregation(rows, ast, timezone, fields, reportWithheld?)` now calls the same `assertAggregationFilterSparesJsonStoredFields` (exported from `having-filter.ts`, not from the package root) once per `aggregations[i].filter` when it is handed `fields`, before any row is judged. Before this, a direct caller reached only the backstop and got a row-dependent answer. This entry point holds no logger, so the closest seam is a new optional trailing `reportWithheld(diagnostic)`, which receives the field, operator and position; without it the diagnostic is dropped and the 400 is unchanged. `engine.aggregate` passes none, because it has already judged and logged the same filter. The gate's declaration parameter is narrowed to just the `fields` and `reportWithheld` members of `AggregationFilterDeclaration`, since `object` is the reference rule's. - **`objectql`** (`engine.ts`): one comment block and the `reportWithheld` log line at `assertAggregationFilterIsEvaluable`'s call site. The log line now reads "as it is for the same refusal in a where" instead of "…cross-field comparison…", since it carries two refusals now. ## Measured findings behind the shape (H1–H5) - **H1, the premise, holds.** `SET_MEMBER_DESCRIPTION`, the `$in` / `$nin` entries of `FILTER_OPERATORS` and the `$contains` docblock give no per-element reading. driver-sql's `where` refuses (it does not answer membership), so triage's "membership, as driver-sql does" misread it. - **H2, the set.** All ten operators, plus null and empty-list comparands, were answered with a wrong count; none was already refused. The bare infix spellings (`in`, `=`, `nin`) are refused earlier at both positions by the nested-relation door, so the evaluator only meets the `$` forms. - **H3, the home.** None existed; per the seat answer, the home is `@objectstack/core`. - **H4, where it fires.** The engine's in-memory lowering is the only evaluator of `aggregations[i].filter`: every driver's aggregate face refuses a per-aggregation filter 501, and the analytics ObjectQL strategy hands measure filters to `engine.aggregate`. - **H5, `having`.** After #21037 (landed, merged here), `min` / `max` over a multi-valued field is refused `INVALID_FIELD` at the aggregate door. Measured on InMemoryDriver at the merged head: `max(owners)` with or without `having` gives 400 `INVALID_FIELD`. So `having` cannot meet a JSON-stored column, and it is left alone. ## Tests Round 1 numbers were read at `6e541b101`; round 2 numbers are marked with the head `f66bed950` (main merged). - `@objectstack/core` `json-column-operator-refusal.test.ts`: 6 passed. It pins the set member for member, and the message and three diagnostics by SHA-256 and length against what driver-sql printed at `8f784959c`. Hashes avoid a third literal copy of the sentence. - `driver-sql` `sql-driver-json-column-refusal-shared-text.test.ts`, run beside the existing JSON-column, compile-refusal-seam and provenance suites: 248 passed. It checks every `FILTER_OPERATORS` member against the shared set: the driver's thrown message and withheld diagnostic equal the core builder's output. - **Byte identity of the move.** A scratch capture through the built driver-sql (`SqlDriver` over SQLite) covered all 22 spellings plus bare equality, unmarked and author-marked, message and diagnostic, 46 entries. Before (`8f784959c`) and after: `cmp` identical, sha256 `dbcf32f5…534b2` on both. driver-sql's `dist` no longer contains the sentence. - `objectql` `engine-aggregate-filter-json-column-refusal.test.ts` (engine-level cell over the `find()` read shape): 74 passed. It covers 18 family cases on each of `owners`, `tags` and `meta` (`code`, `status`, the `$contains` / `$or` prescription, the field absent from the message, field and operator in the logged diagnostic, and the driver never asked for a row), an empty table (pure and grouped), the logged position, 11 answered cases (membership, null predicates, `title` controls) and the per-row floor. - `rest` `aggregation-filter-json-column-refusal.test.ts`: 52 per cell. SQLite passes and a live PostgreSQL 16.14 passes locally; MySQL is a named skip. Every family case asserts that the per-aggregation 400 body's `error` is **the same string** as its `where` twin's. #21004's `aggregation-filter-array-membership.test.ts` still passes beside it. - **Full suites.** Read at merge `1a226419e`: objectql local 6948 passed, rest local 5072 passed / 247 skipped, core 1809 passed. Read before the first merge: driver-sql 3285 passed / 188 skipped. `typecheck` passed for core, driver-sql, objectql and rest. At head `6e541b101`: core, driver-sql (refusal suites), objectql `engine-aggregate*` (571 passed) and rest `aggregation-filter*` (150 passed with PostgreSQL) re-ran green. - **Ablation A: the engine gate call deleted** (`ablation-replace`, plus a rebuilt objectql `dist`, plus `ablation-dist-preflight --absent`): - objectql suite: 56 of 74 red. The 54 family cases, the empty table and the logged position failed; the 11 answered cases and the 7 floor cases stayed green. - rest suite: 92 of 104 red (46 per dialect). Populated `owners` / `tags` cases still got a 400 from the per-row floor, without the logged diagnostic. `meta` negations (`$ne`, `$nin`, `$nin []`, `$not $in`, where `meta` is null on every row) answered 200 `{ n: 6, m: 6 }`; the mechanism was not traced. The empty table answered 200. - Restored: blob equals HEAD, `git diff HEAD` empty, objectql rebuilt, preflight shows the marker present in 4 dist files with a clean tree, and both suites green again (74 and 104). - **Ablation B: the per-row operator floor replaced by a no-op** (src, engine suite): 5 red, the 4 operator floor cases and the no-value row; restored blob equals HEAD. - **Round 2: the direct-caller pins** (`engine-aggregate-filter-json-column-refusal.test.ts`, 92 passed). For `meta` (json) `$ne`, `$nin` and `$not $in`, each on four cells: an empty row set, an empty grouped row set, `meta` null in every row with the filter as spec `lowerFilterCondition` lowers it (the shape that carries the `$null` arm), and the same rows with the filter as written. Each must refuse 400 `INVALID_FILTER` with exactly `engine.aggregate`'s message, and hand the diagnostic (field, operator, `At aggregations[1].filter.…`) to `reportWithheld`. Also pinned: no reporter means the same refusal; no field map means nothing judged (`m: 0`, as before); and `$contains` still answers. - **Ablation C: the new `applyInMemoryAggregation` call deleted** (`ablation-replace`, anchor 1 to 0, blob `c65412761a90` to `f530761c0559`): 13 of 92 red. - The 9 empty, empty-grouped and lowered-null cells, plus the no-reporter case, answered instead of refusing. That is the backstop's 200. - The 3 as-written null-row cells were refused by the backstop but with no diagnostic reported. - Restored: blob equals HEAD `c65412761a90`, `git diff HEAD` empty, 92 passed again. - **Round 2 at `f66bed950`:** objectql local full suite 7008 passed (356 files), rest `aggregation-filter*` 150 passed / 61 skipped with a live PostgreSQL 16.14, core refusal pin 6 passed, driver-sql refusal pins 141 passed. - **Driver conformance ledger:** 50 covered cells, 0 in the DEBT ledger, 0 exempt, both before and after, in both rounds. ## Gates - `node scripts/pm/dispatch-gates.mjs --commands` (no paths) derived 70 families at `6e541b101`. - **68 ran, exit 0.** Among them: `check:adr-0087-registration`, `check:changeset-no-major`, `check:engine-double-contract`, `check:nul-bytes`, `check:doc-authoring`, `check:driver-conformance`, `check:driver-memory-census`, `check:query-options-erasure` and `check:test-source-alias`. - **2 NOT MEASURED (exit 3, prerequisite not met):** `check:dual-build-cjs-loads` and `check:type-check-debt`. Both need the whole workspace built; two attempts at that build timed out in the shared verify-lock queue. CI's lint job builds first. - `--ran` reconciliation: 70 derived, 68 run, 2 NOT MEASURED, 0 unrun. - **Round 2 at `f66bed950`:** re-derived with no paths, the same 70 families; 68 ran with exit 0 and the same 2 were NOT MEASURED (exit 3). `--ran`: 70 derived, 68 run, 2 NOT MEASURED, 0 unrun. `typecheck` passed for core and objectql. Narrowed lint: 10 changed `.ts` files, 0 errors, 0 warnings. - An earlier run caught one real finding, fixed in `e7bd7f667` ("type the shared-text pin's find options"): `check:query-options-erasure`'s test surface grew 236 to 237 because of an `as any` on a `find` options bag in the new driver-sql test. - **Lint, narrowed and declared:** `eslint --no-inline-config --format json` over the 9 changed `.ts` files reports 9 files, 0 errors, 0 warnings. `eslint.config.mjs` sets no `parserOptions.project` and registers no typed rule, so linting is not type-aware and this diff cannot move a verdict on an untouched file. The full `pnpm lint` is CI's. ## Acceptance notes - **Round 2, from the at-tier review (5926186339).** - F10 (b): `applyInMemoryAggregation` is gated (above). - F10 (a): the backstop docblocks are corrected. - F12 and ①.6: the export count is three, and the changeset's "Who is affected" names `applyInMemoryAggregation` direct callers and the new optional `reportWithheld`. - **The "flip the `m: 0` / `m: 6` pins" step had nothing to flip.** No suite on `main` pinned a per-aggregation `$in` / `$nin` count on a JSON-stored field (#21004's two suites pin only `$contains` / `$notContains`). The full objectql, rest and driver-sql runs found no other pin that this change turns. The refusal pins are new files beside #21004's. - **A `{ $field }` comparand on a JSON-stored field** (`{ owners: { $eq: { $field: 'title' } } }`) is refused by this gate in the JSON-column words. driver-sql's `where` refuses it through its cross-field class rule, in that rule's words. Both answers are `INVALID_FILTER` / 400 with the field withheld, so the two faces disagree only on which sentence they print. - **The REST envelope truncates the shared message at 500 characters** on both faces, so it ends "…because the answ…". That is unchanged here by direction, and filed separately by the seat. - **Findings for the seat, not filed here:** - driver-memory's `where` answers the family per element on a multi-valued field, while the SQL family refuses it (engine-level measurement). The seat files it. - service-analytics' native measure-filter compiler has no JSON-column gate (read at source, not measured). Carrier: #20987. --- _Generated by [Claude Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 9939854 commit a11faee

11 files changed

Lines changed: 1124 additions & 94 deletions
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
"@objectstack/objectql": minor
3+
"@objectstack/core": minor
4+
"@objectstack/driver-sql": patch
5+
---
6+
7+
fix(objectql)!: a per-aggregation `filter` refuses `$in` / `$nin` / `$eq` / `$ne` / an ordering / `$between` / implicit equality on a declared JSON-stored field with `INVALID_FILTER` / 400, in the words `where` refuses them in, instead of counting rows the stored arrays cannot support
8+
9+
Clause-②: yes (widening)
10+
11+
<!-- adr-0087: not-required (no-migration-prescription) a refusal of a QUERY shape at the engine's per-aggregation filter position: the operator x declared-type pairs refused are exactly the pairs driver-sql's where has refused on a JSON-stored column since its column-type gate landed, and the per-aggregation position now answers them the same way. No authorable key, spelling or stored metadata shape moves: FilterConditionSchema, AggregationNodeSchema and every object and dataset definition parse and save as before, and nothing reads or rewrites a stored row. There is nothing for objectstack migrate meta to rewrite, since what changes is which query the engine answers, not what any metadata says; the refusal itself names the spelling to use. The other categories are closed on facts: every bumped package publishes (not unpublished); no ADR-0087 id covers a filter operator on a JSON-stored column and this diff adds none (not registered / already-registered); and the change is runtime behaviour plus ADDITIONS only (three new @objectstack/core exports and one new optional trailing parameter on applyInMemoryAggregation), with no published interface or type narrowed or removed (not runtime-interface-only / type-surface-only). -->
12+
13+
**BREAKING** (`@objectstack/objectql`): this narrows what `aggregate` accepts in one position, `aggregations[i].filter`, on every driver and for every caller that reaches the engine: the REST query door (`POST /api/v1/data/:object/query`), a flow or hook, and the analytics strategy that lowers a dataset measure's filter onto `engine.aggregate`. The published `applyInMemoryAggregation(rows, ast, timezone, fields)` narrows the same way when it is handed a field map. It ships as `minor` under the launch-window convention for accept-set narrowings.
14+
15+
**What is refused.** On a field the object declares JSON-stored (a structured-JSON type such as `json` or `address`, an inherently multi-value option type such as `tags`, `multiselect` or `checkboxes`, or a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true`), a per-aggregation `filter` that compares the field with `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$nin` or implicit equality (`{ "owners": "u1" }`) is refused with `INVALID_FILTER` / 400, whatever the comparand (`null` and an empty list included), at any depth under `$and` / `$or` / `$not`, and before any driver is asked for a row, so an empty table refuses it too. That is the set `driver-sql`'s `where` refuses on such a column, for the same reason.
16+
17+
**What an author sees now.** The same 400 body the same filter gets as a `where`: the filter WAS NOT APPLIED, the comparison can never equal one member of a stored list, and the spelling to use, `{ "FIELD": { "$contains": "a" } }` for membership, or an `$or` of `$contains` for any-of. The field and the operator are withheld from the message, as they are for `where`, and the full diagnostic, naming both and the aggregation position, goes to the server log.
18+
19+
**Why a refusal.** The engine evaluates a per-aggregation filter itself, and it compared the whole stored array against a scalar. Measured through `POST /api/v1/data/:object/query` on SQLite and PostgreSQL 16 over six rows of a `multiple: true` lookup, two of them holding `u1`: `{ owners: { $in: ['u1', 'u9'] } }` counted 0, `{ owners: { $nin: ['u1', 'u9'] } }` counted all 6, the two rows it was asked to exclude among them, `$gt` / `$lte` / `$between` counted 4 / 1 / 5, and `{ tags: { $eq: 'red' } }` counted the row holding `['red']` by JS loose equality. The same filters in `where` were 400 on both dialects.
20+
21+
**Who is affected.** A dashboard, report, dataset measure or caller whose per-aggregation filter compares a JSON-stored field with one of those operators and read the count as a real answer. Also a host calling `applyInMemoryAggregation` directly with a `fields` map: it now judges each `aggregations[i].filter` against that map before any row (an empty `rows` array included) and throws the same `INVALID_FILTER` / 400. It takes an optional fifth argument, `reportWithheld(diagnostic)`, which receives the withheld field, operator and position; without it the diagnostic is dropped. A call without `fields` judges nothing, as before. Write `$contains` for "holds this member", an `$or` of `$contains` for "holds any of these", and `$not` around either for the exclusion.
22+
23+
**Unchanged.** `$contains` and `$notContains` (membership on such a field), `$exists`, `$null` and `$empty`; every operator on a field that is not JSON-stored; `having`; `where`; and a host whose engine has no declaration for the object, where nothing is judged.
24+
25+
**`@objectstack/core`** (three new root exports): `JSON_COLUMN_INCOMPATIBLE_OPERATORS`, `jsonColumnOperatorRefusalText(field, op, bare)` and its return type `JsonColumnOperatorRefusalText` (`{ message, diagnostic }`). They are the operator set and the two texts (the withheld message and the full diagnostic) of the JSON-column refusal, so `driver-sql`'s `where` and the engine's per-aggregation filter refuse with one set and one sentence.
26+
27+
**`@objectstack/driver-sql`**: no behaviour change. Its JSON-column gate reads the set and the text from `@objectstack/core`; every refusal it prints is byte for byte what it printed before.

‎packages/core/src/index.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,13 @@ export * from './utils/temporal-comparand.js';
125125
// do not depend on each other, and each driver used to carry its own copy.
126126
export * from './utils/temporal-storage-form.js';
127127

128+
// [#21007] …and the refusal a scalar comparison gets on a field stored as a
129+
// JSON column: the operator set and the words. `driver-sql` refuses it on
130+
// `where`, and `@objectstack/objectql` on the per-aggregation `filter` it
131+
// evaluates itself — one set and one sentence, here for the reason the entry
132+
// above gives.
133+
export * from './utils/json-column-operator-refusal.js';
134+
128135
// [#12350 / ADR-0126 §4] THE activation-ledger row contract, parameterized by
129136
// `metadata_type`. Same reason as the two entries above: its consumers —
130137
// `@objectstack/objectql` (packaged actions) and
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#21007] The JSON-column refusal's operator set and words, moved here from
5+
* `driver-sql` so the engine's per-aggregation `filter` refuses with them too.
6+
*
7+
* Two pins, both against what `driver-sql` answered BEFORE the move:
8+
*
9+
* - **The set** — the 22 spellings `driver-sql`'s module-private
10+
* `JSON_COLUMN_INCOMPATIBLE_OPERATORS` held, member for member.
11+
* - **The words** — the SHA-256 of each text, captured from `driver-sql`'s
12+
* built `jsonColumnOperatorError` at the commit before the move (`8f784959c`)
13+
* through a real `SqlDriver` over SQLite: the withheld message (one text for
14+
* every operator), and the diagnostic for an operator, for `$between`, and
15+
* for the bare equality spelling, on a column named `members`. A hash rather
16+
* than a second literal copy, so this file is not a third place the sentence
17+
* lives; the length beside each hash says how far a failure moved it.
18+
*
19+
* A deliberate change of wording updates the hashes in the PR that makes it —
20+
* and then reaches `where` and the per-aggregation `filter` alike, which is the
21+
* point of the move.
22+
*/
23+
24+
import { describe, it, expect } from 'vitest';
25+
import { createHash } from 'node:crypto';
26+
import { JSON_COLUMN_INCOMPATIBLE_OPERATORS, jsonColumnOperatorRefusalText } from './json-column-operator-refusal.js';
27+
28+
const sha256 = (text: string): string => createHash('sha256').update(text, 'utf8').digest('hex');
29+
30+
describe('[#21007] JSON_COLUMN_INCOMPATIBLE_OPERATORS', () => {
31+
it('holds exactly the spellings driver-sql refused before the move', () => {
32+
expect([...JSON_COLUMN_INCOMPATIBLE_OPERATORS].sort()).toEqual([
33+
'!=', '$between', '$eq', '$gt', '$gte', '$in', '$lt', '$lte', '$ne', '$nin',
34+
'<', '<=', '<>', '=', '==', '>', '>=',
35+
'between', 'in', 'nin', 'not_in', 'notin',
36+
]);
37+
});
38+
39+
it('leaves out the membership spelling, the rest of the text family and the null predicates', () => {
40+
for (const op of ['$contains', '$notContains', '$startsWith', '$endsWith', '$icontains', '$null', '$exists', '$empty']) {
41+
expect(JSON_COLUMN_INCOMPATIBLE_OPERATORS.has(op), op).toBe(false);
42+
}
43+
});
44+
});
45+
46+
describe('[#21007] jsonColumnOperatorRefusalText — byte for byte what driver-sql printed before the move', () => {
47+
const MESSAGE = { sha: 'c6103dd665625ab3a779822fecbe650bd7f67fc6ea38d17d5fd57cf6ae9193ff', length: 748 };
48+
49+
it.each([
50+
['an operator', '$in', false, { sha: '358d5aae39170ab3da198beb39368a3f3476dd057dbd5da5f4295cf244eab67e', length: 648 }],
51+
['$between', '$between', false, { sha: '505c094ac5212ac121cdeb4803787ba036f46177450d542fbccf60a9386924fe', length: 658 }],
52+
['the bare equality spelling', '=', true, { sha: '92bb1728ca012c4cde6deb05be3cb0320e113c8c5ac250ecad5b091f3bb87094', length: 660 }],
53+
] as const)('%s', (_name, op, bare, diagnostic) => {
54+
const text = jsonColumnOperatorRefusalText('members', op, bare);
55+
expect({ sha: sha256(text.message), length: text.message.length }).toEqual(MESSAGE);
56+
expect({ sha: sha256(text.diagnostic), length: text.diagnostic.length }).toEqual(diagnostic);
57+
});
58+
59+
it('the message names neither the field nor the operator, and prescribes $contains and an $or of it', () => {
60+
const { message, diagnostic } = jsonColumnOperatorRefusalText('secret_col', '$nin', false);
61+
expect(message).not.toContain('secret_col');
62+
expect(message).not.toContain('"$nin"');
63+
expect(message).toContain('WAS NOT APPLIED');
64+
expect(message).toContain('{ "FIELD": { "$contains": "a" } }');
65+
expect(message).toContain('{ "$or": [{ "FIELD": { "$contains": "a" } }');
66+
expect(diagnostic).toContain('Operator "$nin" on field "secret_col"');
67+
expect(diagnostic).toContain('{ "secret_col": { "$contains": "a" } }');
68+
});
69+
});
Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#21007] The refusal a SCALAR comparison operator gets when it is aimed at a
5+
* field stored as a JSON column — a `multiple: true` field, an inherently
6+
* multi-value option type (`tags`, `multiselect`, `checkboxes`) or a
7+
* structured-JSON type (`json`, `address`, …): the operator set and the words.
8+
*
9+
* ## Two faces, one rule
10+
*
11+
* `driver-sql` refuses these operators on its `where` (#7398): such a column
12+
* holds the serialization `["a","b"]`, so `$in` / `$eq` compare that whole text
13+
* against one value and match nothing, while `$nin` / `$ne` return the very rows
14+
* they were asked to exclude, and the orderings return a lexicographic verdict
15+
* over the serialization. `@objectstack/objectql` evaluates a per-aggregation
16+
* `filter` itself, row by row, and gave the same three wrong answers in JS —
17+
* `{ owners: { $nin: ['u1'] } }` counted the rows holding `u1`. It now refuses
18+
* the same operators on the same declared fields, before any driver is asked.
19+
*
20+
* The two faces cannot import each other (the engine does not depend on a
21+
* driver), and a copy each is how one refusal comes to answer one mistake in
22+
* two ways. So the set and the sentence live here, on the floor both already
23+
* stand on — beside `temporalStorageForm`, which the same two faces share for
24+
* the same reason. Each face keeps its own error CONSTRUCTOR (the driver's
25+
* carries the #8220 provenance seam, the engine's the ADR-0112 envelope); what
26+
* they share is what the caller reads.
27+
*
28+
* ## The other half of the JSON column's contract
29+
*
30+
* `$contains` is the membership spelling on such a column (`FILTER_OPERATORS`'
31+
* `$contains` docblock, `@objectstack/spec`), and it is what the refusal
32+
* prescribes — `$contains` for one member, an `$or` of `$contains` for any-of.
33+
* That is why it is ABSENT from the set below, with the rest of the text family
34+
* and the null predicates.
35+
*/
36+
37+
/**
38+
* [#7398] Operators whose SQL lowering compares a column's STORED SCALAR to a
39+
* value — every spelling either of `driver-sql`'s two comparison emitters
40+
* answers (`applyFilterCondition`'s plain-column switch and
41+
* `applyNormalizedComparison`'s normalised arms).
42+
*
43+
* The bare infix forms are here for the same reason they are in `driver-sql`'s
44+
* `SCALAR_COMPARAND_OPERATORS`: `applyNormalizedComparison` really does
45+
* answer `in` / `nin` / `not_in` / `notin` / `=` / `<>` / `>` …, so a filter
46+
* spelled that way against a normalised column compiles, and a gate that only
47+
* knew the `$`-forms would leave the failure alive at a different spelling —
48+
* the lesson #5234 already paid for there.
49+
*
50+
* `$between` is included although the card's minimum set stopped at the four
51+
* ordering comparisons: it IS `>= AND <=` (`driver-sql` even decomposes a
52+
* calendar-day `$between` into `$gte`/`$lt` ahead of its emitter), so refusing
53+
* the halves and compiling the compound would be the same wrong answer at one
54+
* more spelling.
55+
*
56+
* Deliberately ABSENT, and this is the load-bearing half of the set: the `LIKE`
57+
* family (`$contains`, `$notContains`, `$startsWith`, `$endsWith`,
58+
* `$icontains`) and the null predicates (`$null`, `$exists`). `$contains` is
59+
* the ONLY working membership spelling on a JSON-array column and downstream
60+
* code depends on it (#7398's own tables), while `IS NULL` asks about the
61+
* column's presence, which is a well-formed question whatever the column holds.
62+
*
63+
* [#21007] Moved here from `driver-sql`, unchanged, so the per-aggregation
64+
* `filter` refuses exactly the operators `where` refuses.
65+
*/
66+
export const JSON_COLUMN_INCOMPATIBLE_OPERATORS: ReadonlySet<string> = new Set([
67+
'$eq', '=', '==',
68+
'$ne', '!=', '<>',
69+
'$gt', '>', '$gte', '>=', '$lt', '<', '$lte', '<=',
70+
'$in', 'in',
71+
'$nin', 'nin', 'not_in', 'notin',
72+
'$between', 'between',
73+
]);
74+
75+
/** The two texts of one JSON-column refusal — see {@link jsonColumnOperatorRefusalText}. */
76+
export interface JsonColumnOperatorRefusalText {
77+
/**
78+
* What the caller is told. It names neither the field nor the operator: on a
79+
* read scope the predicate is an administrator's, so both are withheld
80+
* (#7929 / #8197), and the sentence says where they went.
81+
*/
82+
readonly message: string;
83+
/** The full diagnostic — the field and the operator named — for the server log. */
84+
readonly diagnostic: string;
85+
}
86+
87+
/**
88+
* [#7398] The words of the refusal: a scalar-comparison operator met a field
89+
* stored as JSON TEXT, so the comparison can never mean what the caller wrote.
90+
*
91+
* The mechanism is one line of SQL. A `multiple: true` field is stored as the
92+
* serialization `["U1","U2"]`, so `members in ('U1')` is FALSE — the text
93+
* genuinely is not equal to that id — and `members not in ('U1')` is TRUE:
94+
*
95+
* - `$in` / `$eq` / bare equality → **0 rows**, fail-CLOSED. Silent, and a
96+
* `200` with an empty array is byte-identical to a query that legitimately
97+
* matched nothing, so no caller has anything to key on.
98+
* - `$nin` / `$ne` → **the row it was asked to exclude**, fail-OPEN. That is
99+
* the dangerous half and the reason this is a refusal rather than a
100+
* documented footgun: an exclusion that silently stops excluding WIDENS a
101+
* result set, the direction #3948 / #4209 / #5347 all ruled outranks a
102+
* narrowing one.
103+
* - The ordering comparisons are not even uniformly empty: `$lte` matched,
104+
* because `["usr_…"` sorts below `usr_…` on the leading `[`. A lexicographic
105+
* compare over a serialization is a wrong answer, not a narrow one.
106+
*
107+
* The message states the filter WAS NOT APPLIED, because "no rows" is a
108+
* legitimate answer to a legitimate query, so a caller must be told that this
109+
* one was never asked. The prescription is `$contains` (and an `$or` of
110+
* `$contains` for any-of). It survives redaction with PLACEHOLDER names — the
111+
* SHAPE is the repair, and the shape names nothing.
112+
*
113+
* `bare` is the implicit-equality spelling `{ field: value }`, whose operator
114+
* the diagnostic names as `=`.
115+
*
116+
* [#21007] Moved here from `driver-sql`'s `jsonColumnOperatorError`, byte for
117+
* byte, so `where` and the per-aggregation `filter` print one sentence. The
118+
* caller builds the error: this returns only the text.
119+
*/
120+
export function jsonColumnOperatorRefusalText(
121+
field: string,
122+
op: string,
123+
bare: boolean,
124+
): JsonColumnOperatorRefusalText {
125+
const spelling = bare
126+
? `The bare equality spelling { "${field}": value }`
127+
: `Operator "${op}"`;
128+
const on = bare ? '' : ` on field "${field}"`;
129+
return {
130+
message:
131+
`A constraint in this filter WAS NOT APPLIED: it aims a scalar comparison operator at a ` +
132+
`field this driver stores as a JSON TEXT column (e.g. ["a","b"]), and such an operator ` +
133+
`compares that whole serialized text against a single value — it can never equal one ` +
134+
`member. Use "$contains" for membership ({ "FIELD": { "$contains": "a" } }), or an $or of ` +
135+
`"$contains" for any-of ({ "$or": [{ "FIELD": { "$contains": "a" } }, ` +
136+
`{ "FIELD": { "$contains": "b" } }] }). Refused rather than compiled because the answer ` +
137+
`was silently wrong in BOTH directions: $in/$eq matched nothing, while $nin/$ne returned ` +
138+
`the very rows they were asked to exclude. The field and the operator this filter ` +
139+
`used are withheld from the message; the full diagnostic is in the server log.`,
140+
diagnostic:
141+
`${spelling}${on} WAS NOT APPLIED: "${field}" is a multi-value (or otherwise JSON-valued) ` +
142+
`field, stored by this driver as a JSON TEXT column (e.g. ["a","b"]), and "${op}" compares ` +
143+
`that whole serialized text against a single value — it can never equal one member. ` +
144+
`Use "$contains" for membership ({ "${field}": { "$contains": "a" } }), or an $or of ` +
145+
`"$contains" for any-of ({ "$or": [{ "${field}": { "$contains": "a" } }, ` +
146+
`{ "${field}": { "$contains": "b" } }] }). Refused rather than compiled because the answer ` +
147+
`was silently wrong in BOTH directions: $in/$eq matched nothing, while $nin/$ne returned ` +
148+
`the very rows they were asked to exclude.`,
149+
};
150+
}

0 commit comments

Comments
 (0)