fix(spec)!: a filter carrying a comparand the query faces refuse is refused when it is saved (#20116)

.changeset/20116-filter-save-door-face-parity.md

@@ -0,0 +1,74 @@
+---
+"@objectstack/spec": minor
+---
+
+fix(spec)!: a filter carrying a comparand the query faces refuse is refused when it is saved (#20116)
+
+**BREAKING** — an accept-set narrowing of published authoring schemas, shipped as `minor` under the repo's launch-window convention for accept-set narrowings. The save door narrows to exactly what the query faces already refuse. The hand-migration prescription is registered under protocol major 18 as `filter-query-face-comparands-refused-at-save`.
+
+## What changes
+
+`FilterConditionSchema` now refuses, at parse, every comparand slot the query faces refuse:
+
+- a `$null` or `$exists` flag that is not a boolean — `"false"`, `"x"`, `null`, `1`;
+- a `null` comparand of `$gt` / `$gte` / `$lt` / `$lte`;
+- an `$in` or `$nin` comparand that is not a list, or a list holding `null`;
+- a `$between` comparand that is not a two-element list, or whose endpoint is `null`, blank (`""`) or a `{ $field }` reference;
+- an array under `$ne`.
+
+It judges the field entries of a condition and of every `$and` / `$or` / `$not` member. The dataset `filter` and measure `filter` also judge the same slots INSIDE a nested-relation condition, through the nested-relation walk they already carry, because the analytics `where` door flattens a relation and judges its entries. The shared comparand-shape face (`assertListComparandShapes`) is called read-only as the judge for each slot, so the save door refuses exactly what that face refuses on every query. The two boolean flags, which that face does not judge, are refused on the predicate every flag face uses (`driver-sql`, `driver-memory`, `driver-mongodb`, the read-scope compiler and the analytics `where` door): the comparand is not a boolean.
+
+Measured on `origin/main` `af32cf9a` before the change: a dataset `filter`, a dataset measure `filter`, a dashboard widget `filter` and a report `runtimeFilter` each parsed with `success: true` for one instance of every shape above. The comparand-shape face refused each one but the flags with `INVALID_FILTER` / 400, and the analytics `where` door refused all of them, inside a nested relation too. So such a document published clean and then failed every chart built on it.
+
+Every schema that carries a `FilterCondition` refuses on parse. That covers the dataset `filter` and measure `filter`, the dashboard widget `filter` and options-source `filter`, the report and joined-report-block `runtimeFilter`, the field `relatedListFilter` and rollup `summaryOperations.filter`, the solution-blueprint summary `filter`, the analytics query `where`, the dataset selection `runtimeFilter`, the query `where` and `having`, the data-engine aggregate call's `having`, the aggregation `filter`, and the query-filter `where`. So `defineStack`, `os validate` and a save through the metadata protocol (`422 INVALID_METADATA`) refuse such a document at the slot's path, for example `filter.stage.$null` or `measures.0.filter.amount.$between.0`.
+
+The words are the query face's. For an array under `$ne`, a non-list `$in` / `$nin` and a malformed `$between`, the refusal is the face's sentence without its location clause (`at where.<field>.<op>`), because the issue's path carries the location. For a `null` ordering comparand, a `null` list member or endpoint, and a blank or `{ $field }` endpoint, it is the sentence the enforced operator slot (`FieldOperatorsSchema`) already prints for the same comparand. A non-boolean flag gets the query faces' sentence: `Operator "$null" on field "stage" requires a boolean comparand (true or false).`, then the received value and the prescription.
+
+Two request doors parse these carriers, and they now answer before the analytics compiler does. The REST dataset selection (its `runtimeFilter`, parsed against `DatasetSelectionSchema`) and the analytics query body (its `where`, parsed against `AnalyticsQueryRequestSchema`) answer `VALIDATION_FAILED` / 400 with the sentence at the field, for a top-level or combinator slot. Before, the compiler answered `INVALID_FILTER` / 400 for the same filter.
+
+This also changes the `$ne` note of the equality-slot change earlier in this release: an array under `$ne` is now refused on save too, in the sentence `FieldOperatorsSchema.$ne` and the face print.
+
+## What does NOT change
+
+- **Nothing stored is rewritten, and nothing is dropped.** The parse fails and strips nothing. The read path does not re-validate stored rows, so a stored document keeps loading, and its next save is refused. Such a filter has failed every query since the runtime refusal of its shape, so the refusal is a repair.
+- **The shared reach is the face's, and no wider.** On every carrier but the two dataset ones, a field spec with no `$` key, such as the nested-relation condition `{ account: { region: { $in: ["a", null] } } }`, is not judged, because neither the face nor the drivers' flag checks descend one. A dashboard widget `filter` and a report `runtimeFilter` therefore still save that shape, and the analytics `where` door refuses it when they are charted.
+- **What the face passes still passes:** `$eq: null` and `$ne: null` (the null predicate), a `{ $field }` reference as the whole comparand of a scalar comparison, `$in: []` and `$nin: []`, a whitespace-only `$between` endpoint, and falsy endpoints such as `[0, 0]`.
+- **The data-engine calls' `where` option still parses.** Its type is a union whose first arm is an open record. The face refuses the shape when the call runs.
+- **No key, export or JSON Schema changes.** The published JSON Schema cannot state a refinement, and `FilterCondition`'s already could not state the equality-slot one.
+
+## FROM → TO
+
+| you wrote | write instead |
+|:--|:--|
+| `{ stage: { $null: "true" } }`, `{ stage: { $null: 1 } }` | `{ stage: { $null: true } }` ("has no value") |
+| `{ stage: { $exists: "false" } }`, `{ stage: { $null: null } }` | the boolean you meant: `$exists: false` is "has no value", `$null: false` is "has a value" |
+| `{ amount: { $gt: null } }` | `{ amount: { $eq: null } }` ("has no value") or `{ amount: { $ne: null } }` ("has a value") |
+| `{ stage: { $in: "won" } }` | `{ stage: { $in: ["won"] } }` or `{ stage: "won" }` |
+| `{ stage: { $in: ["won", null] } }` | `{ $or: [{ stage: { $in: ["won"] } }, { stage: { $null: true } }] }` |
+| `{ stage: { $nin: ["lost", null] } }`, meaning "has a value and is not `lost`" | `{ stage: { $nin: ["lost"], $null: false } }` |
+| `{ stage: { $in: [null, ""] } }`, a filter builder's "is empty" | `{ $or: [{ stage: { $null: true } }, { stage: "" }] }` |
+| `{ stage: { $nin: [null, ""] } }`, a filter builder's "is not empty" | `{ stage: { $null: false, $ne: "" } }` |
+| `{ amount: { $between: [null, 5] } }`, `{ amount: { $between: ["", 5] } }` | `{ amount: { $lte: 5 } }`, or the bound you meant |
+| `{ amount: { $between: [{ $field: "floor" }, 5] } }` | `{ amount: { $gte: { $field: "floor" }, $lte: 5 } }` |
+| `{ amount: { $between: 5 } }`, `{ amount: { $between: [1] } }` | `{ amount: { $between: [1, 5] } }` |
+| `{ stage: { $ne: ["won", "lost"] } }` | `{ stage: { $nin: ["won", "lost"] } }` |
+
+### FROM → TO at the HTTP doors
+
+Same status, different code: the refusal now comes from the route's schema door, located on the member, instead of from the analytics filter normalizer.
+
+| request | before | after |
+|:--|:--|:--|
+| `POST /analytics/dataset/query` with `selection.runtimeFilter: { amount: { $between: [10] } }` | `400 INVALID_FILTER` from the analytics normalizer, in the comparand-shape face's sentence | `400 VALIDATION_FAILED`, `details.fields[]` entry `selection.runtimeFilter.amount.$between` with the sentence `Operator "$between" on field "amount" requires a [min, max] value array. Received array ([10]). …` |
+| the same route, any other slot above in `selection.runtimeFilter` (top level or in `$and` / `$or` / `$not`) | `400 INVALID_FILTER` | `400 VALIDATION_FAILED`, located on the slot, with that slot's sentence |
+| `POST /analytics/query` (`AnalyticsQueryRequestSchema`) with the same shape in `where` | `400 INVALID_FILTER` | refused by the request schema at `where.amount.$between`, answered `400 VALIDATION_FAILED` |
+
+A client that branches on `INVALID_FILTER` for these shapes reads `VALIDATION_FAILED` instead. Both are 400 and both name the field.
+
+## Who is affected, measured
+
+A literal-comparand scan of every member shape, with a lit control per shape, over `examples/**` and the non-test `packages/**` of this repository at `af32cf9a`, the console repository at its pinned commit `f8a9d0fb05`, and the cloud repository's `main` at `48d70663ab`, found authored filters carrying one in one place. The console's filter-condition widget writes "is empty" as `{ field: { $in: [null, ""] } }` and "is not empty" as `{ field: { $nin: [null, ""] } }`. That widget edits a field's `relatedListFilter` and a rollup's `summaryOperations.filter` in the Studio field designer, and a sharing rule's criteria. Both shapes carry a `null` list member, which the face has refused on every query since the 2026-08-31 ruling, so a filter saved that way has been failing its related list or rollup since then. After this change, the Studio save is refused instead, with the `$or` / `$null` prescription. Every other hit is prose, a type table or a test fixture. Deployed datasets, dashboards and reports were NOT measured. Validating each stack, or re-saving each document, finds every instance the surface above lists.
+
+Clause-②: no (narrowing) — nothing is widened. No key is added, removed or renamed, no exported symbol moves, and the operator vocabulary is unchanged. Comparand shapes that every query face already refused are now refused on save as well.
+
+<!-- adr-0087: registered filter-query-face-comparands-refused-at-save -->

packages/rest/src/analytics-filter-refusal-envelope.test.ts

@@ -193,19 +193,10 @@ describe('[#5352] POST /analytics/dataset/query — a filter refusal reaches the
       runtimeFilter: { stage: {} },
       message: /carries a field constraint with zero operators/,
     },
-    {
-      // [#20010] RE-WORDED, same verdict: still a real normalizer refusal,
-      // still 400 INVALID_FILTER through this seam. PR #20032 hands every field
-      // entry of an object-form `where` to the shared comparand-shape face
-      // (`assertListComparandShapes`) before any node is built, and the face's
-      // own rule answers first — a list operator takes a list, `$between` a
-      // two-element [min, max] (#5869, moved to the face by #9228). So the
-      // message is the face's, the same bytes the FilterArray spelling gets;
-      // the analytics "needs a two-element" sentence is no longer reached.
-      name: 'a $between with one bound',
-      runtimeFilter: { amount: { $between: [10] } },
-      message: /Operator "\$between" on field "amount" requires a \[min, max\] value array/,
-    },
+    // [#20116] `a $between with one bound` sat here. It no longer reaches the
+    // normalizer: `FilterConditionSchema` now refuses every slot the shared
+    // comparand-shape face refuses, so this route's schema door answers it
+    // first. It moved, with its #20010 note, to the #17551 block below.
     {
       name: 'an unsupported top-level operator',
       runtimeFilter: { $nor: [{ stage: 'won' }] },
@@ -227,7 +218,9 @@ describe('[#5352] POST /analytics/dataset/query — a filter refusal reaches the
 /**
  * [#17551] Three spellings that used to reach the normalizer now stop one layer
  * earlier — at the route's schema door — and the block above no longer claims
- * them.
+ * them. [#20116] A fourth joined them: the one-bound `$between`, once
+ * `FilterConditionSchema` began refusing on save every comparand slot the
+ * shared comparand-shape face refuses on query.
  *
  * ⚠️ This is a CODE change on a live wire surface, so it is recorded with the
  * measurement that justifies it rather than as a test edit. Since #17551 the
@@ -242,25 +235,41 @@ describe('[#5352] POST /analytics/dataset/query — a filter refusal reaches the
  * | `{ $or: [{…}, 'nope'] }`           | refused at the schema | refused at the schema |
  * | `{ $not: 5 }`                      | refused at the schema | refused at the schema |
  * | `{ stage: {} }`                    | passes the schema     | passes the schema     |
- * | `{ amount: { $between: [10] } }`   | passes the schema     | passes the schema     |
+ * | `{ amount: { $between: [10] } }`   | refused at the schema (#20116) | refused at the schema (#20116) |
  * | `{ $nor: [{…}] }`                  | passes the schema     | passes the schema     |
  * | `{ $or: [] }`                      | passes the schema     | passes the schema     |
  *
  * ⇒ the two routes now answer this field IDENTICALLY, which is the whole reason
  * the door exists ("one family, two postures" was the defect). The three rows
- * that changed changed because the dataset route used to be the LOOSER of the
- * two, not because anything narrowed past `FilterCondition`.
+ * that changed under #17551 changed because the dataset route used to be the
+ * LOOSER of the two, not because anything narrowed past `FilterCondition`. The
+ * `$between` row changed under #20116 on BOTH routes at once, because
+ * `FilterCondition` itself narrowed: the schema now refuses what the face
+ * refused on every query, in the face's sentence less its location.
  *
  * ⛔ Nothing here weakens #5352's subject: the seam it exists for — a real
  * `AnalyticsService`, a real `normalizeAnalyticsFilterTree` refusal, and this
  * route's catch reading the envelope rather than a message list — is still
  * driven by every case left in the block above, `$sortOf` included.
  */
 describe('[#17551] the structurally-malformed filter spellings are refused at the door', () => {
-  const AT_THE_DOOR: Array<{ name: string; runtimeFilter: unknown }> = [
+  const AT_THE_DOOR: Array<{ name: string; runtimeFilter: unknown; member?: string; sentence?: RegExp }> = [
     { name: 'an $or that is not an array', runtimeFilter: { $or: 'won' } },
     { name: 'an $or branch that is not a filter object', runtimeFilter: { $or: [{ stage: 'won' }, 'nope'] } },
     { name: 'a $not of a non-object', runtimeFilter: { $not: 5 } },
+    {
+      // [#20010] The face's sentence has named this shape since PR #20032 handed
+      // every field entry of an object-form `where` to the shared comparand-
+      // shape face (`assertListComparandShapes`) — a list operator takes a
+      // list, `$between` a two-element [min, max] (#5869, moved to the face by
+      // #9228) — and it still does. [#20116] What moved is WHERE: the schema
+      // door now asks that face on save and prints its sentence less the
+      // `at where…` location, so the refusal is located on the member instead.
+      name: 'a $between with one bound',
+      runtimeFilter: { amount: { $between: [10] } },
+      member: 'selection.runtimeFilter.amount.$between',
+      sentence: /^Operator "\$between" on field "amount" requires a \[min, max\] value array\. Received array \(\[10\]\)\. A range needs exactly two bounds/,
+    },
   ];
 
   for (const c of AT_THE_DOOR) {
@@ -274,8 +283,17 @@ describe('[#17551] the structurally-malformed filter spellings are refused at th
       expect(res.body.code).not.toBe('ANALYTICS_QUERY_FAILED');
       expect(res.body.code).toBe('VALIDATION_FAILED');
       // …and it says WHICH member, which the deeper refusal never did.
-      const fields: Array<{ field: string }> = res.body.details.fields;
+      const fields: Array<{ field: string; message: string }> = res.body.details.fields;
       expect(fields.map((f) => f.field).some((f) => f.startsWith('selection.runtimeFilter'))).toBe(true);
+      if (c.member) {
+        // [#20116] Exactly one entry, at the slot, carrying the face's sentence.
+        const atMember = fields.filter((f) => f.field === c.member);
+        expect(atMember, JSON.stringify(fields)).toHaveLength(1);
+        expect(atMember[0]!.message).toMatch(c.sentence!);
+        expect(atMember[0]!.message).not.toContain(' at where.');
+        // The normalizer never ran: the refusal is the door's, not INVALID_FILTER.
+        expect(res.body.code).not.toBe('INVALID_FILTER');
+      }
     });
   }
 
@@ -288,12 +306,24 @@ describe('[#17551] the structurally-malformed filter spellings are refused at th
         where: c.runtimeFilter,
       });
       expect(parsed.success, `${c.name} must be refused by the sibling schema too`).toBe(false);
+      if (c.member) {
+        // [#20116] …at the same slot, in the same sentence, one level up: the
+        // sibling body carries the filter as `where`, not `selection.runtimeFilter`.
+        const issues: Array<{ path: PropertyKey[]; message: string }> = parsed.error.issues;
+        const slot = c.member.replace(/^selection\.runtimeFilter\./, 'where.');
+        const atSlot = issues.filter((i) => i.path.join('.') === slot);
+        expect(atSlot, JSON.stringify(issues.map((i) => i.path))).toHaveLength(1);
+        expect(atSlot[0]!.message).toMatch(c.sentence!);
+      }
     }
   });
 
-  it('CONTROL — the four spellings the schema PASSES still cross the seam', async () => {
+  it('CONTROL — the three spellings the schema PASSES still cross the seam', async () => {
+    // [#20116] `{ amount: { $between: [10] } }` left this list: the schema
+    // refuses it now, on both routes (the AT_THE_DOOR row above, and the
+    // sibling-schema control, which iterates that table).
     const { AnalyticsQueryRequestSchema } = await import('@objectstack/spec/api');
-    const passes = [{ stage: {} }, { amount: { $between: [10] } }, { $nor: [{ stage: 'won' }] }, { $or: [] }];
+    const passes = [{ stage: {} }, { $nor: [{ stage: 'won' }] }, { $or: [] }];
     for (const where of passes) {
       const parsed = (AnalyticsQueryRequestSchema as any).safeParse({
         cube: 'opportunity', measures: ['revenue'], where,