fix(objectql,spec)!: a groupBy on a multi-value field and a count_distinct on a JSON-stored field are refused INVALID_FIELD / 400 at the engine aggregate door, on every driver (#20808)

.changeset/20808-json-stored-group-distinct-refused.md

@@ -0,0 +1,33 @@
+---
+"@objectstack/objectql": minor
+"@objectstack/spec": minor
+"@objectstack/lint": patch
+"@objectstack/service-analytics": patch
+---
+
+fix(objectql,spec)!: a `groupBy` on a multi-value field and a `count_distinct` on a JSON-stored field are refused with `INVALID_FIELD` / 400 at the engine's `aggregate`, on every driver, and the aggregate × field-type table stops accepting `count_distinct` over the JSON-stored types
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (already-registered dataset-measure-aggregate-field-type-refused) the one metadata-facing half of this change is a row of AGGREGATE_FIELD_TYPE_COMPATIBILITY narrowing, and that family's hand-migration is already registered under protocol major 18 by this id: "an aggregate the field's type accepts, per AGGREGATE_FIELD_TYPE_COMPATIBILITY", with every refused pair of the table refused at the compile door. The non-temporal sum / avg narrowing rode the same id the same way; this diff amends that entry's surface and acceptance prose to name the count_distinct rider, and corrects the min / max entry's route that called count_distinct valid over every type. The engine-door halves refuse a query shape, not a stored one: no authorable key, export or stored row moves. -->
+
+**BREAKING** (`@objectstack/objectql`): this narrows what `aggregate` accepts, in two positions, on every driver and for every caller that reaches the engine (the REST query door, a flow or hook, and the analytics strategy that lowers a cube query onto `engine.aggregate`). Shipped as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
+
+- A `groupBy` entry that names a **multi-value** field: an inherently-multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. Both entry spellings are judged, the field name and the `{ field }` object.
+- A `count_distinct` aggregation over a **JSON-stored** field: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently-multi option type, or a multi-capable field declared `multiple: true`.
+
+**BREAKING** (`@objectstack/spec`): `AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct` no longer lists the ten JSON-stored types (the structured-JSON seven and `multiselect`, `checkboxes`, `tags`), so `isAggregateCompatibleWithFieldType('count_distinct', type)` answers `false` for them. Every reader of the table refuses those pairs now: the dataset-measure lint rule (`measure-aggregate-field-type-refused`, run by `os validate` and at a runtime dataset save), the analytics dataset compile leg (`400 DATASET_INVALID`), and the engine door above. The `count` row is unchanged.
+
+**What an author sees now.** `400 INVALID_FIELD`, naming the position (`groupBy[0]`, `groupBy[0].field`, or `aggregations[0].field`), the field and its declaration, saying the query was not run, and naming the route inside the first 500 characters the REST door keeps. For a multi-value field the route is to filter by one member: `where` with `$contains` on the field, one query per member. For a structured-JSON field it is to store the part you count in a field of its own, or to count rows with `count`. The thrown error carries `field`, `fields`, `object` and `param` (`groupBy` or `aggregations`).
+
+**Why a refusal.** Every SQL driver stores these values in a JSON column, and the drivers share no meaning for one as a group key or a distinct key. Measured through `POST /api/v1/data/:object/query` over three rows: grouping by any of the eight multi-value declarations answered one group per array on the in-memory driver, one group per serialized array on SQLite, and 500 `DATABASE_ERROR` on PostgreSQL 16. `count_distinct` over any structured-JSON or multi-value field answered 3 on the in-memory driver (equal values counted apart), 2 on SQLite (serialized text compared), and 500 on PostgreSQL (no equality operator for `json`). No example app and no published stack groups by a multi-value field or counts one distinct, so no meaning is defined for either here.
+
+**What to write instead.** A dataset measure or a query that counted a JSON-stored field distinct: use `count` over it, or store the scalar part you meant to count in a field of its own and `count_distinct` that field. A grouping by a multi-value field: filter by each member with `$contains` and count.
+
+**Who is affected.** A caller that grouped by a multi-value field, or counted a JSON-stored field distinct, on the in-memory driver or on SQLite and read the answer as a real one; on PostgreSQL both were already a 500. A dataset whose measure pairs `count_distinct` with a JSON-stored field is refused by the lint rule and the compile leg.
+
+**Unchanged.** (Two shapes the structured-JSON `groupBy` entry of this same release lists as unchanged are narrowed here: a `multiple: true` select as a group key, and `count_distinct` over a structured-JSON field. This entry is the later word on both.) A `groupBy` or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; the `having`, filter and sort positions; and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached.
+
+`@objectstack/lint`: the dataset-measure refusal's hint no longer says `count_distinct` accepts every type.
+
+`@objectstack/service-analytics`: the dataset compile leg's refusal of a `count_distinct` measure over a JSON-stored field says why it diverges (the drivers compare the values for equality three ways) and prescribes `count`, or a scalar field for the part being counted; its other refusals no longer say `count_distinct` accepts every type.

content/docs/deployment/validating-metadata.mdx

@@ -206,9 +206,11 @@ measures: [{ name: 'avg_closed', aggregate: 'avg', field: 'closed_at' }]
 `avg` over a temporal column is where this bites hardest: one SQL family coerces
 the stored text and returns a plausible number (the average *year*), another has
 no such function and fails at query time. `min`/`max` over the same field are
-**accepted** — they return a real instant of the field's own type — and
-`count`/`count_distinct` are accepted over every type, because they read no
-arithmetic off the value. The analytics service refuses the same pair with
+**accepted** — they return a real instant of the field's own type — `count` is
+accepted over every type, because it reads no value, and `count_distinct` over
+every type but the JSON-stored ones (the structured-JSON types and
+`multiselect` / `checkboxes` / `tags`), whose values no two backends compare
+for equality alike. The analytics service refuses the same pair with
 `400 DATASET_INVALID` when a query is built; this is the identical verdict,
 from the identical table, one door earlier. The rule stays silent wherever the
 field's type cannot be resolved (an object this stack does not define, a

packages/lint/src/validate-dataset-measure-aggregates.test.ts

@@ -165,13 +165,28 @@ describe('measure-aggregate-field-type-refused — stays silent on every pair th
     }
   });
 
-  it('accepts count and count_distinct over every declared FieldType', () => {
+  it('accepts count over every declared FieldType', () => {
     for (const fieldType of FieldType.options) {
-      for (const aggregate of ['count', 'count_distinct']) {
-        expect(findings(stackWith(aggregate, fieldType)), `${aggregate}(${fieldType})`).toEqual([]);
-      }
+      expect(findings(stackWith('count', fieldType)), `count(${fieldType})`).toEqual([]);
     }
   });
+
+  // [#20808] `count_distinct` over a JSON-stored type left the table's row: no
+  // two backends compare those values for equality alike (the in-memory driver
+  // counted equal documents apart, SQLite compared serialized text, PostgreSQL
+  // answered 500). Every other declared type stays accepted.
+  it('accepts count_distinct over every declared FieldType except the JSON-stored ones, which it refuses', () => {
+    const jsonStored = ['json', 'composite', 'repeater', 'record', 'location', 'address', 'vector', 'multiselect', 'checkboxes', 'tags'];
+    for (const fieldType of FieldType.options) {
+      const found = findings(stackWith('count_distinct', fieldType));
+      expect(found.length, `count_distinct(${fieldType})`).toBe(jsonStored.includes(fieldType) ? 1 : 0);
+    }
+    const [issue] = findings(stackWith('count_distinct', 'json'));
+    expect(issue.rule).toBe(RULE);
+    // The way out names `count`, and no sentence says count_distinct accepts every type.
+    expect(issue.hint).toContain('accepts: count.');
+    expect(issue.hint).not.toMatch(/count_distinct` accept every type/);
+  });
 });
 
 describe('measure-aggregate-field-type-refused — the rule IS the table, on every pair', () => {

packages/lint/src/validate-dataset-measure-aggregates.ts

@@ -232,8 +232,9 @@ export function validateDatasetMeasureAggregates(stack: unknown): DatasetMeasure
           `Either point "${aggregate}" at a field of an accepted type, or aggregate ` +
           `"${field}" with one its \`${fieldType}\` type accepts: ` +
           `${aggregatesAccepting(fieldType).join(', ')}. ` +
-          `\`count\` / \`count_distinct\` accept every type because they read no arithmetic off ` +
-          `the value; a quantity that must be added up or averaged has to be STORED as a ` +
+          `\`count\` accepts every type because it reads no value, and \`count_distinct\` every ` +
+          `type but the JSON-stored ones, whose values no two backends compare alike; a ` +
+          `quantity that must be added up or averaged has to be STORED as a ` +
           `numeric field (a computed column) and aggregated as one. The compile leg refuses ` +
           `this same pair with \`400 DATASET_INVALID\` before any SQL is emitted, so this is ` +
           `the same fix made earlier.`,

packages/objectql/src/count-distinct-json-stored-door.ts

@@ -0,0 +1,158 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#20808] A `count_distinct` aggregation over a JSON-STORED field is refused
+ * with `INVALID_FIELD` / 400 by the engine's `aggregate`, naming the field,
+ * its declared type and the position, before any driver is asked.
+ *
+ * ## What ran before this door, measured on `origin/main` `42d78b97fe`
+ *
+ * Through `POST /api/v1/data/:object/query` (`{ aggregations: [{ function:
+ * 'count_distinct', field: FIELD, alias: 'n' }] }`) over three rows, two of
+ * which hold equal values under the counted field:
+ *
+ * | counted field | InMemoryDriver | SqlDriver, SQLite | SqlDriver, PostgreSQL 16 |
+ * |:--|:--|:--|:--|
+ * | a `text` or single-value `select` (the control) | 2 | 2 | 2 |
+ * | a structured-JSON field (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) | 3 | 2 (3 for `json`, whose three documents differ) | **500 `DATABASE_ERROR`** |
+ * | a multi-value field (`multiselect`, `checkboxes`, `tags`; `select`, `lookup`, `user`, `file`, `image` with `multiple: true`) | 3 | 2 | **500 `DATABASE_ERROR`** |
+ *
+ * The in-memory driver compares each row's value by identity, so equal
+ * documents count apart; SQLite compares the serialized text; PostgreSQL has
+ * no equality operator for a `json` column ("could not identify an equality
+ * operator for type json") and answers 500. One query, three answers.
+ *
+ * ## Whose verdict it is
+ *
+ * The TYPE half is the spec table's: `AGGREGATE_FIELD_TYPE_COMPATIBILITY`'s
+ * `count_distinct` row, asked through `isAggregateCompatibleWithFieldType`
+ * (`@objectstack/spec/data`) — ⛔ never a second list here. The triage
+ * direction on #20808: the table "stops accepting it" for the structured-JSON
+ * class, on the table's own ground ("can every backend give one answer"); the
+ * row refuses the three multi-option types too, on the same measurement.
+ *
+ * The DECLARATION half is `isMultiValueField`: a multi-capable type flagged
+ * `multiple: true` (`select`, `lookup`, `user`, `file`, `image`) is stored in
+ * a JSON column like the rest, but a per-TYPE table cannot see the flag. The
+ * two answers are asked side by side, so a field is refused if either one
+ * refuses it.
+ *
+ * ⛔ No per-backend JSON distinctness is defined to make the pair answerable:
+ * no caller of it was measured (no dataset measure, widget or `count_distinct`
+ * in `examples/` or the published hotcrm stack counts one).
+ *
+ * ## Where it stands, and what it judges
+ *
+ * At the entry of `aggregate`, right after the `groupBy` door
+ * (`group-by-structured-json-door.ts`), before the per-aggregation `filter`
+ * doors and before any driver is resolved — so it holds for every caller that
+ * reaches the engine: the REST query door, a flow, a hook, and the analytics
+ * strategy that lowers a cube measure onto `engine.aggregate`.
+ *
+ * **Not judged:** any other aggregate function (`count` compares no value; the
+ * table's other rows have their own doors and are not this card's), a
+ * `count_distinct` with no named field, an undeclared name, a registry-less
+ * host (no field map, no verdict), and a declared type outside `FieldType`
+ * (a driver-internal alias such as `string` or `integer` on an introspected
+ * object): the table is fail-closed on vocabulary, and "cannot answer, do not
+ * block" is this consumer's tier, as the table's own TSDoc says.
+ *
+ * `INVALID_FIELD`, the code the `groupBy` door beside it answers: the verdict
+ * is about the NAMED field's type at a position.
+ *
+ * @see https://github.com/objectstack-ai/objectstack/issues/20808
+ */
+
+import { StandardErrorCode } from '@objectstack/spec/api';
+import { FieldType, isAggregateCompatibleWithFieldType, isMultiValueField } from '@objectstack/spec/data';
+
+/** The declared `FieldType` vocabulary — the only types the table can answer for. */
+const DECLARED_FIELD_TYPES: ReadonlySet<string> = new Set(FieldType.options);
+
+/** One `count_distinct` aggregation that names a JSON-stored field. */
+interface JsonStoredDistinctTarget {
+  readonly field: string;
+  readonly type: string;
+  /** The declared field carries `multiple: true` (said in the words). */
+  readonly multiple: boolean;
+  /** A multi-value field (the route differs: filter by one member). */
+  readonly multiValue: boolean;
+  /** `aggregations[i].field`. */
+  readonly position: string;
+}
+
+/**
+ * The `count_distinct` aggregations that name a declared field the table's
+ * `count_distinct` row refuses, or a declared multi-value field, in order.
+ */
+function jsonStoredDistinctTargets(
+  fields: Record<string, unknown>,
+  aggregations: readonly unknown[],
+): JsonStoredDistinctTarget[] {
+  const hits: JsonStoredDistinctTarget[] = [];
+  for (const [i, agg] of aggregations.entries()) {
+    if (agg === null || typeof agg !== 'object' || Array.isArray(agg)) continue;
+    if ((agg as { function?: unknown }).function !== 'count_distinct') continue;
+    const field = (agg as { field?: unknown }).field;
+    if (typeof field !== 'string' || field === '*') continue;
+    if (!Object.prototype.hasOwnProperty.call(fields, field)) continue;
+    const def = fields[field] as { type?: unknown; multiple?: unknown } | undefined;
+    const type = def?.type;
+    if (typeof type !== 'string' || !DECLARED_FIELD_TYPES.has(type)) continue;
+    const multiple = def?.multiple === true;
+    const multiValue = isMultiValueField({ type, multiple });
+    if (isAggregateCompatibleWithFieldType('count_distinct', type) && !multiValue) continue;
+    hits.push({ field, type, multiple, multiValue, position: `aggregations[${i}].field` });
+  }
+  return hits;
+}
+
+/**
+ * Refuse a `count_distinct` aggregation over a declared JSON-stored field —
+ * `INVALID_FIELD` / 400, before any driver is asked. See the module header.
+ *
+ * The words put the position and the verdict first, then that the query did
+ * not run, then the route, then the reason: the REST door keeps the first 500
+ * characters of a 4xx message (`CLIENT_MESSAGE_MAX`), and the route must be
+ * inside them.
+ */
+export function assertCountDistinctNamesNoJsonStoredField(
+  object: string,
+  schema: unknown,
+  aggregations: unknown,
+): void {
+  if (!Array.isArray(aggregations) || aggregations.length === 0) return;
+  const fields = (schema as { fields?: unknown } | undefined)?.fields;
+  if (!fields || typeof fields !== 'object') return;
+  const hits = jsonStoredDistinctTargets(fields as Record<string, unknown>, aggregations);
+  if (hits.length === 0) return;
+  const [first] = hits;
+  const declared = first.multiple ? `${first.type} field with multiple: true` : `${first.type} field`;
+  const kind = first.multiValue ? 'a multi-value field' : 'a structured-JSON value';
+  const route = first.multiValue
+    ? 'Count the records that hold one member instead: count with '
+      + `where { "${first.field}": { "$contains": VALUE } }, one query per member.`
+    : 'Count distinct values of a field that stores one scalar value: store the part you count in a '
+      + 'field of its own and count_distinct that field, or count the rows with count.';
+  const err = new Error(
+    `aggregate('${object}'): ${first.position} counts distinct '${first.field}', a declared ${declared} `
+    + `— ${kind}, which the engine does not count distinct`
+    + (hits.length > 1 ? ` (also: ${hits.slice(1).map((h) => `'${h.field}'`).join(', ')})` : '')
+    + `. The query was NOT run. ${route} `
+    + 'A JSON-stored value is no distinct key the drivers share: one counted every row apart, one '
+    + 'compared the serialized text, one refused the statement.',
+  ) as Error & {
+    code?: string; status?: number; httpStatus?: number;
+    field?: string; fields?: string[]; object?: string; param?: string;
+  };
+  err.code = StandardErrorCode.enum.INVALID_FIELD;
+  err.status = 400;
+  // …and `httpStatus`, the same number under ADR-0112 D5's spelling — what a
+  // consumer holding the THROWN error reads; `status` stays for the HTTP doors.
+  err.httpStatus = 400;
+  err.field = first.field;
+  err.fields = hits.map((h) => h.field);
+  err.object = object;
+  err.param = 'aggregations';
+  throw err;
+}