Skip to content

Commit 975b248

Browse files
fix(objectql,spec)!: a groupBy on a multi-value field and a count_distinct on a JSON-stored field are refused INVALID_FIELD / 400 at the engine aggregate door, on every driver (#20808) (#20911)
Fixes #20808 Clause-②: no (narrowing) ## What this changes Two more JSON-stored keys are refused `INVALID_FIELD` / 400 by `engine.aggregate`, in the engine's words, before any driver is asked. The aggregate × field-type table stops accepting `count_distinct` over the JSON-stored types, so every reader of that row refuses the pair too. Nothing widens. **The narrowed accept sets, per shape and per door:** | shape | door | before | now | |:--|:--|:--|:--| | `groupBy` entry naming a **multi-value** field (`multiselect`, `checkboxes`, `tags`; `select`, `lookup`, `user`, `file`, `image` declared `multiple: true`), as a name or as `{ field }` | engine `aggregate` (`@objectstack/objectql`): REST query door, flows, hooks, the analytics ObjectQL strategy | accepted, one answer per driver | `400 INVALID_FIELD` at `groupBy[i]` / `groupBy[i].field` | | `count_distinct` over a **structured-JSON** field (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) | engine `aggregate` | accepted | `400 INVALID_FIELD` at `aggregations[i].field` | | `count_distinct` over a **multi-value** field (the same eight declarations) | engine `aggregate` | accepted | `400 INVALID_FIELD` at `aggregations[i].field` | | `count_distinct` × the ten JSON-stored types (structured-JSON seven plus `multiselect`, `checkboxes`, `tags`) | `AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct` (`@objectstack/spec`) | in the row | out of the row; `isAggregateCompatibleWithFieldType('count_distinct', t)` is `false` | | a dataset measure `count_distinct` × a JSON-stored type | lint rule `measure-aggregate-field-type-refused` (`os validate`, and the runtime gate on a dataset save) | clean | error at authoring / save | | the same measure | analytics dataset compile leg | compiled | `400 DATASET_INVALID` | The words, as `POST /api/v1/data/:object/query` returns them (the route sits inside the 500 characters the REST door keeps; the REST pins assert it there): ```text aggregate('probe_20808'): groupBy[0] names 'tags', a declared select field with multiple: true — a multi-value field, which the engine does not group by. The query was NOT run. Filter by one member instead: where { "tags": { "$contains": VALUE } } counts or lists the records that hold VALUE, one query per member. A list of values is no group key the drivers share: one grouped each list apart, one grouped each serialized list apart, one refused the statement. aggregate('probe_20808'): aggregations[0].field counts distinct 'meta', a declared json field — a structured-JSON value, which the engine does not count distinct. The query was NOT run. Count distinct values of a field that stores one scalar value: store the part you count in a field of its own and count_distinct that field, or count the rows with count. A JSON-stored value is no distinct key the drivers share: one counted every row apart, one compared the serialized text, one refused the statement. ``` The thrown error carries `code: 'INVALID_FIELD'`, `status` and `httpStatus` 400, `field`, `fields` (every offender), `object` and `param` (`groupBy` or `aggregations`). **Landing site.** - `packages/objectql/src/group-by-structured-json-door.ts`: the structured-JSON door gains its second class, the multi-value field, judged by `@objectstack/spec/data`'s `isMultiValueField` in the SAME walk over the entries, so the first offending position is named whichever class it is. The export is renamed `assertGroupByNamesNoJsonStoredField`. The structured-JSON words keep their verdict and route; their reason clause now also holds for `vector` (the review note carried on the previous door's landing). - `packages/objectql/src/count-distinct-json-stored-door.ts` (new): `assertCountDistinctNamesNoJsonStoredField`. The TYPE half asks the spec table (`isAggregateCompatibleWithFieldType('count_distinct', type)`), never a second list. The DECLARATION half asks `isMultiValueField`, because a per-type table cannot see `multiple: true`. Not judged: any other function, a `count_distinct` naming no field, an undeclared name, a host with no field map, and a type outside `FieldType` (the table is fail-closed on vocabulary, so an introspected `integer` or `object` column is left alone). - `packages/objectql/src/engine.ts`: one call each, at the entry of `aggregate`, right after the credential refusal and in the order groupBy, then count_distinct. - `packages/spec/src/data/aggregate-field-type-compatibility.ts`: the `count_distinct` row is every `FieldType` except `JSON_STORED_AGGREGATE_FIELD_TYPES` (spelled out, held equal to `STRUCTURED_JSON_TYPES` ∪ `MULTI_OPTION_TYPES` by the pin, as the file does for its other classes). The TSDoc states the ground, the measurement, and the third reader. `INVALID_FIELD`, an existing code: the verdict is about the named field's type at a position, the question the structured-JSON door beside it answers the same way. ## Before, measured on `origin/main` `42d78b97fe` Through `POST /api/v1/data/:object/query` (the real `RestServer` route over `ObjectStackProtocolImplementation` and `ObjectQL`). Drivers: InMemoryDriver, SqlDriver on SQLite (better-sqlite3), and SqlDriver on a private PostgreSQL 16.13 started for this run. Three rows; two of them hold EQUAL values under every JSON-stored field. | query | InMemoryDriver | SQLite | PostgreSQL 16 | |:--|:--|:--|:--| | `groupBy` `title` (text) / `status` (single-value select), the controls | 200, `x` 2 · `y` 1 / `a` 2 · `b` 1 | same | same | | `groupBy` each of the 8 multi-value declarations, and `{ field: 'tags' }` | 200, one group per array | 200, one group per serialized array | **500 `DATABASE_ERROR`** (could not identify an equality operator for type json) | | `count_distinct` `title` / `status`, the controls | 2 | 2 | 2 | | `count_distinct` `json` (three different documents) | 3 | 3 | **500** | | `count_distinct` each other structured-JSON type | 3 | 2 | **500** | | `count_distinct` each multi-value declaration | 3 | 2 | **500** | | `count` over `json` / `tags` (unchanged) | 3 | 3 | 3 | ## After, the same run on this branch Every multi-value `groupBy` and every JSON-stored `count_distinct` above answers `400 INVALID_FIELD` in the engine's words on all three drivers, naming the position, the field and its declaration. The controls and `count` answer exactly as before. The InMemoryDriver cells of both runs come from an uncommitted scratch script over the built packages; `check:driver-memory-census` refuses a new test consumer of that driver without a ruling, so the committed memory cell is the recording driver below, by construction. ## Hypotheses (zone 2): which held - **H1: held, refined.** The door is `group-by-structured-json-door.ts` at the aggregate entry, and the multi-value refusal is one more class there. It is judged on `isMultiValueField`, which reads the declaration AND the type: `select`, `radio`, `lookup`, `user`, `file`, `image` carry `multiple` (`MULTI_CAPABLE_TYPES`; `radio` plus `multiple` is already refused at parse), while `multiselect`, `checkboxes` and `tags` are multi-valued by type with no `multiple` at all (`MULTI_OPTION_TYPES`). Those three split exactly the same way (measured), so they are in the class; judging `multiple: true` alone would have left them grouping per serialization. A multi-valued `lookup` (and `user`) is in scope: it is a `multiple: true` field, and it measured the same (PostgreSQL 500). - **H2: held.** Both layers moved, and the door asks the table. The table's readers: ① the lint rule `validateDatasetMeasureAggregates` (`gating`, run by `os validate` / `os lint` and, since the dataset runtime gate, on a runtime dataset save), which now **refuses at save time** a dataset measure pairing `count_distinct` with a JSON-stored field; ② the analytics dataset compile leg (`assertAggregateFieldTypeCompatible`), `400 DATASET_INVALID`; ③ `measureResultType` (result typing only: a refused pair gets no corrected type); ④ the new engine door; ⑤ the prose of two step-18 migration entries. The narrowing on authored metadata is dataset measures only; the census below finds zero authored `count_distinct` anywhere. Ablation A3 below proves the door reads the table: putting the old row back reds the engine pins. - **H3: measured, a reach remains.** Through `AnalyticsService` wired as `AnalyticsServicePlugin`'s own bridges (`executeAggregate` to `engine.aggregate`, `executeRawSql` to `engine.execute`), after merging `origin/main` `00a92e18da` (which carries the analytics structured-JSON dimension door): - the ObjectQL strategy (the in-memory driver) reaches this door: a cube or dataset dimension on a multi-value field and a cube measure `meta_count_distinct` answer this 400; - `NativeSQLStrategy` on SQLite and PostgreSQL does NOT: a cube or dataset dimension on a multi-value field answers one group per serialized array on SQLite and 500 on PostgreSQL; an inferred cube measure `FIELD_count_distinct` over a JSON-stored field answers 2 on SQLite and 500 on PostgreSQL; a dataset measure `count_distinct` over a `select` declared `multiple: true` (the table sees only `select`) answers 200 on SQLite (distinct serialized arrays) and 500 on PostgreSQL; - a dataset measure `count_distinct` over a JSON-stored TYPE is refused by the compile leg on every driver (`DATASET_INVALID`). The native-SQL reach is reported to the seat, not fixed here. The memory cube (`MemoryAnalyticsService`) has zero constructors outside tests (`git grep "new MemoryAnalyticsService"` excluding tests: 0; including them: 5 files), so it was not measured. - **H4: held; zero hits.** Census below. ## Census (before narrowing) A `groupBy` / grouping / dimension on a multi-value field, and a `count_distinct` on a JSON-stored field, in datasets, reports, views, dashboards, pages and code. - `examples/**` at `42d78b97fe` (unchanged at this head): the four `objectstack.config.ts` apps and `embed-objectql`. Stacks loaded through `tsx` (showcase through its metadata modules, its plugin imports unbuilt) and walked for every grouping-shaped position. - Multi-value fields: todo 1 (`todo_task.tags`), showcase 8 (`showcase_field_zoo.f_multiselect`, `f_checkboxes`, `f_tags`, `f_lookups`, `f_users`; `showcase_project.labels`, `team_members`; `showcase_task.labels`); crm, multi-package and embed-objectql 0. Structured-JSON fields: showcase 10. - Grouping positions naming ANY declared field (the positive control): showcase 74 (kanban `groupByField`, chart `dimensions`, dataset dimensions, page `dimensions`, report `xAxis`). Report `rows` / `columns`: `status`, `priority` (showcase), `status`, `priority`, `owner`, `category` (todo). **Hits naming a multi-value or structured-JSON field: 0.** - `count_distinct`: 0 occurrences in `examples/`. - The published hotcrm stack: `objectstack-ai/hotcrm` cloned at `4ca8e2d4bb` (the repository's `main`; the npm tarball itself was not read). Its dependency tree was not installed, so the census is by source text over `src/**`, `test/**` and `scripts/**`. - Multi-value fields: `crm_knowledge_article.tags` (`select`, `multiple: true`), and `attendee_contacts` / `attendee_users` (`lookup`, `multiple: true`) in an activity action. Structured-JSON fields: `billing_address` (contract, quote, account), `shipping_address` (quote), `address` (lead), `office_location` (account). - Grouping positions: `groupByField` `status`, `owner_id`, `stage`, `crm_account`, `channel`; `xAxis` `stage`, `status`, `priority`, `owner`; report `rows` / `columns` over dataset dimensions `industry`, `type`, `lead_source`, `last_contacted_date`, `stage`, `owner`, `forecast_category`, `close_quarter`; dataset dimension `field:` values none of the fields above. **Hits: 0.** Its `_picklists.ts` already records why a grouped field stays single-valued ("a `multiple: true` column groups by the COMBINATION"). - `count_distinct`: 1 occurrence, a comment in `scripts/analytics-reconcile/reconcile.ts`. **Hits: 0.** - `packages/platform-objects`: 0 `count_distinct`. No hit, so neither narrowing went back to triage. ## Scope beyond the declared surface, and why The claim names `packages/objectql/src/**`, the one spec file, and `rest` pins. Four more places move, each because the ruled table change makes a pin red or a shipped sentence false. They sit in separate commits so each can be judged or dropped on its own: - `packages/lint`: the table-agreement pin `accepts count and count_distinct over every declared FieldType` goes red on the ruled row, so it is triaged (count over every type; count_distinct over every type but the ten, which it refuses). The refusal hint said "`count` / `count_distinct` accept every type"; it now says `count_distinct` accepts every type but the JSON-stored ones. - `packages/services/service-analytics`: the census pin `the refused set … 155 pairs` counts the table's refused pairs, so it moves to 165 with a `distinct` bucket of 10. The compile leg's words for a `count_distinct` refusal fell into the "derives a NUMBER … coerces the stored form" branch and then said "`count`/`count_distinct` accept every type"; `DIVERGENCE_BY_AGGREGATE` gains a `count_distinct` branch (equality, not arithmetic or order), `REMEDY_BY_SOURCE_CLASS` gains the distinct prescription, and the two other sentences stop claiming `count_distinct` accepts every type. A new case pins those words. The dispatch said not to edit `packages/services/**`; the two claims on that package (the analytics dimension door and the analytics seam lowering) had both landed (`00a92e18da`, `793fb839`) before this edit, and no file here overlaps theirs. - `packages/spec/src/migrations/entries/semantic/18.dataset-measure-*.ts` (+ the regenerated `registry.ts`): the min / max entry's route "① … `count` / `count_distinct`, which accept every type" is false for the ten types now; the sum / avg entry's surface and acceptance prose name the `count_distinct` rider this change's ADR-0087 marker points at. - `content/docs/deployment/validating-metadata.mdx`: the sentence "`count`/`count_distinct` are accepted over every type". ## Tests - New `packages/objectql/src/engine-json-stored-group-distinct-door.test.ts` (7 tests, recording driver, so the in-memory cell by construction): every multi-value declaration refused as a `groupBy` with the full envelope, the declaration and the `$contains` route; the `{ field }` form and the first offending position across both classes; every structured-JSON type and multi-value declaration refused as a `count_distinct`, and the first of two offenders named; the REST door into `findData`; controls (scalar group keys and distinct counts, `count` over JSON-stored columns, the `$contains` route itself) reach the driver; GUARDs: the `count_distinct` door agrees with the spec table on every `FieldType` (floor: exactly 10 refused) and refuses every flagged multi-capable type; no verdict without a field map, for an undeclared name, an off-vocabulary type or any other function. - New `packages/rest/src/data-json-stored-group-distinct-door.test.ts`: SQLite always, PostgreSQL / MySQL where `OS_TEST_POSTGRES_URL` / `OS_TEST_MYSQL_URL` are set. Both shapes answer 400 with the route in the REST body and zero reads; the controls (`status` groups, `title` and `status` distinct counts) and the named route (`count` with `where { tags: { $contains } }` answers `a` 3 · `b` 2) are served by the driver. ⚠️ No CI job sets those URLs for this package, so the live cells run only locally. Local run with the private PostgreSQL 16.13: **8 passed (sqlite 4, live postgres 4) / 4 skipped (mysql, no URL)**. - Fixture triage: - `engine-group-by-json-door.test.ts` (the structured-JSON door's pin) named a `multiple: true` select as a CONTROL reaching the driver. That branch is closed now, so the control is **replaced** by a single-value select. Its GUARD over every `FieldType` now expects the multi-option types refused too. - The lint and service-analytics pins above. - Suites, all at `fb239eb2f3` (the merge of `origin/main` `00a92e18da`); the later commits touch only the changeset: - `pnpm --filter @objectstack/objectql test`: **347 files / 6799 passed**. - `pnpm --filter @objectstack/rest test` with the live PostgreSQL URL: **239 files / 4731 passed / 39 skipped**. - `pnpm --filter @objectstack/spec test`: **582 files / 17164 passed / 1 todo**. - `pnpm --filter @objectstack/lint test`: **117 files / 5438 passed**. - `pnpm --filter @objectstack/service-analytics test`: **145 files / 3326 passed**. - `typecheck` for objectql, spec, lint, rest and service-analytics: exit 0. Each `check:test-typecheck` held its ledger (objectql 40 files / 234 errors / 65 signatures, spec 52 / 249 / 137, lint 2 / 6 / 2, rest 0), so the new test files compile. **Reverse verification (three ablations)**, each from committed code through `scripts/ablation-replace.mjs` WRAP (trap-restored), the package rebuilt, and `ablation-dist-preflight` finding the marker in the built files before any reading: | ablation | mutation | predicted | observed | |:--|:--|:--|:--| | A1, multi-value `groupBy` | the `isMultiValueField(...)` class test fed a global flag that is never set (anchor 1 to 0, marker 0 to 1, blob `322c8098d2d1` to `39a2feb82b08`); objectql dist marker in 4 files | red | objectql pins **4 failed / 9 passed** (multi-value groupBy, object form, findData, the structured-JSON GUARD over `FieldType`); rest pins **2 failed / 12 passed / 7 skipped** (sqlite + postgres multi-value groupBy); every count_distinct case and control green | | A2, `count_distinct` door | the engine call fed an always-undefined property instead of `query.aggregations` (blob `cd1e9e39967f` to `3123c092583e`); marker in 4 files | red | objectql **2 failed / 11 passed**; rest **2 failed / 12 passed / 7 skipped** (sqlite + postgres count_distinct); groupBy cases green | | A3, the table row | `count_distinct:` fed the old every-type row (blob `94cf5a2e3f56` to `a343f47f2212`); spec dist marker in 4 files | red, and the engine door red with it | objectql **3 failed / 10 passed** (count_distinct refusal, findData, the table GUARD); rest **2 failed**; spec pin **3 failed / 21 passed**; lint pin **1 failed / 23 passed** | Restore leg for each: blob equals HEAD, `git diff HEAD` empty. After rebuilding spec and objectql, `--absent` found all three markers absent (spec 230 built files, objectql 14), whole-tree porcelain empty, and the pins green again (objectql 13 passed; rest 14 passed / 7 skipped; spec 24; lint 24). ## Gates `node scripts/pm/dispatch-gates.mjs --commands` (no paths) at `f53718f1ee` derived **117** commands over 17 paths vs merge base `00a92e18d`. All 117 were run at `f53718f1ee` with their exit codes recorded, and `--ran` reconciles them: **117 derived, 117 run, 0 NOT-MEASURED, 0 UNRUN, all exit 0**. Among them: - `check:adr-0087-registration --base origin/main`: `not-required (already-registered dataset-measure-aggregate-field-type-refused)` accepted. - `check:changeset-no-major`, `check-empty-changeset` ("No changeset from the merge base modified or deleted by this diff"), `check:doc-authoring`, `check:nul-bytes`, `check:issue-citations`, `check:engine-double-contract`, `check:driver-memory-census`, `check:query-options-erasure`, `check:cross-package-test-inputs`, `check:test-source-alias`, `check:type-check-coverage`. - The `@objectstack/spec` generated-artifact gates, `check:migration-registry`, `check:spec-changes` and `check:upgrade-guide` included. The registry was regenerated for the amended entries, and again after the merge (no diff). - `check:skill-examples`, `check:dual-build-cjs-loads` and `check:type-check-debt` first answered PREREQUISITE NOT MET (exit 3). After `turbo run build --filter='./packages/*' --filter='./packages/*/*'` they answered 0; that build is of `fb239eb2f3`, whose package sources equal this head's. Lint, narrowed and proven at `f53718f1ee`: eslint with inline config disabled, over the 15 changed `.ts` files, found **15 files, 0 errors, 0 warnings**. Three facts make this narrowing a measurement: - The population comes from eslint's own config: `isPathIgnored` is `false` for all 15. - The count comes from the results: 15. - Untouched files cannot change verdict: `parserOptions.project` and `projectService` are `null` for every file, so type-aware linting is not enabled. ## Changeset `.changeset/20808-json-stored-group-distinct-refused.md`: - `@objectstack/objectql` `minor` and `@objectstack/spec` `minor`, each with its own **BREAKING** banner. `@objectstack/lint` and `@objectstack/service-analytics` are `patch`, for their words. - `Clause-②: no (narrowing)`, and exactly one ADR-0087 marker: `not-required (already-registered dataset-measure-aggregate-field-type-refused)`. The table row is that family's narrowing, and the non-temporal sum / avg narrowing rode the same id the same way; this diff amends that entry's prose to name the rider. The engine-door halves refuse a query shape, not a stored one. - It names the two shapes the structured-JSON `groupBy` entry of this same release lists as unchanged. That pending note is left as it landed: rewriting it reds `check-empty-changeset` for a human's confirmation. No export or published type changes (`assertGroupByNamesNoJsonStoredField` and the new door are internal; objectql's root and `./core` exports are unchanged). ## Acceptance notes - **Reported to the PM, not filed here (the same family: a JSON-stored column as a group, distinct or aggregate key, answering per driver with a PostgreSQL 500):** - The native-SQL analytics reach (H3 above). Landing: `packages/services/service-analytics`, beside its structured-JSON dimension door. - `min` / `max` over a JSON-stored field at the engine door: `POST /api/v1/data/:object/query` with `max(meta)` answered `{a:1}` on memory, `'{"b":1}'` (a string) on SQLite and 500 on PostgreSQL (function max(json) does not exist), and `max(tags)` the same way. The table already refuses those pairs, and its dataset legs enforce that; the engine enforces only the `count_distinct` row, as ruled here. - A single-value media field (`file`, `image`, `avatar`, `video`, `audio`) is a JSON column on a deployment that has not moved its media columns (`JSON_COLUMN_TYPES`' header in driver-sql), so a `groupBy` or `count_distinct` on one there would split the same way. This is by reading, not measured: a fresh deployment creates the string column. Not judged here, because the type alone cannot tell the two deployments apart. - A refusal relayed through the analytics ObjectQL strategy names the engine's position (`groupBy[0]`, `aggregations[0].field`), not the cube member the caller wrote. The same note was made on the structured-JSON door. --- _Generated by [Claude Code](https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 0efbdc3 commit 975b248

17 files changed

Lines changed: 1067 additions & 81 deletions
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
"@objectstack/objectql": minor
3+
"@objectstack/spec": minor
4+
"@objectstack/lint": patch
5+
"@objectstack/service-analytics": patch
6+
---
7+
8+
fix(objectql,spec)!: a `groupBy` on a multi-value field and a `count_distinct` on a JSON-stored field are refused with `INVALID_FIELD` / 400 at the engine's `aggregate`, on every driver, and the aggregate × field-type table stops accepting `count_distinct` over the JSON-stored types
9+
10+
Clause-②: no (narrowing)
11+
12+
<!-- adr-0087: not-required (already-registered dataset-measure-aggregate-field-type-refused) the one metadata-facing half of this change is a row of AGGREGATE_FIELD_TYPE_COMPATIBILITY narrowing, and that family's hand-migration is already registered under protocol major 18 by this id: "an aggregate the field's type accepts, per AGGREGATE_FIELD_TYPE_COMPATIBILITY", with every refused pair of the table refused at the compile door. The non-temporal sum / avg narrowing rode the same id the same way; this diff amends that entry's surface and acceptance prose to name the count_distinct rider, and corrects the min / max entry's route that called count_distinct valid over every type. The engine-door halves refuse a query shape, not a stored one: no authorable key, export or stored row moves. -->
13+
14+
**BREAKING** (`@objectstack/objectql`): this narrows what `aggregate` accepts, in two positions, on every driver and for every caller that reaches the engine (the REST query door, a flow or hook, and the analytics strategy that lowers a cube query onto `engine.aggregate`). Shipped as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
15+
16+
- A `groupBy` entry that names a **multi-value** field: an inherently-multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. Both entry spellings are judged, the field name and the `{ field }` object.
17+
- A `count_distinct` aggregation over a **JSON-stored** field: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently-multi option type, or a multi-capable field declared `multiple: true`.
18+
19+
**BREAKING** (`@objectstack/spec`): `AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct` no longer lists the ten JSON-stored types (the structured-JSON seven and `multiselect`, `checkboxes`, `tags`), so `isAggregateCompatibleWithFieldType('count_distinct', type)` answers `false` for them. Every reader of the table refuses those pairs now: the dataset-measure lint rule (`measure-aggregate-field-type-refused`, run by `os validate` and at a runtime dataset save), the analytics dataset compile leg (`400 DATASET_INVALID`), and the engine door above. The `count` row is unchanged.
20+
21+
**What an author sees now.** `400 INVALID_FIELD`, naming the position (`groupBy[0]`, `groupBy[0].field`, or `aggregations[0].field`), the field and its declaration, saying the query was not run, and naming the route inside the first 500 characters the REST door keeps. For a multi-value field the route is to filter by one member: `where` with `$contains` on the field, one query per member. For a structured-JSON field it is to store the part you count in a field of its own, or to count rows with `count`. The thrown error carries `field`, `fields`, `object` and `param` (`groupBy` or `aggregations`).
22+
23+
**Why a refusal.** Every SQL driver stores these values in a JSON column, and the drivers share no meaning for one as a group key or a distinct key. Measured through `POST /api/v1/data/:object/query` over three rows: grouping by any of the eight multi-value declarations answered one group per array on the in-memory driver, one group per serialized array on SQLite, and 500 `DATABASE_ERROR` on PostgreSQL 16. `count_distinct` over any structured-JSON or multi-value field answered 3 on the in-memory driver (equal values counted apart), 2 on SQLite (serialized text compared), and 500 on PostgreSQL (no equality operator for `json`). No example app and no published stack groups by a multi-value field or counts one distinct, so no meaning is defined for either here.
24+
25+
**What to write instead.** A dataset measure or a query that counted a JSON-stored field distinct: use `count` over it, or store the scalar part you meant to count in a field of its own and `count_distinct` that field. A grouping by a multi-value field: filter by each member with `$contains` and count.
26+
27+
**Who is affected.** A caller that grouped by a multi-value field, or counted a JSON-stored field distinct, on the in-memory driver or on SQLite and read the answer as a real one; on PostgreSQL both were already a 500. A dataset whose measure pairs `count_distinct` with a JSON-stored field is refused by the lint rule and the compile leg.
28+
29+
**Unchanged.** (Two shapes the structured-JSON `groupBy` entry of this same release lists as unchanged are narrowed here: a `multiple: true` select as a group key, and `count_distinct` over a structured-JSON field. This entry is the later word on both.) A `groupBy` or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; the `having`, filter and sort positions; and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached.
30+
31+
`@objectstack/lint`: the dataset-measure refusal's hint no longer says `count_distinct` accepts every type.
32+
33+
`@objectstack/service-analytics`: the dataset compile leg's refusal of a `count_distinct` measure over a JSON-stored field says why it diverges (the drivers compare the values for equality three ways) and prescribes `count`, or a scalar field for the part being counted; its other refusals no longer say `count_distinct` accepts every type.

‎content/docs/deployment/validating-metadata.mdx‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -206,9 +206,11 @@ measures: [{ name: 'avg_closed', aggregate: 'avg', field: 'closed_at' }]
206206
`avg` over a temporal column is where this bites hardest: one SQL family coerces
207207
the stored text and returns a plausible number (the average *year*), another has
208208
no such function and fails at query time. `min`/`max` over the same field are
209-
**accepted** — they return a real instant of the field's own type — and
210-
`count`/`count_distinct` are accepted over every type, because they read no
211-
arithmetic off the value. The analytics service refuses the same pair with
209+
**accepted** — they return a real instant of the field's own type — `count` is
210+
accepted over every type, because it reads no value, and `count_distinct` over
211+
every type but the JSON-stored ones (the structured-JSON types and
212+
`multiselect` / `checkboxes` / `tags`), whose values no two backends compare
213+
for equality alike. The analytics service refuses the same pair with
212214
`400 DATASET_INVALID` when a query is built; this is the identical verdict,
213215
from the identical table, one door earlier. The rule stays silent wherever the
214216
field's type cannot be resolved (an object this stack does not define, a

‎packages/lint/src/validate-dataset-measure-aggregates.test.ts‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -165,13 +165,28 @@ describe('measure-aggregate-field-type-refused — stays silent on every pair th
165165
}
166166
});
167167

168-
it('accepts count and count_distinct over every declared FieldType', () => {
168+
it('accepts count over every declared FieldType', () => {
169169
for (const fieldType of FieldType.options) {
170-
for (const aggregate of ['count', 'count_distinct']) {
171-
expect(findings(stackWith(aggregate, fieldType)), `${aggregate}(${fieldType})`).toEqual([]);
172-
}
170+
expect(findings(stackWith('count', fieldType)), `count(${fieldType})`).toEqual([]);
173171
}
174172
});
173+
174+
// [#20808] `count_distinct` over a JSON-stored type left the table's row: no
175+
// two backends compare those values for equality alike (the in-memory driver
176+
// counted equal documents apart, SQLite compared serialized text, PostgreSQL
177+
// answered 500). Every other declared type stays accepted.
178+
it('accepts count_distinct over every declared FieldType except the JSON-stored ones, which it refuses', () => {
179+
const jsonStored = ['json', 'composite', 'repeater', 'record', 'location', 'address', 'vector', 'multiselect', 'checkboxes', 'tags'];
180+
for (const fieldType of FieldType.options) {
181+
const found = findings(stackWith('count_distinct', fieldType));
182+
expect(found.length, `count_distinct(${fieldType})`).toBe(jsonStored.includes(fieldType) ? 1 : 0);
183+
}
184+
const [issue] = findings(stackWith('count_distinct', 'json'));
185+
expect(issue.rule).toBe(RULE);
186+
// The way out names `count`, and no sentence says count_distinct accepts every type.
187+
expect(issue.hint).toContain('accepts: count.');
188+
expect(issue.hint).not.toMatch(/count_distinct` accept every type/);
189+
});
175190
});
176191

177192
describe('measure-aggregate-field-type-refused — the rule IS the table, on every pair', () => {

‎packages/lint/src/validate-dataset-measure-aggregates.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -232,8 +232,9 @@ export function validateDatasetMeasureAggregates(stack: unknown): DatasetMeasure
232232
`Either point "${aggregate}" at a field of an accepted type, or aggregate ` +
233233
`"${field}" with one its \`${fieldType}\` type accepts: ` +
234234
`${aggregatesAccepting(fieldType).join(', ')}. ` +
235-
`\`count\` / \`count_distinct\` accept every type because they read no arithmetic off ` +
236-
`the value; a quantity that must be added up or averaged has to be STORED as a ` +
235+
`\`count\` accepts every type because it reads no value, and \`count_distinct\` every ` +
236+
`type but the JSON-stored ones, whose values no two backends compare alike; a ` +
237+
`quantity that must be added up or averaged has to be STORED as a ` +
237238
`numeric field (a computed column) and aggregated as one. The compile leg refuses ` +
238239
`this same pair with \`400 DATASET_INVALID\` before any SQL is emitted, so this is ` +
239240
`the same fix made earlier.`,
Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#20808] A `count_distinct` aggregation over a JSON-STORED field is refused
5+
* with `INVALID_FIELD` / 400 by the engine's `aggregate`, naming the field,
6+
* its declared type and the position, before any driver is asked.
7+
*
8+
* ## What ran before this door, measured on `origin/main` `42d78b97fe`
9+
*
10+
* Through `POST /api/v1/data/:object/query` (`{ aggregations: [{ function:
11+
* 'count_distinct', field: FIELD, alias: 'n' }] }`) over three rows, two of
12+
* which hold equal values under the counted field:
13+
*
14+
* | counted field | InMemoryDriver | SqlDriver, SQLite | SqlDriver, PostgreSQL 16 |
15+
* |:--|:--|:--|:--|
16+
* | a `text` or single-value `select` (the control) | 2 | 2 | 2 |
17+
* | a structured-JSON field (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) | 3 | 2 (3 for `json`, whose three documents differ) | **500 `DATABASE_ERROR`** |
18+
* | a multi-value field (`multiselect`, `checkboxes`, `tags`; `select`, `lookup`, `user`, `file`, `image` with `multiple: true`) | 3 | 2 | **500 `DATABASE_ERROR`** |
19+
*
20+
* The in-memory driver compares each row's value by identity, so equal
21+
* documents count apart; SQLite compares the serialized text; PostgreSQL has
22+
* no equality operator for a `json` column ("could not identify an equality
23+
* operator for type json") and answers 500. One query, three answers.
24+
*
25+
* ## Whose verdict it is
26+
*
27+
* The TYPE half is the spec table's: `AGGREGATE_FIELD_TYPE_COMPATIBILITY`'s
28+
* `count_distinct` row, asked through `isAggregateCompatibleWithFieldType`
29+
* (`@objectstack/spec/data`) — ⛔ never a second list here. The triage
30+
* direction on #20808: the table "stops accepting it" for the structured-JSON
31+
* class, on the table's own ground ("can every backend give one answer"); the
32+
* row refuses the three multi-option types too, on the same measurement.
33+
*
34+
* The DECLARATION half is `isMultiValueField`: a multi-capable type flagged
35+
* `multiple: true` (`select`, `lookup`, `user`, `file`, `image`) is stored in
36+
* a JSON column like the rest, but a per-TYPE table cannot see the flag. The
37+
* two answers are asked side by side, so a field is refused if either one
38+
* refuses it.
39+
*
40+
* ⛔ No per-backend JSON distinctness is defined to make the pair answerable:
41+
* no caller of it was measured (no dataset measure, widget or `count_distinct`
42+
* in `examples/` or the published hotcrm stack counts one).
43+
*
44+
* ## Where it stands, and what it judges
45+
*
46+
* At the entry of `aggregate`, right after the `groupBy` door
47+
* (`group-by-structured-json-door.ts`), before the per-aggregation `filter`
48+
* doors and before any driver is resolved — so it holds for every caller that
49+
* reaches the engine: the REST query door, a flow, a hook, and the analytics
50+
* strategy that lowers a cube measure onto `engine.aggregate`.
51+
*
52+
* **Not judged:** any other aggregate function (`count` compares no value; the
53+
* table's other rows have their own doors and are not this card's), a
54+
* `count_distinct` with no named field, an undeclared name, a registry-less
55+
* host (no field map, no verdict), and a declared type outside `FieldType`
56+
* (a driver-internal alias such as `string` or `integer` on an introspected
57+
* object): the table is fail-closed on vocabulary, and "cannot answer, do not
58+
* block" is this consumer's tier, as the table's own TSDoc says.
59+
*
60+
* `INVALID_FIELD`, the code the `groupBy` door beside it answers: the verdict
61+
* is about the NAMED field's type at a position.
62+
*
63+
* @see https://github.com/objectstack-ai/objectstack/issues/20808
64+
*/
65+
66+
import { StandardErrorCode } from '@objectstack/spec/api';
67+
import { FieldType, isAggregateCompatibleWithFieldType, isMultiValueField } from '@objectstack/spec/data';
68+
69+
/** The declared `FieldType` vocabulary — the only types the table can answer for. */
70+
const DECLARED_FIELD_TYPES: ReadonlySet<string> = new Set(FieldType.options);
71+
72+
/** One `count_distinct` aggregation that names a JSON-stored field. */
73+
interface JsonStoredDistinctTarget {
74+
readonly field: string;
75+
readonly type: string;
76+
/** The declared field carries `multiple: true` (said in the words). */
77+
readonly multiple: boolean;
78+
/** A multi-value field (the route differs: filter by one member). */
79+
readonly multiValue: boolean;
80+
/** `aggregations[i].field`. */
81+
readonly position: string;
82+
}
83+
84+
/**
85+
* The `count_distinct` aggregations that name a declared field the table's
86+
* `count_distinct` row refuses, or a declared multi-value field, in order.
87+
*/
88+
function jsonStoredDistinctTargets(
89+
fields: Record<string, unknown>,
90+
aggregations: readonly unknown[],
91+
): JsonStoredDistinctTarget[] {
92+
const hits: JsonStoredDistinctTarget[] = [];
93+
for (const [i, agg] of aggregations.entries()) {
94+
if (agg === null || typeof agg !== 'object' || Array.isArray(agg)) continue;
95+
if ((agg as { function?: unknown }).function !== 'count_distinct') continue;
96+
const field = (agg as { field?: unknown }).field;
97+
if (typeof field !== 'string' || field === '*') continue;
98+
if (!Object.prototype.hasOwnProperty.call(fields, field)) continue;
99+
const def = fields[field] as { type?: unknown; multiple?: unknown } | undefined;
100+
const type = def?.type;
101+
if (typeof type !== 'string' || !DECLARED_FIELD_TYPES.has(type)) continue;
102+
const multiple = def?.multiple === true;
103+
const multiValue = isMultiValueField({ type, multiple });
104+
if (isAggregateCompatibleWithFieldType('count_distinct', type) && !multiValue) continue;
105+
hits.push({ field, type, multiple, multiValue, position: `aggregations[${i}].field` });
106+
}
107+
return hits;
108+
}
109+
110+
/**
111+
* Refuse a `count_distinct` aggregation over a declared JSON-stored field —
112+
* `INVALID_FIELD` / 400, before any driver is asked. See the module header.
113+
*
114+
* The words put the position and the verdict first, then that the query did
115+
* not run, then the route, then the reason: the REST door keeps the first 500
116+
* characters of a 4xx message (`CLIENT_MESSAGE_MAX`), and the route must be
117+
* inside them.
118+
*/
119+
export function assertCountDistinctNamesNoJsonStoredField(
120+
object: string,
121+
schema: unknown,
122+
aggregations: unknown,
123+
): void {
124+
if (!Array.isArray(aggregations) || aggregations.length === 0) return;
125+
const fields = (schema as { fields?: unknown } | undefined)?.fields;
126+
if (!fields || typeof fields !== 'object') return;
127+
const hits = jsonStoredDistinctTargets(fields as Record<string, unknown>, aggregations);
128+
if (hits.length === 0) return;
129+
const [first] = hits;
130+
const declared = first.multiple ? `${first.type} field with multiple: true` : `${first.type} field`;
131+
const kind = first.multiValue ? 'a multi-value field' : 'a structured-JSON value';
132+
const route = first.multiValue
133+
? 'Count the records that hold one member instead: count with '
134+
+ `where { "${first.field}": { "$contains": VALUE } }, one query per member.`
135+
: 'Count distinct values of a field that stores one scalar value: store the part you count in a '
136+
+ 'field of its own and count_distinct that field, or count the rows with count.';
137+
const err = new Error(
138+
`aggregate('${object}'): ${first.position} counts distinct '${first.field}', a declared ${declared} `
139+
+ `— ${kind}, which the engine does not count distinct`
140+
+ (hits.length > 1 ? ` (also: ${hits.slice(1).map((h) => `'${h.field}'`).join(', ')})` : '')
141+
+ `. The query was NOT run. ${route} `
142+
+ 'A JSON-stored value is no distinct key the drivers share: one counted every row apart, one '
143+
+ 'compared the serialized text, one refused the statement.',
144+
) as Error & {
145+
code?: string; status?: number; httpStatus?: number;
146+
field?: string; fields?: string[]; object?: string; param?: string;
147+
};
148+
err.code = StandardErrorCode.enum.INVALID_FIELD;
149+
err.status = 400;
150+
// …and `httpStatus`, the same number under ADR-0112 D5's spelling — what a
151+
// consumer holding the THROWN error reads; `status` stays for the HTTP doors.
152+
err.httpStatus = 400;
153+
err.field = first.field;
154+
err.fields = hits.map((h) => h.field);
155+
err.object = object;
156+
err.param = 'aggregations';
157+
throw err;
158+
}

0 commit comments

Comments
 (0)