From 024fcb9c249053e861545e1f3fd2468a8d56fa88 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 15:51:36 +0000 Subject: [PATCH 1/8] wip(objectql,spec): refuse groupBy on a multi-value field and count_distinct on a JSON-stored field at the engine aggregate door (#20808) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- .../src/count-distinct-json-stored-door.ts | 158 ++++++++++ .../src/engine-group-by-json-door.test.ts | 25 +- ...ne-json-stored-group-distinct-door.test.ts | 294 ++++++++++++++++++ packages/objectql/src/engine.ts | 16 +- .../src/group-by-structured-json-door.ts | 114 +++++-- ...aggregate-field-type-compatibility.test.ts | 44 ++- .../aggregate-field-type-compatibility.ts | 59 +++- 7 files changed, 659 insertions(+), 51 deletions(-) create mode 100644 packages/objectql/src/count-distinct-json-stored-door.ts create mode 100644 packages/objectql/src/engine-json-stored-group-distinct-door.test.ts diff --git a/packages/objectql/src/count-distinct-json-stored-door.ts b/packages/objectql/src/count-distinct-json-stored-door.ts new file mode 100644 index 00000000000..e14c10f4377 --- /dev/null +++ b/packages/objectql/src/count-distinct-json-stored-door.ts @@ -0,0 +1,158 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#20808] A `count_distinct` aggregation over a JSON-STORED field is refused + * with `INVALID_FIELD` / 400 by the engine's `aggregate`, naming the field, + * its declared type and the position, before any driver is asked. + * + * ## What ran before this door, measured on `origin/main` `42d78b97fe` + * + * Through `POST /api/v1/data/:object/query` (`{ aggregations: [{ function: + * 'count_distinct', field: FIELD, alias: 'n' }] }`) over three rows, two of + * which hold equal values under the counted field: + * + * | counted field | InMemoryDriver | SqlDriver, SQLite | SqlDriver, PostgreSQL 16 | + * |:--|:--|:--|:--| + * | a `text` or single-value `select` (the control) | 2 | 2 | 2 | + * | a structured-JSON field (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) | 3 | 2 (3 for `json`, whose three documents differ) | **500 `DATABASE_ERROR`** | + * | a multi-value field (`multiselect`, `checkboxes`, `tags`; `select`, `lookup`, `user`, `file`, `image` with `multiple: true`) | 3 | 2 | **500 `DATABASE_ERROR`** | + * + * The in-memory driver compares each row's value by identity, so equal + * documents count apart; SQLite compares the serialized text; PostgreSQL has + * no equality operator for a `json` column ("could not identify an equality + * operator for type json") and answers 500. One query, three answers. + * + * ## Whose verdict it is + * + * The TYPE half is the spec table's: `AGGREGATE_FIELD_TYPE_COMPATIBILITY`'s + * `count_distinct` row, asked through `isAggregateCompatibleWithFieldType` + * (`@objectstack/spec/data`) — ⛔ never a second list here. The triage + * direction on #20808: the table "stops accepting it" for the structured-JSON + * class, on the table's own ground ("can every backend give one answer"); the + * row refuses the three multi-option types too, on the same measurement. + * + * The DECLARATION half is `isMultiValueField`: a multi-capable type flagged + * `multiple: true` (`select`, `lookup`, `user`, `file`, `image`) is stored in + * a JSON column like the rest, but a per-TYPE table cannot see the flag. The + * two answers are asked side by side, so a field is refused if either one + * refuses it. + * + * ⛔ No per-backend JSON distinctness is defined to make the pair answerable: + * no caller of it was measured (no dataset measure, widget or `count_distinct` + * in `examples/` or the published hotcrm stack counts one). + * + * ## Where it stands, and what it judges + * + * At the entry of `aggregate`, right after the `groupBy` door + * (`group-by-structured-json-door.ts`), before the per-aggregation `filter` + * doors and before any driver is resolved — so it holds for every caller that + * reaches the engine: the REST query door, a flow, a hook, and the analytics + * strategy that lowers a cube measure onto `engine.aggregate`. + * + * **Not judged:** any other aggregate function (`count` compares no value; the + * table's other rows have their own doors and are not this card's), a + * `count_distinct` with no named field, an undeclared name, a registry-less + * host (no field map, no verdict), and a declared type outside `FieldType` + * (a driver-internal alias such as `string` or `integer` on an introspected + * object): the table is fail-closed on vocabulary, and "cannot answer, do not + * block" is this consumer's tier, as the table's own TSDoc says. + * + * `INVALID_FIELD`, the code the `groupBy` door beside it answers: the verdict + * is about the NAMED field's type at a position. + * + * @see https://github.com/objectstack-ai/objectstack/issues/20808 + */ + +import { StandardErrorCode } from '@objectstack/spec/api'; +import { FieldType, isAggregateCompatibleWithFieldType, isMultiValueField } from '@objectstack/spec/data'; + +/** The declared `FieldType` vocabulary — the only types the table can answer for. */ +const DECLARED_FIELD_TYPES: ReadonlySet = new Set(FieldType.options); + +/** One `count_distinct` aggregation that names a JSON-stored field. */ +interface JsonStoredDistinctTarget { + readonly field: string; + readonly type: string; + /** The declared field carries `multiple: true` (said in the words). */ + readonly multiple: boolean; + /** A multi-value field (the route differs: filter by one member). */ + readonly multiValue: boolean; + /** `aggregations[i].field`. */ + readonly position: string; +} + +/** + * The `count_distinct` aggregations that name a declared field the table's + * `count_distinct` row refuses, or a declared multi-value field, in order. + */ +function jsonStoredDistinctTargets( + fields: Record, + aggregations: readonly unknown[], +): JsonStoredDistinctTarget[] { + const hits: JsonStoredDistinctTarget[] = []; + for (const [i, agg] of aggregations.entries()) { + if (agg === null || typeof agg !== 'object' || Array.isArray(agg)) continue; + if ((agg as { function?: unknown }).function !== 'count_distinct') continue; + const field = (agg as { field?: unknown }).field; + if (typeof field !== 'string' || field === '*') continue; + if (!Object.prototype.hasOwnProperty.call(fields, field)) continue; + const def = fields[field] as { type?: unknown; multiple?: unknown } | undefined; + const type = def?.type; + if (typeof type !== 'string' || !DECLARED_FIELD_TYPES.has(type)) continue; + const multiple = def?.multiple === true; + const multiValue = isMultiValueField({ type, multiple }); + if (isAggregateCompatibleWithFieldType('count_distinct', type) && !multiValue) continue; + hits.push({ field, type, multiple, multiValue, position: `aggregations[${i}].field` }); + } + return hits; +} + +/** + * Refuse a `count_distinct` aggregation over a declared JSON-stored field — + * `INVALID_FIELD` / 400, before any driver is asked. See the module header. + * + * The words put the position and the verdict first, then that the query did + * not run, then the route, then the reason: the REST door keeps the first 500 + * characters of a 4xx message (`CLIENT_MESSAGE_MAX`), and the route must be + * inside them. + */ +export function assertCountDistinctNamesNoJsonStoredField( + object: string, + schema: unknown, + aggregations: unknown, +): void { + if (!Array.isArray(aggregations) || aggregations.length === 0) return; + const fields = (schema as { fields?: unknown } | undefined)?.fields; + if (!fields || typeof fields !== 'object') return; + const hits = jsonStoredDistinctTargets(fields as Record, aggregations); + if (hits.length === 0) return; + const [first] = hits; + const declared = first.multiple ? `${first.type} field with multiple: true` : `${first.type} field`; + const kind = first.multiValue ? 'a multi-value field' : 'a structured-JSON value'; + const route = first.multiValue + ? 'Count the records that hold one member instead: count with ' + + `where { "${first.field}": { "$contains": VALUE } }, one query per member.` + : 'Count distinct values of a field that stores one scalar value: store the part you count in a ' + + 'field of its own and count_distinct that field, or count the rows with count.'; + const err = new Error( + `aggregate('${object}'): ${first.position} counts distinct '${first.field}', a declared ${declared} ` + + `— ${kind}, which the engine does not count distinct` + + (hits.length > 1 ? ` (also: ${hits.slice(1).map((h) => `'${h.field}'`).join(', ')})` : '') + + `. The query was NOT run. ${route} ` + + 'A JSON-stored value is no distinct key the drivers share: one counted every row apart, one ' + + 'compared the serialized text, one refused the statement.', + ) as Error & { + code?: string; status?: number; httpStatus?: number; + field?: string; fields?: string[]; object?: string; param?: string; + }; + err.code = StandardErrorCode.enum.INVALID_FIELD; + err.status = 400; + // …and `httpStatus`, the same number under ADR-0112 D5's spelling — what a + // consumer holding the THROWN error reads; `status` stays for the HTTP doors. + err.httpStatus = 400; + err.field = first.field; + err.fields = hits.map((h) => h.field); + err.object = object; + err.param = 'aggregations'; + throw err; +} diff --git a/packages/objectql/src/engine-group-by-json-door.test.ts b/packages/objectql/src/engine-group-by-json-door.test.ts index ee8a3571ee9..95fb0bd087f 100644 --- a/packages/objectql/src/engine-group-by-json-door.test.ts +++ b/packages/objectql/src/engine-group-by-json-door.test.ts @@ -27,9 +27,9 @@ import { describe, it, expect, beforeEach } from 'vitest'; import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; -import { FieldType, STRUCTURED_JSON_TYPES, type EngineAggregateOptions } from '@objectstack/spec/data'; +import { FieldType, STRUCTURED_JSON_TYPES, isMultiValueField, type EngineAggregateOptions } from '@objectstack/spec/data'; import { ObjectQL } from './engine.js'; -import { assertGroupByNamesNoStructuredJsonField } from './group-by-structured-json-door.js'; +import { assertGroupByNamesNoJsonStoredField } from './group-by-structured-json-door.js'; const OBJECT = 'group_by_json_probe'; @@ -50,7 +50,7 @@ const PROBE = { fields: { title: { name: 'title', type: 'text' }, amount: { name: 'amount', type: 'number' }, - tags: { name: 'tags', type: 'select', multiple: true, options: [{ label: 'A', value: 'a' }] }, + status: { name: 'status', type: 'select', options: [{ label: 'A', value: 'a' }] }, photo: { name: 'photo', type: 'image' }, ...Object.fromEntries(JSONS.map(([name, type]) => [name, { name, type }])), }, @@ -142,8 +142,11 @@ describe('[#20783] a groupBy on a structured-JSON field, at the engine\'s aggreg expect(reads).toHaveLength(0); }); - it('CONTROL a text, number, multi-value select, file or undeclared groupBy reaches the driver, never this refusal', async () => { - for (const field of ['title', 'amount', 'tags', 'photo', 'not_declared']) { + it('CONTROL a text, number, single-value select, file or undeclared groupBy reaches the driver, never this refusal', async () => { + // [#20808] The multi-value select this control named is refused now (the + // second class, `engine-json-stored-group-distinct-door.test.ts`); a + // single-value select is the scalar control in its place. + for (const field of ['title', 'amount', 'status', 'photo', 'not_declared']) { const before = reads.length; const out = await engine.aggregate(OBJECT, { groupBy: [field], aggregations: COUNT }).then( () => null, @@ -170,23 +173,27 @@ describe('[#20783] a groupBy on a structured-JSON field, at the engine\'s aggreg expect(reads).toHaveLength(0); }); - it('GUARD the judged types are exactly the spec\'s STRUCTURED_JSON_TYPES, over every FieldType', () => { + it('GUARD the judged types are exactly the spec\'s STRUCTURED_JSON_TYPES plus the multi-value ones, over every FieldType', () => { + // [#20808] The inherently-multi option types (`multiselect`, `checkboxes`, + // `tags`) are refused without `multiple` — `isMultiValueField` answers for + // them by type; the flagged multi-capable types are this guard's twin in + // the #20808 suite. for (const type of FieldType.options) { const thrown = (() => { try { - assertGroupByNamesNoStructuredJsonField(OBJECT, { fields: { f: { type } } }, ['f']); + assertGroupByNamesNoJsonStoredField(OBJECT, { fields: { f: { type } } }, ['f']); return null; } catch (e) { return e as Thrown; } })(); expect(thrown === null ? null : envelopeOf(thrown), type) - .toEqual(STRUCTURED_JSON_TYPES.has(type) ? ENVELOPE : null); + .toEqual(STRUCTURED_JSON_TYPES.has(type) || isMultiValueField({ type }) ? ENVELOPE : null); } }); it('GUARD no verdict without a field map, for an undeclared name, or for an entry that names no field', () => { - const judge = (schema: unknown, groupBy: unknown) => () => assertGroupByNamesNoStructuredJsonField(OBJECT, schema, groupBy); + const judge = (schema: unknown, groupBy: unknown) => () => assertGroupByNamesNoJsonStoredField(OBJECT, schema, groupBy); expect(judge(undefined, ['meta'])).not.toThrow(); expect(judge({}, ['meta'])).not.toThrow(); expect(judge(PROBE, ['nope', { field: 'nope' }, { dateGranularity: 'month' }, 7, null])).not.toThrow(); diff --git a/packages/objectql/src/engine-json-stored-group-distinct-door.test.ts b/packages/objectql/src/engine-json-stored-group-distinct-door.test.ts new file mode 100644 index 00000000000..a6f87940213 --- /dev/null +++ b/packages/objectql/src/engine-json-stored-group-distinct-door.test.ts @@ -0,0 +1,294 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#20808] Two more JSON-stored keys are refused `INVALID_FIELD` / 400 by + * `engine.aggregate`, naming the field, its declared type and the position, + * before any driver is asked: + * + * - a `groupBy` entry naming a MULTI-VALUE field (`isMultiValueField`: the + * inherently-multi option types, and a multi-capable type flagged + * `multiple: true`) — `group-by-structured-json-door.ts`, its second class; + * - a `count_distinct` over a field the spec table's `count_distinct` row + * refuses (the structured-JSON class and the multi-option types), or over a + * multi-value field — `count-distinct-json-stored-door.ts`. + * + * Measured on the base (`origin/main` `42d78b97fe`) through + * `POST /api/v1/data/:object/query` over three rows: + * + * | query | InMemoryDriver | SqlDriver, SQLite | SqlDriver, PostgreSQL 16 | + * |:--|:--|:--|:--| + * | `groupBy` a single-value `select` (the control) | 200, one group per value | same | same | + * | `groupBy` any multi-value field (8 declarations) | 200, one group per array | 200, one group per serialized array | 500 `DATABASE_ERROR` | + * | `count_distinct` a `text` (the control) | 2 | 2 | 2 | + * | `count_distinct` any structured-JSON or multi-value field | 3 | 2 | 500 `DATABASE_ERROR` | + * + * The InMemoryDriver cell is this suite's recording driver by construction: + * the doors answer before a driver is resolved, so no read runs. The SQL cells + * over a real driver live in `@objectstack/rest`'s + * `data-json-stored-group-distinct-door.test.ts`; that driver's test consumers + * are a ruled, closed census (`check:driver-memory-census`), so no new suite + * of it here. + */ + +import { describe, it, expect, beforeEach } from 'vitest'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { + FieldType, + MULTI_CAPABLE_TYPES, + isAggregateCompatibleWithFieldType, + isMultiValueField, + type EngineAggregateOptions, +} from '@objectstack/spec/data'; +import { ObjectQL } from './engine.js'; +import { assertCountDistinctNamesNoJsonStoredField } from './count-distinct-json-stored-door.js'; + +const OBJECT = 'json_stored_key_probe'; + +const OPTIONS = [{ label: 'A', value: 'a' }, { label: 'B', value: 'b' }]; + +/** field · declared type · `multiple: true` — every multi-value declaration the spec admits. */ +const MULTIS: ReadonlyArray = [ + ['tags', 'select', true], + ['labels', 'tags', false], + ['ms', 'multiselect', false], + ['cb', 'checkboxes', false], + ['refs', 'lookup', true], + ['watchers', 'user', true], + ['files', 'file', true], + ['imgs', 'image', true], +]; + +/** field · declared type — one field of every structured-JSON type. */ +const JSONS: ReadonlyArray = [ + ['meta', 'json'], + ['spec', 'composite'], + ['rep', 'repeater'], + ['rec', 'record'], + ['loc', 'location'], + ['ship_to', 'address'], + ['vec', 'vector'], +]; + +const PROBE = { + name: OBJECT, + label: 'JSON-stored key probe', + fields: { + title: { name: 'title', type: 'text' }, + amount: { name: 'amount', type: 'number' }, + status: { name: 'status', type: 'select', options: OPTIONS }, + owner: { name: 'owner', type: 'lookup', reference: 'json_stored_key_target' }, + photo: { name: 'photo', type: 'image' }, + ...Object.fromEntries(MULTIS.map(([name, type, multiple]) => [ + name, + { + name, + type, + ...(multiple ? { multiple: true } : {}), + ...(type === 'select' || type === 'multiselect' || type === 'checkboxes' ? { options: OPTIONS } : {}), + ...(type === 'lookup' ? { reference: 'json_stored_key_target' } : {}), + }, + ])), + ...Object.fromEntries(JSONS.map(([name, type]) => [name, { name, type }])), + }, +}; + +const COUNT = [{ function: 'count', alias: 'n' }] as EngineAggregateOptions['aggregations']; +const distinct = (field: string) => + [{ function: 'count_distinct', field, alias: 'n' }] as EngineAggregateOptions['aggregations']; + +interface SeenRead { ast: any } + +/** Minimal recording driver — the same witness shape as the sibling door suites. */ +function makeRecordingDriver() { + const rows = new Map>(); + const reads: SeenRead[] = []; + const run = (_ast: any) => [...rows.values()]; + const driver: any = { + name: 'recording', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, async execute() { return null; }, + async find(_o: string, ast: any) { reads.push({ ast }); return run(ast); }, + async findOne(_o: string, ast: any) { reads.push({ ast }); return run(ast)[0] ?? null; }, + async count(_o: string, ast: any) { reads.push({ ast }); return run(ast).length; }, + async create(_o: string, data: Record) { + const id = (data.id as string) ?? `r_${rows.size + 1}`; + const row = { ...data, id }; rows.set(id, row); return row; + }, + async update(_o: string, id: string, data: Record) { + const up = { ...(rows.get(id) ?? {}), ...data, id }; rows.set(id, up); return up; + }, + async delete(_o: string, id: string) { return rows.delete(id); }, + async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return { driver, reads }; +} + +type Thrown = (Error & { + code?: string; status?: number; httpStatus?: number; field?: string; fields?: string[]; object?: string; param?: string; +}) | null; + +const refusalOf = async (p: Promise): Promise => p.then(() => null, (e: any) => e); + +const ENVELOPE = { code: 'INVALID_FIELD', status: 400, httpStatus: 400 }; +const envelopeOf = (err: Thrown) => ({ code: err?.code, status: err?.status, httpStatus: err?.httpStatus }); + +/** Each door's own words — a control must never be answered in them. */ +const GROUP_WORDS = 'which the engine does not group by'; +const DISTINCT_WORDS = 'which the engine does not count distinct'; + +const declaredOf = (type: string, multiple: boolean) => (multiple ? `${type} field with multiple: true` : `${type} field`); + +describe('[#20808] a groupBy on a multi-value field, and a count_distinct on a JSON-stored field, at the engine\'s aggregate door', () => { + let engine: ObjectQL; + let reads: SeenRead[]; + + beforeEach(async () => { + const rec = makeRecordingDriver(); + reads = rec.reads; + engine = new ObjectQL(); + engine.registerDriver(rec.driver, true); + await engine.init(); + engine.registry.registerObject(PROBE as any, 'test'); + reads.length = 0; + }); + + it('refuses a groupBy on every multi-value declaration with INVALID_FIELD / 400, naming the field, its declaration, the position and the $contains route — no read', async () => { + for (const [field, type, multiple] of MULTIS) { + const err = await refusalOf(engine.aggregate(OBJECT, { groupBy: [field], aggregations: COUNT })); + expect(envelopeOf(err), field).toEqual(ENVELOPE); + expect({ field: err?.field, fields: err?.fields, object: err?.object, param: err?.param }, field) + .toEqual({ field, fields: [field], object: OBJECT, param: 'groupBy' }); + expect(err!.message, field).toMatch( + new RegExp(`^aggregate\\('${OBJECT}'\\): groupBy\\[0\\] names '${field}', a declared ${declaredOf(type, multiple)} — a multi-value field, ${GROUP_WORDS}`), + ); + expect(err!.message, field).toContain('The query was NOT run.'); + expect(err!.message, field).toContain(`where { "${field}": { "$contains": VALUE } }`); + } + expect(reads, 'every refusal precedes the driver').toHaveLength(0); + }); + + it('judges the { field } object form, and names the first offending position across both classes', async () => { + const cases: ReadonlyArray = [ + [[{ field: 'tags' }], 'groupBy[0].field', ['tags']], + [['title', 'labels'], 'groupBy[1]', ['labels']], + [['refs', 'meta'], 'groupBy[0]', ['refs', 'meta']], + [['meta', 'refs'], 'groupBy[0]', ['meta', 'refs']], + ] as ReadonlyArray; + for (const [groupBy, position, fields] of cases) { + const label = JSON.stringify(groupBy); + const err = await refusalOf(engine.aggregate(OBJECT, { groupBy, aggregations: COUNT })); + expect(envelopeOf(err), label).toEqual(ENVELOPE); + expect(err?.fields, label).toEqual(fields); + expect(err!.message, label).toContain(`): ${position} names '${fields[0]}',`); + } + expect(reads).toHaveLength(0); + }); + + it('refuses a count_distinct on every structured-JSON type and every multi-value declaration, at aggregations[i].field — no read', async () => { + const all: ReadonlyArray = [ + ...JSONS.map(([f, t]) => [f, t, false, false] as const), + ...MULTIS.map(([f, t, m]) => [f, t, m, true] as const), + ]; + for (const [field, type, multiple, multiValue] of all) { + const err = await refusalOf(engine.aggregate(OBJECT, { aggregations: distinct(field) })); + expect(envelopeOf(err), field).toEqual(ENVELOPE); + expect({ field: err?.field, fields: err?.fields, object: err?.object, param: err?.param }, field) + .toEqual({ field, fields: [field], object: OBJECT, param: 'aggregations' }); + const kind = multiValue ? 'a multi-value field' : 'a structured-JSON value'; + expect(err!.message, field).toMatch( + new RegExp(`^aggregate\\('${OBJECT}'\\): aggregations\\[0\\]\\.field counts distinct '${field}', a declared ${declaredOf(type, multiple)} — ${kind}, ${DISTINCT_WORDS}`), + ); + expect(err!.message, field).toContain('The query was NOT run.'); + } + // A later aggregation, and two offenders: the first position is named. + const err = await refusalOf(engine.aggregate(OBJECT, { + groupBy: ['status'], + aggregations: [ + { function: 'count', alias: 'n' }, + { function: 'count_distinct', field: 'title', alias: 'titles' }, + { function: 'count_distinct', field: 'ship_to', alias: 'a' }, + { function: 'count_distinct', field: 'tags', alias: 'b' }, + ], + } as EngineAggregateOptions)); + expect(envelopeOf(err)).toEqual(ENVELOPE); + expect(err?.fields).toEqual(['ship_to', 'tags']); + expect(err!.message).toContain("): aggregations[2].field counts distinct 'ship_to',"); + expect(reads, 'every refusal precedes the driver').toHaveLength(0); + }); + + it('CONTROL scalar group keys and scalar distinct counts reach the driver, never these refusals', async () => { + const shapes: ReadonlyArray = [ + ...['title', 'amount', 'status', 'owner', 'photo', 'not_declared'].map((f) => + [`groupBy ${f}`, { groupBy: [f], aggregations: COUNT }] as const), + ...['title', 'amount', 'status', 'owner', 'photo', 'not_declared'].map((f) => + [`count_distinct ${f}`, { aggregations: distinct(f) }] as const), + // `count` compares no value, so a JSON-stored column is countable. + ...['meta', 'tags', 'labels'].map((f) => + [`count ${f}`, { aggregations: [{ function: 'count', field: f, alias: 'n' }] }] as const), + // The route the multi-value refusal names: filter by one member. + ['count where tags $contains', { where: { tags: { $contains: 'a' } }, aggregations: COUNT }], + ] as ReadonlyArray; + for (const [label, query] of shapes) { + const before = reads.length; + const out = await engine.aggregate(OBJECT, query).then(() => null, (e: Error) => e.message); + expect(out ?? '', label).not.toContain(GROUP_WORDS); + expect(out ?? '', label).not.toContain(DISTINCT_WORDS); + expect(reads.length - before, `${label}: the driver was asked`).toBe(1); + } + }); + + it('the REST door into findData answers the same refusals — no read', async () => { + const protocol = new ObjectStackProtocolImplementation(engine); + const grouped = await refusalOf(protocol.findData({ + object: OBJECT, + query: { groupBy: ['tags'], aggregations: COUNT }, + } as any)); + expect(envelopeOf(grouped)).toEqual(ENVELOPE); + expect(grouped!.message).toContain(`groupBy[0] names 'tags', a declared select field with multiple: true`); + const counted = await refusalOf(protocol.findData({ + object: OBJECT, + query: { aggregations: distinct('meta') }, + } as any)); + expect(envelopeOf(counted)).toEqual(ENVELOPE); + expect(counted!.message).toContain(`aggregations[0].field counts distinct 'meta', a declared json field`); + expect(reads).toHaveLength(0); + }); + + it('GUARD the count_distinct door asks the spec table for every FieldType, and isMultiValueField for every flagged multi-capable type', () => { + const judged = (def: Record) => { + try { + assertCountDistinctNamesNoJsonStoredField(OBJECT, { fields: { f: def } }, distinct('f')); + return null; + } catch (e) { + return envelopeOf(e as Thrown); + } + }; + let refused = 0; + for (const type of FieldType.options) { + const tableRefuses = !isAggregateCompatibleWithFieldType('count_distinct', type); + expect(judged({ type }), type).toEqual(tableRefuses || isMultiValueField({ type }) ? ENVELOPE : null); + if (tableRefuses) refused += 1; + } + // Floor: the row refuses the JSON-stored ten, so the equality above is not vacuous. + expect(refused).toBe(10); + for (const type of MULTI_CAPABLE_TYPES) { + expect(judged({ type, multiple: true }), `${type} + multiple`).toEqual(ENVELOPE); + } + }); + + it('GUARD no verdict without a field map, for an undeclared name, an off-vocabulary type, or any other function', () => { + const judge = (schema: unknown, aggregations: unknown) => () => assertCountDistinctNamesNoJsonStoredField(OBJECT, schema, aggregations); + expect(judge(undefined, distinct('meta'))).not.toThrow(); + expect(judge({}, distinct('meta'))).not.toThrow(); + expect(judge(PROBE, distinct('nope'))).not.toThrow(); + // A driver-internal alias on an introspected object: the table cannot answer, so no block. + expect(judge({ fields: { f: { type: 'object' } } }, distinct('f'))).not.toThrow(); + expect(judge({ fields: { f: { type: 'integer' } } }, distinct('f'))).not.toThrow(); + for (const fn of ['count', 'sum', 'avg', 'min', 'max']) { + expect(judge(PROBE, [{ function: fn, field: 'meta', alias: 'n' }]), fn).not.toThrow(); + } + expect(judge(PROBE, [{ function: 'count_distinct', alias: 'n' }, { function: 'count_distinct', field: '*' }, null, 7])).not.toThrow(); + expect(judge(PROBE, 'meta')).not.toThrow(); + expect(judge(PROBE, [])).not.toThrow(); + }); +}); diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index 2b09ab7b657..cd1e9e39967 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -211,7 +211,8 @@ import { EmptyCredentialWriteError, SECRET_MASK, } from './secret-fields.js'; -import { assertGroupByNamesNoStructuredJsonField } from './group-by-structured-json-door.js'; +import { assertGroupByNamesNoJsonStoredField } from './group-by-structured-json-door.js'; +import { assertCountDistinctNamesNoJsonStoredField } from './count-distinct-json-stored-door.js'; import { pluralToSingular, ExternalWriteForbiddenError } from '@objectstack/spec/shared'; import { SchemaRegistry, computeFQN, type ArtifactInstallScope } from './registry.js'; import { expandSearchToFilter } from './search-filter.js'; @@ -16863,8 +16864,17 @@ export class ObjectQL implements IObjectQLEngine { // a group key (memory merged every row into one group, SQLite grouped each // serialized document apart, PostgreSQL answered 500). After the // credential refusal, which reads the same entries, so a protected field - // keeps that refusal's words. - assertGroupByNamesNoStructuredJsonField(object, this._registry.getObject(object), query.groupBy); + // keeps that refusal's words. [#20808] The same door refuses a `groupBy` + // entry naming a MULTI-VALUE field (`isMultiValueField`: a list stored in + // a JSON column, split the same three ways), judged in one walk so the + // first offending position is the one named. + assertGroupByNamesNoJsonStoredField(object, this._registry.getObject(object), query.groupBy); + // [#20808] …and a `count_distinct` over a JSON-stored field: the spec + // table's `count_distinct` row (`isAggregateCompatibleWithFieldType`) + // beside `isMultiValueField`, before any driver is asked — memory counted + // equal documents apart, SQLite compared serialized text, PostgreSQL + // answered 500 (no equality operator for `json`). + assertCountDistinctNamesNoJsonStoredField(object, this._registry.getObject(object), query.aggregations); // [#10576] The per-aggregation `filter` (`AggregationNodeSchema.filter`, // the contract half of #10413) is a second filter position on this verb, // so it walks through the same refusal doors `where` does at this seam: diff --git a/packages/objectql/src/group-by-structured-json-door.ts b/packages/objectql/src/group-by-structured-json-door.ts index 4987e17655b..322c8098d2d 100644 --- a/packages/objectql/src/group-by-structured-json-door.ts +++ b/packages/objectql/src/group-by-structured-json-door.ts @@ -5,6 +5,8 @@ * `composite`, `repeater`, `record`, `location`, `address`, `vector` — is * refused with `INVALID_FIELD` / 400 by the engine's `aggregate`, naming the * field, its declared type and the position, before any driver is asked. + * [#20808] So is a `groupBy` that names a MULTI-VALUE field (the second class, + * below): every JSON-stored group key is refused at this one door. * * ## What ran before this door, measured on `origin/main` `7a09eee1b1` * @@ -48,8 +50,35 @@ * set #20745's JSON arm judges too, never a list minted here. **Not judged:** * an undeclared name (the engine's registry-less tolerance — the ingress door * answers an unknown one `INVALID_FIELD` first), a registry-less host (no - * field map, no verdict), and every other type, `multiple: true` lists and - * file fields included: those are not this card's class. + * field map, no verdict), and every scalar-stored type, single-value file + * fields included. + * + * ## [#20808] The second class: a MULTI-VALUE field + * + * A field whose value is a list — an inherently-multi option type + * (`multiselect`, `checkboxes`, `tags`) or a multi-capable type flagged + * `multiple: true` (`select`, `lookup`, `user`, `file`, `image`) — is stored + * in a JSON column too, and split the same three ways, measured on + * `origin/main` `42d78b97fe` through `POST /api/v1/data/:object/query` over + * every one of those eight declarations: the in-memory driver answered one + * group per array, SQLite one group per serialized array, and PostgreSQL 16 + * refused the statement (`could not identify an equality operator for type + * json`, a 500). The triage direction on #20808: "`groupBy` on a + * `multiple: true` field is refused the same way" — "one bucket per member" + * is a capability no caller was measured to need, and "one group per + * serialized array" is not a meaning any caller could rely on; no dataset, + * cube, report, view grouping or `groupBy` in `examples/` or in the published + * hotcrm stack names a multi-value field. ⛔ No + * bucket-per-member grouping. The refusal names what works: filter by one + * member with `$contains`, the membership operator every driver lowers for a + * JSON-stored list. + * + * The class is `@objectstack/spec/data`'s {@link isMultiValueField} — THE + * definition of "is this field multi-valued", which reads the declaration + * (`multiple`) as well as the type, so a single-value `select` or `lookup` + * is untouched. Both classes are judged in ONE walk over the entries, so the + * refusal names the first offending position whichever class it is, and + * `fields` lists every offender. * * `INVALID_FIELD`, not a new code: the verdict is about the NAMED field's * type at a position, the question the ingress door answers with @@ -57,29 +86,36 @@ * with `INVALID_FIELD` for a field whose type it cannot scan. * * @see https://github.com/objectstack-ai/objectstack/issues/20783 + * @see https://github.com/objectstack-ai/objectstack/issues/20808 */ import { StandardErrorCode } from '@objectstack/spec/api'; -import { STRUCTURED_JSON_TYPES } from '@objectstack/spec/data'; +import { STRUCTURED_JSON_TYPES, isMultiValueField } from '@objectstack/spec/data'; -/** One `groupBy` entry that names a structured-JSON field. */ -interface StructuredJsonGroupTarget { +/** Which JSON-stored class a `groupBy` entry's field belongs to. */ +type JsonStoredGroupClass = 'structured-json' | 'multi-value'; + +/** One `groupBy` entry that names a JSON-stored field. */ +interface JsonStoredGroupTarget { readonly field: string; readonly type: string; + /** The declared field carries `multiple: true` (said in the words). */ + readonly multiple: boolean; + readonly cls: JsonStoredGroupClass; /** `groupBy[i]` for a name, `groupBy[i].field` for the object form. */ readonly position: string; } /** - * The entries of `groupBy` that name a declared structured-JSON field, in - * order. An entry that names no field, a field the map does not declare, or a - * field of any other type is not collected. + * The entries of `groupBy` that name a declared structured-JSON or + * multi-value field, in order. An entry that names no field, a field the map + * does not declare, or a field of any other type is not collected. */ -function structuredJsonGroupTargets( +function jsonStoredGroupTargets( fields: Record, groupBy: readonly unknown[], -): StructuredJsonGroupTarget[] { - const hits: StructuredJsonGroupTarget[] = []; +): JsonStoredGroupTarget[] { + const hits: JsonStoredGroupTarget[] = []; for (const [i, entry] of groupBy.entries()) { const objectForm = entry !== null && typeof entry === 'object' && !Array.isArray(entry); const field = typeof entry === 'string' @@ -87,23 +123,54 @@ function structuredJsonGroupTargets( : objectForm ? (entry as { field?: unknown }).field : undefined; if (typeof field !== 'string') continue; if (!Object.prototype.hasOwnProperty.call(fields, field)) continue; - const type = (fields[field] as { type?: unknown } | undefined)?.type; - if (typeof type !== 'string' || !STRUCTURED_JSON_TYPES.has(type)) continue; - hits.push({ field, type, position: objectForm ? `groupBy[${i}].field` : `groupBy[${i}]` }); + const def = fields[field] as { type?: unknown; multiple?: unknown } | undefined; + const type = def?.type; + if (typeof type !== 'string') continue; + const multiple = def?.multiple === true; + const cls: JsonStoredGroupClass | null = STRUCTURED_JSON_TYPES.has(type) + ? 'structured-json' + : isMultiValueField({ type, multiple }) ? 'multi-value' : null; + if (cls === null) continue; + hits.push({ field, type, multiple, cls, position: objectForm ? `groupBy[${i}].field` : `groupBy[${i}]` }); } return hits; } /** - * Refuse a `groupBy` entry that names a declared structured-JSON field — - * `INVALID_FIELD` / 400, before any driver is asked. See the module header. + * The verdict, the route and the reason for the FIRST offender's class. The + * route comes before the reason so it lands inside the 500 characters the + * REST door keeps. + */ +function groupByRefusalWords(first: JsonStoredGroupTarget): { verdict: string; tail: string } { + if (first.cls === 'multi-value') { + return { + verdict: '— a multi-value field, which the engine does not group by', + tail: '. The query was NOT run. Filter by one member instead: ' + + `where { "${first.field}": { "$contains": VALUE } } counts or lists the records that hold VALUE, ` + + 'one query per member. A list of values is no group key the drivers share: one grouped each ' + + 'list apart, one grouped each serialized list apart, one refused the statement.', + }; + } + return { + verdict: '— a structured-JSON value, which the engine does not group by', + tail: '. The query was NOT run. Group by a field that stores one scalar value: store the part you ' + + 'group on in a field of its own and group by that field. A JSON document is no group key the ' + + 'drivers share: one merged documents that differ into one group (or split them per array), ' + + 'one grouped each serialized document apart, one refused the statement.', + }; +} + +/** + * Refuse a `groupBy` entry that names a declared structured-JSON field + * (#20783) or a declared multi-value field (#20808) — `INVALID_FIELD` / 400, + * before any driver is asked. See the module header. * * The words put the position and the verdict first, then that the query did * not run, then the route, then the reason: the REST door keeps the first 500 * characters of a 4xx message (`CLIENT_MESSAGE_MAX`), and the route must be * inside them. */ -export function assertGroupByNamesNoStructuredJsonField( +export function assertGroupByNamesNoJsonStoredField( object: string, schema: unknown, groupBy: unknown, @@ -111,17 +178,16 @@ export function assertGroupByNamesNoStructuredJsonField( if (!Array.isArray(groupBy) || groupBy.length === 0) return; const fields = (schema as { fields?: unknown } | undefined)?.fields; if (!fields || typeof fields !== 'object') return; - const hits = structuredJsonGroupTargets(fields as Record, groupBy); + const hits = jsonStoredGroupTargets(fields as Record, groupBy); if (hits.length === 0) return; const [first] = hits; + const { verdict, tail } = groupByRefusalWords(first); + const declared = first.multiple ? `${first.type} field with multiple: true` : `${first.type} field`; const err = new Error( - `aggregate('${object}'): ${first.position} names '${first.field}', a declared ${first.type} field ` - + '— a structured-JSON value, which the engine does not group by' + `aggregate('${object}'): ${first.position} names '${first.field}', a declared ${declared} ` + + verdict + (hits.length > 1 ? ` (also: ${hits.slice(1).map((h) => `'${h.field}'`).join(', ')})` : '') - + '. The query was NOT run. Group by a field that stores one scalar value: store the part you ' - + 'group on in a field of its own and group by that field. A JSON document is no group key the ' - + 'drivers share: one merged every row into a single group, one grouped each serialized ' - + 'document apart, one refused the statement.', + + tail, ) as Error & { code?: string; status?: number; httpStatus?: number; field?: string; fields?: string[]; object?: string; param?: string; diff --git a/packages/spec/src/data/aggregate-field-type-compatibility.test.ts b/packages/spec/src/data/aggregate-field-type-compatibility.test.ts index 4d40eecbb5c..9b10ed71519 100644 --- a/packages/spec/src/data/aggregate-field-type-compatibility.test.ts +++ b/packages/spec/src/data/aggregate-field-type-compatibility.test.ts @@ -26,6 +26,8 @@ import { INSTANT_TYPES, CLOCK_TIME_TYPES, BOOLEAN_VALUE_TYPES, + STRUCTURED_JSON_TYPES, + MULTI_OPTION_TYPES, } from './field-value.zod'; import { isIncoherentAggregate } from './aggregation-policy'; import { AGGREGATION_CASES } from './aggregation-conformance'; @@ -40,6 +42,9 @@ const NUMERIC = ['currency', 'number', 'percent', 'progress', 'rating', 'slider' const ADDITIVE = NUMERIC.filter((t) => t !== 'percent'); const TEMPORAL = ['date', 'datetime', 'time']; const BOOLEAN = ['boolean', 'toggle']; +const JSON_STORED = [ + 'address', 'checkboxes', 'composite', 'json', 'location', 'multiselect', 'record', 'repeater', 'tags', 'vector', +]; describe('AGGREGATE_FIELD_TYPE_COMPATIBILITY — totality', () => { it('every AggregationFunction member has a row, and no row is for a non-member', () => { @@ -69,9 +74,22 @@ describe('AGGREGATE_FIELD_TYPE_COMPATIBILITY — totality', () => { }); describe('AGGREGATE_FIELD_TYPE_COMPATIBILITY — the ruled rows, resolved against the membership', () => { - it('`count` / `count_distinct`: every FieldType', () => { + it('`count`: every FieldType', () => { expect(sorted(AGGREGATE_FIELD_TYPE_COMPATIBILITY.count)).toEqual(sorted(FieldType.options)); - expect(sorted(AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct)).toEqual(sorted(FieldType.options)); + }); + + it('[#20808] `count_distinct`: every FieldType EXCEPT the JSON-stored ones', () => { + expect(sorted(AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct)) + .toEqual(sorted(FieldType.options.filter((t) => !JSON_STORED.includes(t)))); + }); + + it('[#20808] the JSON-stored bucket IS the field-value structured-JSON class plus the multi-option types', () => { + // A type joining either class elsewhere is stored in a JSON column by every + // SQL driver, and no two backends compare such values alike — so it reds + // here until the count_distinct row records a decision. + expect(sorted([...STRUCTURED_JSON_TYPES, ...MULTI_OPTION_TYPES])).toEqual(JSON_STORED); + const refused = FieldType.options.filter((t) => !AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct.includes(t)); + expect(sorted(refused)).toEqual(JSON_STORED); }); it('`sum`: the numeric class EXCEPT `percent`, plus the boolean class', () => { @@ -157,10 +175,21 @@ describe('isAggregateCompatibleWithFieldType — the pairs the card is about', ( } }); - it('accepts `count` / `count_distinct` over anything, `vector` and `formula` included', () => { + it('accepts `count` over anything, `vector` and `formula` included', () => { for (const t of FieldType.options) { expect(isAggregateCompatibleWithFieldType('count', t)).toBe(true); - expect(isAggregateCompatibleWithFieldType('count_distinct', t)).toBe(true); + } + }); + + it('[#20808] accepts `count_distinct` over every scalar-stored type — `formula`, `percent`, `select`, `lookup`, `file` included — and refuses the JSON-stored ones', () => { + for (const t of FieldType.options) { + expect(isAggregateCompatibleWithFieldType('count_distinct', t), t).toBe(!JSON_STORED.includes(t)); + } + for (const t of ['formula', 'percent', 'text', 'select', 'lookup', 'user', 'file', 'image']) { + expect(isAggregateCompatibleWithFieldType('count_distinct', t), t).toBe(true); + } + for (const t of ['json', 'vector', 'address', 'tags', 'multiselect']) { + expect(isAggregateCompatibleWithFieldType('count_distinct', t), t).toBe(false); } }); @@ -168,9 +197,10 @@ describe('isAggregateCompatibleWithFieldType — the pairs the card is about', ( // Both refuse the sum of a rate. expect(isIncoherentAggregate('sum', 'percent')).toBe(true); expect(isAggregateCompatibleWithFieldType('sum', 'percent')).toBe(false); - // The semantic opinion flags count_distinct of a rate; the ruling reads - // `count_distinct` as "any type" and this table follows the ruling. Pinned - // so the divergence is visible, not discovered (reported on #16353). + // The semantic opinion flags count_distinct of a rate; the ruling read + // `count_distinct` as "any type", the JSON-stored narrowing (#20808) does + // not reach a rate, and this table follows both. Pinned so the divergence + // is visible, not discovered (reported on #16353). expect(isIncoherentAggregate('count_distinct', 'percent')).toBe(true); expect(isAggregateCompatibleWithFieldType('count_distinct', 'percent')).toBe(true); }); diff --git a/packages/spec/src/data/aggregate-field-type-compatibility.ts b/packages/spec/src/data/aggregate-field-type-compatibility.ts index 2c4004e03a2..94cf5a2e3f5 100644 --- a/packages/spec/src/data/aggregate-field-type-compatibility.ts +++ b/packages/spec/src/data/aggregate-field-type-compatibility.ts @@ -8,7 +8,9 @@ * below). A `DatasetMeasure` pairs an `aggregate` with a `field`; this * table is the contract both consumer legs execute — the compile-time refusal * in the dataset compiler (#16099) and the authoring-time lint rule — so the - * two cannot drift into two accounts of one pair. + * two cannot drift into two accounts of one pair. The `count_distinct` row has + * a third reader, the engine's `aggregate` door, which refuses a query-time + * pair the row refuses (the JSON-stored rows below). * * ## Why this exists * @@ -26,7 +28,8 @@ * * | Aggregate | Accepted field types | * |---|---| - * | `count`, `count_distinct` | every `FieldType` — counting rows or distinct values reads no arithmetic off the value | + * | `count` | every `FieldType` — counting rows reads no arithmetic off the value | + * | `count_distinct` | every `FieldType` EXCEPT the JSON-stored ones — the structured-JSON class and the multi-option types, whose values no two backends compare for equality alike (#20808, see below) | * | `sum` | the numeric class EXCEPT `percent` — a rate does not add (see `isIncoherentAggregate`) — plus the boolean class | * | `avg` | the numeric class, `percent` included, plus the boolean class | * | `min`, `max` | the numeric class plus the temporal class — both return a value of the field's OWN type (#15768) — plus the boolean class | @@ -77,6 +80,24 @@ * refused". `formula` carries a declared `returnType`, but it is VIRTUAL in * SQL storage (no column is emitted), so no arithmetic aggregate can be * lowered to it whatever that type says; `autonumber` is a formatted string. + * - **JSON-stored** = `STRUCTURED_JSON_TYPES` (`json`, `composite`, + * `repeater`, `record`, `location`, `address`, `vector`) ∪ + * `MULTI_OPTION_TYPES` (`multiselect`, `checkboxes`, `tags`): the types + * every SQL driver stores in a JSON column. They are refused for + * `count_distinct` on this table's own ground — "can every backend give one + * answer" — because no backend pair does (triage direction on #20808, + * 2026-09-30: "`AGGREGATE_FIELD_TYPE_COMPATIBILITY` stops accepting it"). + * Measured through `POST /api/v1/data/:object/query` over three rows, two + * of them holding equal values: the in-memory driver counted 3 (each row's + * value apart), SQLite counted 2 (equal serialized text), and PostgreSQL 16 + * refused the statement (`could not identify an equality operator for type + * json`, a 500), on every member of both classes. `count` is untouched: + * counting rows compares no value. The one other JSON-stored shape, a + * multi-capable type flagged `multiple: true` (`select`, `lookup`, `user`, + * `file`, `image`), is invisible to a per-TYPE table, so the engine's + * aggregate door refuses it by the declaration (`isMultiValueField`) beside + * this row's verdict. ⛔ No per-backend JSON distinctness is defined to + * make the pair answerable: no caller of it was measured. * * One row the ruling's default covers used to be recorded here as an OVERRIDE * of an existing opinion. It is SETTLED GROUND now, and the opinion it @@ -105,14 +126,17 @@ * That predicate (`aggregation-policy.ts`) is the SEMANTIC opinion — "does * this number mean anything" — and `sum` × `percent` is refused here on its * authority. It also flags `count_distinct` × `percent`, which this table - * ACCEPTS: the ruling reads `count_distinct` as "any type", and counting - * distinct rates is backend-consistent even where it is odd. The two stay + * ACCEPTS: the ruling read `count_distinct` as "any type", the JSON-stored + * narrowing above does not reach a rate, and counting distinct rates is + * backend-consistent even where it is odd. The two stay * separate on purpose: this table answers "can every backend give one * answer", the lint warning answers "is that answer meaningful". * * ## What this module deliberately does NOT do * - * It refuses nothing itself. The refusals are the two consumer legs; a + * It refuses nothing itself. The refusals are the consumer legs — the dataset + * compile leg and the authoring lint leg on every row, and the engine's + * `aggregate` door on the `count_distinct` row (#20808); a * consumer that cannot resolve a field's type (a relationship PATH it has no * metadata for) must NOT call the predicate with a guess — "cannot answer, do * not block" is the consumer's tier, not this table's. @@ -152,9 +176,27 @@ const BOOLEAN_AGGREGATE_FIELD_TYPES = [ 'boolean', 'toggle', ] as const satisfies readonly FieldType[]; -/** Every declared `FieldType` — the `count` / `count_distinct` row. */ +/** Every declared `FieldType` — the `count` row. */ const ANY_FIELD_TYPE: readonly FieldType[] = Object.freeze([...FieldType.options]); +/** + * The JSON-stored types — the `STRUCTURED_JSON_TYPES` ∪ `MULTI_OPTION_TYPES` + * membership, spelled out for the same reason as the numeric class above (the + * pin test holds the two equal): a type joining either class elsewhere is a + * DECISION here. No two backends compare these values for equality alike, so + * the `count_distinct` row is every declared `FieldType` except these (#20808 + * — see the module TSDoc). + */ +const JSON_STORED_AGGREGATE_FIELD_TYPES = [ + 'json', 'composite', 'repeater', 'record', 'location', 'address', 'vector', + 'multiselect', 'checkboxes', 'tags', +] as const satisfies readonly FieldType[]; + +/** Every declared `FieldType` except the JSON-stored ones — the `count_distinct` row. */ +const DISTINCT_COMPARABLE_FIELD_TYPES: readonly FieldType[] = Object.freeze( + FieldType.options.filter((t) => !(JSON_STORED_AGGREGATE_FIELD_TYPES as readonly string[]).includes(t)), +); + /** * Which `FieldType`s each `AggregationFunction` may be applied to. Total over * `AggregationFunction` (the `Record` key type makes a missing row a `tsc` @@ -166,7 +208,7 @@ const ANY_FIELD_TYPE: readonly FieldType[] = Object.freeze([...FieldType.options export const AGGREGATE_FIELD_TYPE_COMPATIBILITY: Readonly> = Object.freeze({ count: ANY_FIELD_TYPE, - count_distinct: ANY_FIELD_TYPE, + count_distinct: DISTINCT_COMPARABLE_FIELD_TYPES, sum: Object.freeze([...ADDITIVE_AGGREGATE_FIELD_TYPES, ...BOOLEAN_AGGREGATE_FIELD_TYPES]), avg: Object.freeze([...NUMERIC_AGGREGATE_FIELD_TYPES, ...BOOLEAN_AGGREGATE_FIELD_TYPES]), min: Object.freeze([ @@ -180,7 +222,8 @@ export const AGGREGATE_FIELD_TYPE_COMPATIBILITY: Readonly Date: Wed, 30 Sep 2026 15:54:20 +0000 Subject: [PATCH 2/8] wip(rest): public-door pins for the multi-value groupBy and JSON-stored count_distinct refusals (#20808) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- ...ta-json-stored-group-distinct-door.test.ts | 267 ++++++++++++++++++ 1 file changed, 267 insertions(+) create mode 100644 packages/rest/src/data-json-stored-group-distinct-door.test.ts diff --git a/packages/rest/src/data-json-stored-group-distinct-door.test.ts b/packages/rest/src/data-json-stored-group-distinct-door.test.ts new file mode 100644 index 00000000000..993069cceb6 --- /dev/null +++ b/packages/rest/src/data-json-stored-group-distinct-door.test.ts @@ -0,0 +1,267 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#20808] At the public door — `POST /api/v1/data/:object/query` over a real + * `SqlDriver` — a `groupBy` on a MULTI-VALUE field and a `count_distinct` on a + * JSON-STORED field answer `400 INVALID_FIELD` in the engine's words, naming + * the field, its declaration, the position and the route, before any read; + * and the scalar controls, plus the `$contains` route the multi-value refusal + * names, are served by the driver unchanged. + * + * Measured on the base (`origin/main` `42d78b97fe`) through this door, three + * rows: + * + * | query | InMemoryDriver | SQLite | PostgreSQL 16 | + * |:--|:--|:--|:--| + * | `groupBy: ['status']`, a single-value select (the control) | 200, `a` 2 · `b` 1 | same | same | + * | `groupBy` a multi-value field (`select` / `lookup` / `user` / `file` / `image` with `multiple: true`; `tags`, `multiselect`, `checkboxes`) | 200, one group per array | 200, one group per serialized array | 500 `DATABASE_ERROR` | + * | `count_distinct` `title`, a text (the control) | 2 | 2 | 2 | + * | `count_distinct` a structured-JSON field (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) | 3 | 2 (`json`: 3) | 500 `DATABASE_ERROR` | + * | `count_distinct` a multi-value field | 3 | 2 | 500 `DATABASE_ERROR` | + * + * The refusals sit in the engine, in front of every driver, so one verdict + * holds on each cell. InMemoryDriver's row is `@objectstack/objectql`'s + * `engine-json-stored-group-distinct-door.test.ts` by construction (the doors + * answer before a driver is resolved); this package does not depend on the + * in-memory driver, and that driver's test consumers are a ruled, closed + * census (`check:driver-memory-census`). + * + * ## The dialect axis of THIS file + * + * The SQLite cell always runs. The PostgreSQL and MySQL cells run where + * `OS_TEST_POSTGRES_URL` / `OS_TEST_MYSQL_URL` are set and are a named skip + * otherwise. ⚠️ No CI job provisions those variables for this package, so the + * live cells are red-capable and un-run in CI; the PR that landed this file + * carries their local PostgreSQL run. Each live cell owns its tables, dropped + * before and after. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import type { EngineAggregateOptions } from '@objectstack/spec/data'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { RestServer } from './rest-server'; + +const OBJECT = 'rest_json_stored_key_20808'; +const TARGET = 'rest_json_stored_key_target_20808'; + +const OPTIONS = [{ label: 'A', value: 'a' }, { label: 'B', value: 'b' }]; + +/** field · declared type · `multiple: true` — every multi-value declaration the spec admits. */ +const MULTIS: ReadonlyArray = [ + ['tags', 'select', true], + ['labels', 'tags', false], + ['ms', 'multiselect', false], + ['cb', 'checkboxes', false], + ['refs', 'lookup', true], + ['watchers', 'user', true], + ['files', 'file', true], + ['imgs', 'image', true], +]; + +/** field · declared type — one field of every structured-JSON type. */ +const JSONS: ReadonlyArray = [ + ['meta', 'json'], + ['spec', 'composite'], + ['rep', 'repeater'], + ['rec', 'record'], + ['loc', 'location'], + ['ship_to', 'address'], + ['vec', 'vector'], +]; + +const LEDGER = { + name: OBJECT, + label: 'Ledger 20808', + fields: { + title: { name: 'title', type: 'text' as const }, + status: { name: 'status', type: 'select' as const, options: OPTIONS }, + ...Object.fromEntries(MULTIS.map(([name, type, multiple]) => [ + name, + { + name, + type, + ...(multiple ? { multiple: true } : {}), + ...(type === 'select' || type === 'multiselect' || type === 'checkboxes' ? { options: OPTIONS } : {}), + ...(type === 'lookup' ? { reference: TARGET } : {}), + }, + ])), + ...Object.fromEntries(JSONS.map(([name, type]) => [name, { name, type }])), + }, +}; + +const TARGET_OBJECT = { name: TARGET, label: 'Target 20808', fields: { name: { name: 'name', type: 'text' as const } } }; + +/** Two of the three rows hold EQUAL values under every JSON-stored field — the case the drivers counted apart. */ +const ROWS = [ + { id: 'd1', title: 'x', status: 'a', tags: ['a', 'b'], labels: ['p', 'q'], ms: ['a'], cb: ['a', 'b'], refs: ['t1', 't2'], watchers: ['u1'], files: ['f1'], imgs: ['i1'], meta: { a: 1 }, spec: { k: 1 }, rep: [{ q: 1 }], rec: { r: 1 }, loc: { lat: 1, lng: 2 }, ship_to: { city: 'Paris' }, vec: [1, 2] }, + { id: 'd2', title: 'x', status: 'a', tags: ['a'], labels: ['p'], ms: ['a', 'b'], cb: ['a'], refs: ['t1'], watchers: ['u1', 'u2'], files: ['f1', 'f2'], imgs: ['i1', 'i2'], meta: { a: 2 }, spec: { k: 2 }, rep: [{ q: 2 }], rec: { r: 2 }, loc: { lat: 3, lng: 4 }, ship_to: { city: 'Rome' }, vec: [3, 4] }, + { id: 'd3', title: 'y', status: 'b', tags: ['a', 'b'], labels: ['p', 'q'], ms: ['a'], cb: ['a', 'b'], refs: ['t1', 't2'], watchers: ['u1'], files: ['f1'], imgs: ['i1'], meta: { b: 1 }, spec: { k: 1 }, rep: [{ q: 1 }], rec: { r: 1 }, loc: { lat: 1, lng: 2 }, ship_to: { city: 'Paris' }, vec: [1, 2] }, +]; + +const COUNT: EngineAggregateOptions['aggregations'] = [{ function: 'count', alias: 'n' }]; +const distinct = (field: string): EngineAggregateOptions['aggregations'] => + [{ function: 'count_distinct', field, alias: 'n' }]; + +/** The routes the refusals name — asserted on the REST body, so each must land inside the door's 500-character bound. */ +const MEMBER_ROUTE = (field: string) => `where { "${field}": { "$contains": VALUE } }`; +const SCALAR_DISTINCT_ROUTE = 'Count distinct values of a field that stores one scalar value: store the part you count in a field of its own and count_distinct that field, or count the rows with count.'; + +interface Cell { + id: 'sqlite' | 'pg' | 'mysql'; + label: string; + env: string | null; + config: () => Record | null; +} + +const CELLS: readonly Cell[] = [ + { id: 'sqlite', label: 'sqlite', env: null, config: () => ({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }) }, + { + id: 'pg', + label: 'live postgres', + env: 'OS_TEST_POSTGRES_URL', + config: () => (process.env.OS_TEST_POSTGRES_URL ? { client: 'pg', connection: process.env.OS_TEST_POSTGRES_URL } : null), + }, + { + id: 'mysql', + label: 'live mysql', + env: 'OS_TEST_MYSQL_URL', + config: () => (process.env.OS_TEST_MYSQL_URL ? { client: 'mysql2', connection: process.env.OS_TEST_MYSQL_URL } : null), + }, +]; + +function createMockServer() { + const noop = () => {}; + return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} }; +} + +function makeRes() { + const res: any = { + write: () => true, end: () => {}, + header: () => res, + status: (code: number) => { res._status = code; return res; }, + json: (body: any) => { res._json = body; return res; }, + }; + return res; +} + +for (const cell of CELLS) { + const config = cell.config(); + describe.skipIf(!config)( + `[#20808] a groupBy on a multi-value field and a count_distinct on a JSON-stored field at the public door — ${cell.label}${config ? '' : ` (skipped: set ${cell.env} to run this cell)`}`, + () => { + let engine: ObjectQL; + let driver: any; + const reads = { n: 0 }; + let query: (body: Record) => Promise<{ status: number; body: any }>; + + const dropTables = async () => { + if (cell.id === 'sqlite') return; + for (const t of [OBJECT, TARGET]) await driver?.execute(`drop table if exists ${t}`).catch(() => {}); + }; + + beforeAll(async () => { + driver = new SqlDriver(config as any); + await dropTables(); + engine = new ObjectQL(); + engine.registerDriver(driver, true); + await engine.init(); + engine.registry.registerObject(TARGET_OBJECT as any); + engine.registry.registerObject(LEDGER as any); + await engine.syncSchemas(); + for (const id of ['t1', 't2']) await engine.insert(TARGET, { id, name: id } as any); + for (const row of ROWS) await engine.insert(OBJECT, { ...row } as any); + + // Reads of THIS object — the protocol's own metadata traffic is not the question. + for (const verb of ['find', 'findOne', 'count', 'aggregate'] as const) { + const real = driver[verb].bind(driver); + driver[verb] = (o: string, ...rest: unknown[]) => { if (o === OBJECT) reads.n += 1; return real(o, ...rest); }; + } + + const protocol = new ObjectStackProtocolImplementation(engine as any); + const rest = new RestServer(createMockServer() as any, protocol as any, { api: { requireAuth: false } } as any); + (rest as any).resolveExecCtx = async () => ({ userId: 'test-user' }); + rest.registerRoutes(); + const route = rest.getRoutes().find((r: any) => r.method === 'POST' && r.path === '/api/v1/data/:object/query'); + expect(route).toBeDefined(); + query = async (body) => { + const res = makeRes(); + // What the wire carries: JSON. + await route!.handler({ params: { object: OBJECT }, body: JSON.parse(JSON.stringify(body)), query: {}, headers: {} } as any, res); + return { status: res._status ?? 200, body: res._json }; + }; + }); + + afterAll(async () => { + await dropTables(); + try { await engine?.destroy(); } catch { /* noop */ } + }); + + it('a groupBy on every multi-value declaration answers 400 INVALID_FIELD in the engine\'s words, naming the field, its declaration and the $contains route — no read', async () => { + const before = reads.n; + for (const [field, type, multiple] of MULTIS) { + const res = await query({ groupBy: [field], aggregations: COUNT }); + expect(res.status, `${field}: ${JSON.stringify(res.body)}`).toBe(400); + expect(res.body.code, field).toBe('INVALID_FIELD'); + const declared = multiple ? `${type} field with multiple: true` : `${type} field`; + expect(res.body.error, field).toContain(`groupBy[0] names '${field}', a declared ${declared} — a multi-value field`); + expect(res.body.error, field).toContain(MEMBER_ROUTE(field)); + const err = await engine.aggregate(OBJECT, { groupBy: [field], aggregations: COUNT }).then(() => null, (e: any) => e); + expect({ code: err?.code, status: err?.status }, `engine.aggregate, ${field}`).toEqual({ code: 'INVALID_FIELD', status: 400 }); + } + const objectForm = await query({ groupBy: [{ field: 'tags' }], aggregations: COUNT }); + expect(objectForm.status, JSON.stringify(objectForm.body)).toBe(400); + expect(objectForm.body.error).toContain(`groupBy[0].field names 'tags'`); + expect(reads.n - before, 'no read of the object — every refusal precedes the driver').toBe(0); + }); + + it('a count_distinct on every structured-JSON type and every multi-value declaration answers 400 INVALID_FIELD at aggregations[0].field — no read', async () => { + const before = reads.n; + const cases: ReadonlyArray = [ + ...JSONS.map(([f, t]) => [f, `${t} field — a structured-JSON value`, false] as const), + ...MULTIS.map(([f, t, m]) => [f, `${m ? `${t} field with multiple: true` : `${t} field`} — a multi-value field`, true] as const), + ]; + for (const [field, declared, multiValue] of cases) { + const res = await query({ aggregations: distinct(field) }); + expect(res.status, `${field}: ${JSON.stringify(res.body)}`).toBe(400); + expect(res.body.code, field).toBe('INVALID_FIELD'); + expect(res.body.error, field).toContain(`aggregations[0].field counts distinct '${field}', a declared ${declared}`); + expect(res.body.error, field).toContain(multiValue ? MEMBER_ROUTE(field) : SCALAR_DISTINCT_ROUTE); + const err = await engine.aggregate(OBJECT, { aggregations: distinct(field) }).then(() => null, (e: any) => e); + expect({ code: err?.code, status: err?.status }, `engine.aggregate, ${field}`).toEqual({ code: 'INVALID_FIELD', status: 400 }); + } + expect(reads.n - before, 'no read of the object — every refusal precedes the driver').toBe(0); + }); + + it('CONTROL a single-value select groupBy and a scalar count_distinct are served unchanged, from the driver', async () => { + const before = reads.n; + const grouped = await query({ groupBy: ['status'], aggregations: COUNT }); + expect(grouped.status, JSON.stringify(grouped.body)).toBe(200); + const groups = (grouped.body.records as Array<{ status: string; n: number | string }>) + .map((r) => [r.status, Number(r.n)] as const) + .sort(([a], [b]) => a.localeCompare(b)); + expect(groups).toEqual([['a', 2], ['b', 1]]); + for (const field of ['title', 'status']) { + const counted = await query({ aggregations: distinct(field) }); + expect(counted.status, `${field}: ${JSON.stringify(counted.body)}`).toBe(200); + expect(Number(counted.body.records[0].n), field).toBe(2); + } + expect(reads.n - before, 'the driver was asked, once per query').toBe(3); + }); + + it('CONTROL the route the multi-value refusal names answers from the driver: count with where { tags: { $contains } }', async () => { + const before = reads.n; + const counts: Record = {}; + for (const member of ['a', 'b']) { + const res = await query({ where: { tags: { $contains: member } }, aggregations: COUNT }); + expect(res.status, `${member}: ${JSON.stringify(res.body)}`).toBe(200); + counts[member] = Number(res.body.records[0].n); + } + // d1 [a,b] · d2 [a] · d3 [a,b]: every row holds `a`, two hold `b`. + expect(counts).toEqual({ a: 3, b: 2 }); + expect(reads.n - before, 'the driver was asked').toBe(2); + }); + }, + ); +} From c633ea51d7e07b3762471ba10626343567791db6 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 15:56:10 +0000 Subject: [PATCH 3/8] wip(lint,spec,docs): the sentences the count_distinct narrowing makes false, and the lint fixture that pinned the old row (#20808) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- .../docs/deployment/validating-metadata.mdx | 8 ++++--- ...alidate-dataset-measure-aggregates.test.ts | 23 +++++++++++++++---- .../validate-dataset-measure-aggregates.ts | 5 ++-- ...et-measure-aggregate-field-type-refused.ts | 9 ++++++-- ...-selecting-aggregate-field-type-refused.ts | 6 +++-- 5 files changed, 38 insertions(+), 13 deletions(-) diff --git a/content/docs/deployment/validating-metadata.mdx b/content/docs/deployment/validating-metadata.mdx index 9bd26e1a3c1..2383fcf59b9 100644 --- a/content/docs/deployment/validating-metadata.mdx +++ b/content/docs/deployment/validating-metadata.mdx @@ -206,9 +206,11 @@ measures: [{ name: 'avg_closed', aggregate: 'avg', field: 'closed_at' }] `avg` over a temporal column is where this bites hardest: one SQL family coerces the stored text and returns a plausible number (the average *year*), another has no such function and fails at query time. `min`/`max` over the same field are -**accepted** — they return a real instant of the field's own type — and -`count`/`count_distinct` are accepted over every type, because they read no -arithmetic off the value. The analytics service refuses the same pair with +**accepted** — they return a real instant of the field's own type — `count` is +accepted over every type, because it reads no value, and `count_distinct` over +every type but the JSON-stored ones (the structured-JSON types and +`multiselect` / `checkboxes` / `tags`), whose values no two backends compare +for equality alike. The analytics service refuses the same pair with `400 DATASET_INVALID` when a query is built; this is the identical verdict, from the identical table, one door earlier. The rule stays silent wherever the field's type cannot be resolved (an object this stack does not define, a diff --git a/packages/lint/src/validate-dataset-measure-aggregates.test.ts b/packages/lint/src/validate-dataset-measure-aggregates.test.ts index cc2a1fce978..7c7a969192c 100644 --- a/packages/lint/src/validate-dataset-measure-aggregates.test.ts +++ b/packages/lint/src/validate-dataset-measure-aggregates.test.ts @@ -165,13 +165,28 @@ describe('measure-aggregate-field-type-refused — stays silent on every pair th } }); - it('accepts count and count_distinct over every declared FieldType', () => { + it('accepts count over every declared FieldType', () => { for (const fieldType of FieldType.options) { - for (const aggregate of ['count', 'count_distinct']) { - expect(findings(stackWith(aggregate, fieldType)), `${aggregate}(${fieldType})`).toEqual([]); - } + expect(findings(stackWith('count', fieldType)), `count(${fieldType})`).toEqual([]); } }); + + // [#20808] `count_distinct` over a JSON-stored type left the table's row: no + // two backends compare those values for equality alike (the in-memory driver + // counted equal documents apart, SQLite compared serialized text, PostgreSQL + // answered 500). Every other declared type stays accepted. + it('accepts count_distinct over every declared FieldType except the JSON-stored ones, which it refuses', () => { + const jsonStored = ['json', 'composite', 'repeater', 'record', 'location', 'address', 'vector', 'multiselect', 'checkboxes', 'tags']; + for (const fieldType of FieldType.options) { + const found = findings(stackWith('count_distinct', fieldType)); + expect(found.length, `count_distinct(${fieldType})`).toBe(jsonStored.includes(fieldType) ? 1 : 0); + } + const [issue] = findings(stackWith('count_distinct', 'json')); + expect(issue.rule).toBe(RULE); + // The way out names `count`, and no sentence says count_distinct accepts every type. + expect(issue.hint).toContain('accepts: count.'); + expect(issue.hint).not.toMatch(/count_distinct` accept every type/); + }); }); describe('measure-aggregate-field-type-refused — the rule IS the table, on every pair', () => { diff --git a/packages/lint/src/validate-dataset-measure-aggregates.ts b/packages/lint/src/validate-dataset-measure-aggregates.ts index 4d2666de8e0..4a371f2e651 100644 --- a/packages/lint/src/validate-dataset-measure-aggregates.ts +++ b/packages/lint/src/validate-dataset-measure-aggregates.ts @@ -232,8 +232,9 @@ export function validateDatasetMeasureAggregates(stack: unknown): DatasetMeasure `Either point "${aggregate}" at a field of an accepted type, or aggregate ` + `"${field}" with one its \`${fieldType}\` type accepts: ` + `${aggregatesAccepting(fieldType).join(', ')}. ` + - `\`count\` / \`count_distinct\` accept every type because they read no arithmetic off ` + - `the value; a quantity that must be added up or averaged has to be STORED as a ` + + `\`count\` accepts every type because it reads no value, and \`count_distinct\` every ` + + `type but the JSON-stored ones, whose values no two backends compare alike; a ` + + `quantity that must be added up or averaged has to be STORED as a ` + `numeric field (a computed column) and aggregated as one. The compile leg refuses ` + `this same pair with \`400 DATASET_INVALID\` before any SQL is emitted, so this is ` + `the same fix made earlier.`, diff --git a/packages/spec/src/migrations/entries/semantic/18.dataset-measure-aggregate-field-type-refused.ts b/packages/spec/src/migrations/entries/semantic/18.dataset-measure-aggregate-field-type-refused.ts index 4cd11a9d633..835eee15a7d 100644 --- a/packages/spec/src/migrations/entries/semantic/18.dataset-measure-aggregate-field-type-refused.ts +++ b/packages/spec/src/migrations/entries/semantic/18.dataset-measure-aggregate-field-type-refused.ts @@ -12,7 +12,11 @@ export const entry: SemanticMigration = { + 'The non-temporal `sum` / `avg` rows followed in a later change, which registered NO ' + 'entry of its own — it declared `not-required (already-registered ' + 'dataset-measure-aggregate-field-type-refused)` against THIS id — so its widening ' - + 'rides this entry\'s prescription rather than a separate one. The `min` / `max` rows ' + + 'rides this entry\'s prescription rather than a separate one. The `count_distinct` ' + + 'row\'s JSON-stored types (`json`, `composite`, `repeater`, `record`, `location`, ' + + '`address`, `vector`, `multiselect`, `checkboxes`, `tags`) left the table in a third ' + + 'change, which likewise registered no entry and rides this one: no two backends compare ' + + 'those values for equality alike, so `count` is the aggregate that stays. The `min` / `max` rows ' + 'over every class the table refuses are the second entry, ' + '`dataset-measure-selecting-aggregate-field-type-refused`. ⇒ Read BOTH when ' + 'migrating; there is no third', @@ -56,7 +60,8 @@ export const entry: SemanticMigration = { + 'further: a measure over a field of any other class was not judged by the leg this ' + 'entry was registered for. It is covered all the same — by this entry\'s own ' + 'prescription, widened by a later change (which registered `not-required` against this ' - + 'id rather than an entry of its own) to `sum` / `avg` over every field class; and by ' + + 'id rather than an entry of its own) to `sum` / `avg` over every field class, and by a ' + + 'third, the same way, to `count_distinct` over the JSON-stored types; and by ' + '`dataset-measure-selecting-aggregate-field-type-refused` for `min` / ' + '`max`. ⛔ There is no third entry to look for. At protocol major 18 as a whole, ' + 'every refused pair in `AGGREGATE_FIELD_TYPE_COMPATIBILITY` is refused at the ' diff --git a/packages/spec/src/migrations/entries/semantic/18.dataset-measure-selecting-aggregate-field-type-refused.ts b/packages/spec/src/migrations/entries/semantic/18.dataset-measure-selecting-aggregate-field-type-refused.ts index fec6edde6ff..9a06ef83826 100644 --- a/packages/spec/src/migrations/entries/semantic/18.dataset-measure-selecting-aggregate-field-type-refused.ts +++ b/packages/spec/src/migrations/entries/semantic/18.dataset-measure-selecting-aggregate-field-type-refused.ts @@ -22,8 +22,10 @@ export const entry: SemanticMigration = { + 'text value" in a way every backend agrees on, so no transform can preserve the ' + 'answer. The three routes an author actually has, per intent: ' + '① the measure was COUNTING in disguise ("how many distinct owners") ⇒ ' - + '`count` / `count_distinct`, which accept every type because they read neither ' - + 'arithmetic nor order off the value; ' + + '`count`, which accepts every type because it reads no value, or `count_distinct`, ' + + 'which accepts every type but the JSON-stored ones (the structured-JSON types and ' + + '`multiselect` / `checkboxes` / `tags`, whose values no two backends compare for ' + + 'equality alike); ' + '② the measure wanted a FIRST or LAST RECORD ("the earliest-titled task") ⇒ that is ' + 'a SORT on a list or report, which orders once in a declared direction, not an ' + 'aggregate that asks each backend for its own smallest value; ' From b5c33c0e3188fa45f9136076883f9b48f3149589 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 15:58:43 +0000 Subject: [PATCH 4/8] chore(spec): regenerate the migration registry for the amended step-18 entry prose (#20808) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- packages/spec/src/migrations/registry.ts | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/packages/spec/src/migrations/registry.ts b/packages/spec/src/migrations/registry.ts index 2307120c3f9..d30ad64ef5d 100644 --- a/packages/spec/src/migrations/registry.ts +++ b/packages/spec/src/migrations/registry.ts @@ -8726,7 +8726,11 @@ const step18: MigrationStep = { + 'The non-temporal `sum` / `avg` rows followed in a later change, which registered NO ' + 'entry of its own — it declared `not-required (already-registered ' + 'dataset-measure-aggregate-field-type-refused)` against THIS id — so its widening ' - + 'rides this entry\'s prescription rather than a separate one. The `min` / `max` rows ' + + 'rides this entry\'s prescription rather than a separate one. The `count_distinct` ' + + 'row\'s JSON-stored types (`json`, `composite`, `repeater`, `record`, `location`, ' + + '`address`, `vector`, `multiselect`, `checkboxes`, `tags`) left the table in a third ' + + 'change, which likewise registered no entry and rides this one: no two backends compare ' + + 'those values for equality alike, so `count` is the aggregate that stays. The `min` / `max` rows ' + 'over every class the table refuses are the second entry, ' + '`dataset-measure-selecting-aggregate-field-type-refused`. ⇒ Read BOTH when ' + 'migrating; there is no third', @@ -8770,7 +8774,8 @@ const step18: MigrationStep = { + 'further: a measure over a field of any other class was not judged by the leg this ' + 'entry was registered for. It is covered all the same — by this entry\'s own ' + 'prescription, widened by a later change (which registered `not-required` against this ' - + 'id rather than an entry of its own) to `sum` / `avg` over every field class; and by ' + + 'id rather than an entry of its own) to `sum` / `avg` over every field class, and by a ' + + 'third, the same way, to `count_distinct` over the JSON-stored types; and by ' + '`dataset-measure-selecting-aggregate-field-type-refused` for `min` / ' + '`max`. ⛔ There is no third entry to look for. At protocol major 18 as a whole, ' + 'every refused pair in `AGGREGATE_FIELD_TYPE_COMPATIBILITY` is refused at the ' @@ -8803,8 +8808,10 @@ const step18: MigrationStep = { + 'text value" in a way every backend agrees on, so no transform can preserve the ' + 'answer. The three routes an author actually has, per intent: ' + '① the measure was COUNTING in disguise ("how many distinct owners") ⇒ ' - + '`count` / `count_distinct`, which accept every type because they read neither ' - + 'arithmetic nor order off the value; ' + + '`count`, which accepts every type because it reads no value, or `count_distinct`, ' + + 'which accepts every type but the JSON-stored ones (the structured-JSON types and ' + + '`multiselect` / `checkboxes` / `tags`, whose values no two backends compare for ' + + 'equality alike); ' + '② the measure wanted a FIRST or LAST RECORD ("the earliest-titled task") ⇒ that is ' + 'a SORT on a list or report, which orders once in a declared direction, not an ' + 'aggregate that asks each backend for its own smallest value; ' From d2a605896fd13c0b2824d9ecb974fabe137ae1eb Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 15:59:34 +0000 Subject: [PATCH 5/8] chore(changeset): declare the multi-value groupBy and JSON-stored count_distinct narrowings (#20808) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- ...0808-json-stored-group-distinct-refused.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 .changeset/20808-json-stored-group-distinct-refused.md diff --git a/.changeset/20808-json-stored-group-distinct-refused.md b/.changeset/20808-json-stored-group-distinct-refused.md new file mode 100644 index 00000000000..d3492fccbc8 --- /dev/null +++ b/.changeset/20808-json-stored-group-distinct-refused.md @@ -0,0 +1,30 @@ +--- +"@objectstack/objectql": minor +"@objectstack/spec": minor +"@objectstack/lint": patch +--- + +fix(objectql,spec)!: a `groupBy` on a multi-value field and a `count_distinct` on a JSON-stored field are refused with `INVALID_FIELD` / 400 at the engine's `aggregate`, on every driver, and the aggregate × field-type table stops accepting `count_distinct` over the JSON-stored types + +Clause-②: no (narrowing) + + + +**BREAKING** (`@objectstack/objectql`): this narrows what `aggregate` accepts, in two positions, on every driver and for every caller that reaches the engine (the REST query door, a flow or hook, and the analytics strategy that lowers a cube query onto `engine.aggregate`). Shipped as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes. + +- A `groupBy` entry that names a **multi-value** field: an inherently-multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. Both entry spellings are judged, the field name and the `{ field }` object. +- A `count_distinct` aggregation over a **JSON-stored** field: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently-multi option type, or a multi-capable field declared `multiple: true`. + +**BREAKING** (`@objectstack/spec`): `AGGREGATE_FIELD_TYPE_COMPATIBILITY.count_distinct` no longer lists the ten JSON-stored types (the structured-JSON seven and `multiselect`, `checkboxes`, `tags`), so `isAggregateCompatibleWithFieldType('count_distinct', type)` answers `false` for them. Every reader of the table refuses those pairs now: the dataset-measure lint rule (`measure-aggregate-field-type-refused`, run by `os validate` and at a runtime dataset save), the analytics dataset compile leg (`400 DATASET_INVALID`), and the engine door above. The `count` row is unchanged. + +**What an author sees now.** `400 INVALID_FIELD`, naming the position (`groupBy[0]`, `groupBy[0].field`, or `aggregations[0].field`), the field and its declaration, saying the query was not run, and naming the route inside the first 500 characters the REST door keeps. For a multi-value field the route is to filter by one member: `where` with `$contains` on the field, one query per member. For a structured-JSON field it is to store the part you count in a field of its own, or to count rows with `count`. The thrown error carries `field`, `fields`, `object` and `param` (`groupBy` or `aggregations`). + +**Why a refusal.** Every SQL driver stores these values in a JSON column, and the drivers share no meaning for one as a group key or a distinct key. Measured through `POST /api/v1/data/:object/query` over three rows: grouping by any of the eight multi-value declarations answered one group per array on the in-memory driver, one group per serialized array on SQLite, and 500 `DATABASE_ERROR` on PostgreSQL 16. `count_distinct` over any structured-JSON or multi-value field answered 3 on the in-memory driver (equal values counted apart), 2 on SQLite (serialized text compared), and 500 on PostgreSQL (no equality operator for `json`). No example app and no published stack groups by a multi-value field or counts one distinct, so no meaning is defined for either here. + +**What to write instead.** A dataset measure or a query that counted a JSON-stored field distinct: use `count` over it, or store the scalar part you meant to count in a field of its own and `count_distinct` that field. A grouping by a multi-value field: filter by each member with `$contains` and count. + +**Who is affected.** A caller that grouped by a multi-value field, or counted a JSON-stored field distinct, on the in-memory driver or on SQLite and read the answer as a real one; on PostgreSQL both were already a 500. A dataset whose measure pairs `count_distinct` with a JSON-stored field is refused by the lint rule and the compile leg. + +**Unchanged.** A `groupBy` or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; the `having`, filter and sort positions; and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached. + +`@objectstack/lint`: the dataset-measure refusal's hint no longer says `count_distinct` accepts every type. From 95b7278ad6033be3a851f78f609d76a55010e574 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 16:15:18 +0000 Subject: [PATCH 6/8] fix(service-analytics): the compile leg's words for a count_distinct refusal, and the census pin the narrowed table moves (#20808) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- ...0808-json-stored-group-distinct-refused.md | 3 ++ ...regate-nontemporal-measure-refusal.test.ts | 36 ++++++++++++++++--- .../service-analytics/src/dataset-compiler.ts | 36 ++++++++++++++----- 3 files changed, 62 insertions(+), 13 deletions(-) diff --git a/.changeset/20808-json-stored-group-distinct-refused.md b/.changeset/20808-json-stored-group-distinct-refused.md index d3492fccbc8..eb330b3ee9b 100644 --- a/.changeset/20808-json-stored-group-distinct-refused.md +++ b/.changeset/20808-json-stored-group-distinct-refused.md @@ -2,6 +2,7 @@ "@objectstack/objectql": minor "@objectstack/spec": minor "@objectstack/lint": patch +"@objectstack/service-analytics": patch --- fix(objectql,spec)!: a `groupBy` on a multi-value field and a `count_distinct` on a JSON-stored field are refused with `INVALID_FIELD` / 400 at the engine's `aggregate`, on every driver, and the aggregate × field-type table stops accepting `count_distinct` over the JSON-stored types @@ -28,3 +29,5 @@ Clause-②: no (narrowing) **Unchanged.** A `groupBy` or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; the `having`, filter and sort positions; and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached. `@objectstack/lint`: the dataset-measure refusal's hint no longer says `count_distinct` accepts every type. + +`@objectstack/service-analytics`: the dataset compile leg's refusal of a `count_distinct` measure over a JSON-stored field says why it diverges (the drivers compare the values for equality three ways) and prescribes `count`, or a scalar field for the part being counted; its other refusals no longer say `count_distinct` accepts every type. diff --git a/packages/services/service-analytics/src/__tests__/aggregate-nontemporal-measure-refusal.test.ts b/packages/services/service-analytics/src/__tests__/aggregate-nontemporal-measure-refusal.test.ts index 286208619e2..8be8f70bbda 100644 --- a/packages/services/service-analytics/src/__tests__/aggregate-nontemporal-measure-refusal.test.ts +++ b/packages/services/service-analytics/src/__tests__/aggregate-nontemporal-measure-refusal.test.ts @@ -157,7 +157,7 @@ describe('#16099 — the pairs this leg refuses are the TABLE\'s, not this packa expect(isAggregateCompatibleWithFieldType('avg', 'percent')).toBe(true); }); - it('⭐ the refused set, enumerated by the card that enforced each part — 155 pairs, 0 left over', () => { + it('⭐ the refused set, enumerated by the card that enforced each part — 165 pairs, 0 left over', () => { // The arithmetic the PR bodies show, asserted rather than narrated, so a // row moving upstream moves this count instead of leaving a stale claim. // ⚠️ [#17560] The `minmaxString` / `selecting` split this case used to carry @@ -165,21 +165,23 @@ describe('#16099 — the pairs this leg refuses are the TABLE\'s, not this packa // apart because they were "ruled to be AMENDED" under #17513, and decision // batch #127 found no ruling behind that and declined to amend the table. // The 42 and the 32 are one population again, enforced in one pass. - let refusedByTable = 0, temporal = 0, deriving = 0, selecting = 0; + let refusedByTable = 0, temporal = 0, deriving = 0, selecting = 0, distinct = 0; for (const a of Object.keys(AGGREGATE_FIELD_TYPE_COMPATIBILITY)) { for (const ft of FieldType.options) { if (isAggregateCompatibleWithFieldType(a, ft)) continue; refusedByTable++; if (a === 'min' || a === 'max') { selecting++; continue; } + if (a === 'count_distinct') { distinct++; continue; } if (TEMPORAL_SOURCE_FIELD_TYPES.has(ft)) { temporal++; continue; } deriving++; } } - expect(refusedByTable).toBe(155); + expect(refusedByTable).toBe(165); expect(temporal).toBe(6); // commit 357f4992b's — `sum`/`avg` over the temporal class expect(deriving).toBe(75); // #16099's — `sum`/`avg` over everything else expect(selecting).toBe(74); // #17560's — `min`/`max`, 42 string + 32 non-string - expect(temporal + deriving + selecting).toBe(refusedByTable); + expect(distinct).toBe(10); // #20808's — `count_distinct` over the JSON-stored types + expect(temporal + deriving + selecting + distinct).toBe(refusedByTable); // ⭐ And nothing is left declared-but-unenforced: one door judges all six. expect(Object.keys(AGGREGATE_FIELD_TYPE_COMPATIBILITY).length).toBe(6); }); @@ -326,7 +328,7 @@ describe('#16099 — the controls: every pair the table accepts still compiles', expect(err.message).not.toContain('derives a NUMBER'); }); - it('`count` / `count_distinct` accept every type — they read no arithmetic off the value', async () => { + it('`count` / `count_distinct` over a scalar-stored field compile — they read no arithmetic off the value', async () => { for (const aggregate of ['count', 'count_distinct'] as const) { const { go, sqls } = run(aggregate, 'note'); const result: any = await go(); @@ -335,6 +337,30 @@ describe('#16099 — the controls: every pair the table accepts still compiles', } }); + // [#20808] The table's `count_distinct` row stops accepting the JSON-stored + // types, and this door judges every row: the refusal says why a distinct + // count diverges there (EQUALITY, not arithmetic or order) and never says + // `count_distinct` accepts every type. + it('[#20808] `count_distinct` over a JSON-stored field → DATASET_INVALID / 400 in the distinct words, before any SQL', async () => { + for (const [field, declared] of [['payload', 'json'], ['embedding', 'vector'], ['tags_list', 'multiselect']] as const) { + expect(isAggregateCompatibleWithFieldType('count_distinct', declared), declared).toBe(false); + const { go, sqls } = run('count_distinct', field); + const err = await refusalOf(go); + expect({ code: err.code, status: err.status }, declared).toEqual({ code: 'DATASET_INVALID', status: 400 }); + expect(err.message, declared).toContain(`declares as \`${declared}\``); + expect(err.message, declared).toContain('COMPARES the stored values for equality'); + expect(err.message, declared).not.toContain('derives a NUMBER'); + expect(err.message, declared).not.toMatch(/count_distinct` accept every type/); + expect(err.message, declared).toContain('`count` counts the rows'); + expect(sqls.length, declared).toBe(0); + } + // `count` over the same field compiles — it compares nothing. + const { go, sqls } = run('count', 'payload'); + const result: any = await go(); + expect(result.rows.length).toBe(1); + expect(sqls.length).toBe(1); + }); + it('the three cannot-answer tiers still do not block — unchanged by the widened scope', async () => { // 1. a field the hook cannot resolve const unknown = run('sum', 'no_such_field'); diff --git a/packages/services/service-analytics/src/dataset-compiler.ts b/packages/services/service-analytics/src/dataset-compiler.ts index 982c0784e72..e0c76a3bd14 100644 --- a/packages/services/service-analytics/src/dataset-compiler.ts +++ b/packages/services/service-analytics/src/dataset-compiler.ts @@ -215,11 +215,22 @@ function aggregateToMetricType(m: DatasetMeasure): Metric['type'] { * the ORDER — collation-dependent for text on every backend, and absent * altogether where the storage form has no ordering operator (`min(jsonb)` * does not exist on PostgreSQL). + * - [#20808] `count_distinct` COMPARES the stored values for equality, and + * reaches this sentence only over a JSON-stored field (the table's + * `count_distinct` row refuses the structured-JSON and multi-option types): + * the in-memory driver counted equal documents apart, SQLite compared the + * serialized text, and PostgreSQL has no equality operator for `json`. * - * `count` / `count_distinct` accept every type and never reach this sentence. + * `count` accepts every type and never reaches this sentence. */ const DIVERGENCE_BY_AGGREGATE = (aggregate: string, fieldType: string): string => - aggregate === 'min' || aggregate === 'max' + aggregate === 'count_distinct' + ? `"${aggregate}" COMPARES the stored values for equality, so over a \`${fieldType}\` column ` + + 'the answer is decided by how each backend compares a JSON-stored value rather than by ' + + 'the data — one counts every row apart, one compares the serialized text, another has no ' + + 'equality for the type and fails at query time — and one dataset would mean two things ' + + 'on two deployments. ' + : aggregate === 'min' || aggregate === 'max' ? `"${aggregate}" SELECTS one of the stored values, so over a \`${fieldType}\` column the ` + 'answer is decided by the ORDER the SQL dialect happens to impose rather than by the ' + 'data — string order is collation-dependent, and some storage forms have no ordering ' @@ -244,19 +255,28 @@ const DIVERGENCE_BY_AGGREGATE = (aggregate: string, fieldType: string): string = * wants. [#17560] The selecting sentence is the third: an author who wrote * `min` over a text column wanted a FIRST ROW, and a sort delivers that in one * declared order instead of asking each backend for its own smallest value. + * [#20808] The distinct sentence is the fourth: `count_distinct` over a + * JSON-stored field has no value every backend compares alike, so the author + * counts rows, or counts distinct a scalar field holding the part they meant. + * The other two non-temporal sentences no longer say `count_distinct` accepts + * every type: since #20808 it accepts every type but the JSON-stored ones. */ const REMEDY_BY_SOURCE_CLASS = (fieldType: string, aggregate: string): string => TEMPORAL_SOURCE_FIELD_TYPES.has(fieldType) ? 'For a temporal field, `min`/`max` return a real instant; a DURATION has to be ' + 'stored as a number (a computed "days open" field) and aggregated as one.' + : aggregate === 'count_distinct' + ? 'For a JSON-stored field, `count` counts the rows; a distinct count has to be taken ' + + 'over a field that stores one scalar value, so store the part you count in a field ' + + 'of its own and `count_distinct` that field.' : aggregate === 'min' || aggregate === 'max' - ? 'For a field with no backend-independent order, `count`/`count_distinct` accept every ' - + 'type because they read neither arithmetic nor order off the value; a "first" or ' - + '"last" record is a SORT on the record list, which orders once in a declared ' + ? 'For a field with no backend-independent order, `count` accepts every type because it ' + + 'reads no value, and `count_distinct` every type but the JSON-stored ones; a "first" ' + + 'or "last" record is a SORT on the record list, which orders once in a declared ' + 'direction, not an aggregate that asks every backend for its own smallest value.' - : 'For a non-numeric field, `count`/`count_distinct` accept every type because they ' - + 'read no arithmetic off the value; a quantity that should be added up has to be ' - + 'stored as a numeric field and aggregated as one.'; + : 'For a non-numeric field, `count` accepts every type because it reads no value, and ' + + '`count_distinct` every type but the JSON-stored ones; a quantity that should be ' + + 'added up has to be stored as a numeric field and aggregated as one.'; /** * [#16737 / #16099 / #17560] Refuse a measure whose AGGREGATE cannot meaningfully From b613f60a0bfd47d16ed63f404d082cb2bd72c0c9 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 17:13:11 +0000 Subject: [PATCH 7/8] docs(changeset): the structured-JSON groupBy entry no longer lists as unchanged the two shapes this release also refuses (#20808) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- .changeset/20783-groupby-structured-json-refused.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.changeset/20783-groupby-structured-json-refused.md b/.changeset/20783-groupby-structured-json-refused.md index 77c31eee744..9b4887008ef 100644 --- a/.changeset/20783-groupby-structured-json-refused.md +++ b/.changeset/20783-groupby-structured-json-refused.md @@ -16,4 +16,4 @@ Clause-②: no (narrowing) **Who is affected.** A caller of `engine.aggregate` or of the REST query door that grouped by such a field on the in-memory driver or on SQLite and read the merged or per-serialization groups as real ones. On PostgreSQL the same query was already a 500. The analytics service's aggregate path (a cube query the native-SQL strategy declines, such as a time dimension with a granularity, or any cube query on the in-memory driver) reaches the engine and answers this refusal too. -**Unchanged.** A `groupBy` on any other type (`text`, `number`, a `multiple: true` select, a file field), a structured-JSON field as an AGGREGATED column (`count`, `count_distinct`, `min`, `max`), and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached. +**Unchanged by this entry.** A `groupBy` on a scalar-stored type (`text`, `number`, a single-value file field), a structured-JSON field as an AGGREGATED column (`count`, `min`, `max`), and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached. A `groupBy` on a multi-value field and a `count_distinct` over a structured-JSON field are refused too, by the multi-value / JSON-stored entry of this same release. From f53718f1ee9182aa33ff33e83218ef15dd164ab7 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 17:19:13 +0000 Subject: [PATCH 8/8] docs(changeset): leave the structured-JSON groupBy entry as it landed; this entry names the two shapes it narrows (#20808) Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude --- .changeset/20783-groupby-structured-json-refused.md | 2 +- .changeset/20808-json-stored-group-distinct-refused.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.changeset/20783-groupby-structured-json-refused.md b/.changeset/20783-groupby-structured-json-refused.md index 9b4887008ef..77c31eee744 100644 --- a/.changeset/20783-groupby-structured-json-refused.md +++ b/.changeset/20783-groupby-structured-json-refused.md @@ -16,4 +16,4 @@ Clause-②: no (narrowing) **Who is affected.** A caller of `engine.aggregate` or of the REST query door that grouped by such a field on the in-memory driver or on SQLite and read the merged or per-serialization groups as real ones. On PostgreSQL the same query was already a 500. The analytics service's aggregate path (a cube query the native-SQL strategy declines, such as a time dimension with a granularity, or any cube query on the in-memory driver) reaches the engine and answers this refusal too. -**Unchanged by this entry.** A `groupBy` on a scalar-stored type (`text`, `number`, a single-value file field), a structured-JSON field as an AGGREGATED column (`count`, `min`, `max`), and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached. A `groupBy` on a multi-value field and a `count_distinct` over a structured-JSON field are refused too, by the multi-value / JSON-stored entry of this same release. +**Unchanged.** A `groupBy` on any other type (`text`, `number`, a `multiple: true` select, a file field), a structured-JSON field as an AGGREGATED column (`count`, `count_distinct`, `min`, `max`), and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached. diff --git a/.changeset/20808-json-stored-group-distinct-refused.md b/.changeset/20808-json-stored-group-distinct-refused.md index eb330b3ee9b..c4b90cf36cc 100644 --- a/.changeset/20808-json-stored-group-distinct-refused.md +++ b/.changeset/20808-json-stored-group-distinct-refused.md @@ -26,7 +26,7 @@ Clause-②: no (narrowing) **Who is affected.** A caller that grouped by a multi-value field, or counted a JSON-stored field distinct, on the in-memory driver or on SQLite and read the answer as a real one; on PostgreSQL both were already a 500. A dataset whose measure pairs `count_distinct` with a JSON-stored field is refused by the lint rule and the compile leg. -**Unchanged.** A `groupBy` or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; the `having`, filter and sort positions; and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached. +**Unchanged.** (Two shapes the structured-JSON `groupBy` entry of this same release lists as unchanged are narrowed here: a `multiple: true` select as a group key, and `count_distinct` over a structured-JSON field. This entry is the later word on both.) A `groupBy` or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; the `having`, filter and sort positions; and an undeclared name, which the REST door answers `INVALID_FIELD` as unknown before the engine is reached. `@objectstack/lint`: the dataset-measure refusal's hint no longer says `count_distinct` accepts every type.