diff --git a/.changeset/20890-dataset-dimension-json-stored-refused.md b/.changeset/20890-dataset-dimension-json-stored-refused.md new file mode 100644 index 00000000000..46095ee43ba --- /dev/null +++ b/.changeset/20890-dataset-dimension-json-stored-refused.md @@ -0,0 +1,17 @@ +--- +"@objectstack/lint": minor +--- + +fix(lint)!: `os validate`, `os build` and `os lint` refuse a dataset dimension over a JSON-stored field, the group key the analytics door already refuses at query time + +Clause-②: yes (narrowing) + + + +**BREAKING**: metadata that passed `os validate`, `os build` and `os lint` can now fail, and so can a runtime dataset save (Studio, REST `/meta`, MCP), which runs the same rule. A dataset dimension is a group key, and the analytics door refuses a query that groups by a JSON-stored column with `400 INVALID_FIELD` before any SQL is built, so such a dimension could be declared but never served. The new rule `dimension-json-stored-field-refused` (gating, `error`) refuses it where the author writes it. It ships as `minor` under the launch-window convention for accept-set narrowings. No export is removed. The package entry exports the new id as `DIMENSION_JSON_STORED_FIELD_REFUSED`, beside `MEASURE_AGGREGATE_FIELD_TYPE_REFUSED`, and the `rule` member of `DatasetMeasureAggregateFinding` gains it. + +**What is refused.** A dimension whose `field` resolves, on the dataset's object or across its join chain, to a field declared with a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) or a multi-value declaration (`multiselect`, `checkboxes`, `tags`, or a `select`, `radio`, `lookup`, `user`, `file` or `image` declared `multiple: true`). The two classes are `@objectstack/spec/data`'s `STRUCTURED_JSON_TYPES` and `isMultiValueField`, the predicates the analytics door reads. + +**What an author sees now.** The finding names the dataset, the dimension, the field, the object that declares it and its declaration, and says the analytics door refuses every query that groups by it. It names the route: group by a field that stores one scalar value, storing the part of the document you group on in a field of its own; for a multi-value field, filter by one member with `$contains` in a record query, one query per member. It is located at `datasets[N].dimensions[M].field`, name-keyed on the runtime wire. + +**Unchanged.** A dimension over any other field, a single-value `select` or `lookup` included; a dimension whose field does not resolve (`dataset-field-unknown` reports that) or declares no type; a dataset over an object this stack does not define; measures, filters and every other position. A cube dimension (`analyticsCubes`) is not judged: no authoring rule reads cubes. diff --git a/.changeset/20890-dataset-distinct-multiple-refused.md b/.changeset/20890-dataset-distinct-multiple-refused.md new file mode 100644 index 00000000000..fb261ac5371 --- /dev/null +++ b/.changeset/20890-dataset-distinct-multiple-refused.md @@ -0,0 +1,17 @@ +--- +"@objectstack/lint": minor +--- + +fix(lint)!: a dataset `count_distinct` measure over a field declared `multiple: true` is refused by `measure-aggregate-field-type-refused`, as the compile leg and the engine already refuse it + +Clause-②: yes (narrowing) + + + +**BREAKING**: metadata that passed `os validate`, `os build` and `os lint` can now fail, and so can a runtime dataset save, which runs the same rule. `measure-aggregate-field-type-refused` reads the field's declaration, not its type alone: `count_distinct` over a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true` is refused, because that field is a list stored as JSON and no two backends compare such values for equality alike. The dataset compile leg already answers the pair `400 DATASET_INVALID`, and the engine's `count_distinct` door answers it `400 INVALID_FIELD`. It ships as `minor` under the launch-window convention for accept-set narrowings. + +**What an author sees now.** The finding names the measure, the field, the object and the declaration with its flag (`select` with `multiple: true`), and says the aggregate accepts its row's types, none of them with `multiple: true`. The hint names the aggregates the declaration does accept, read from the same predicate: `count`. + +**What to write instead.** `count` over the field, or `count_distinct` over a field that stores one scalar value. To count the records holding one member, filter by it with `$contains` in a record query. + +**Unchanged.** `count_distinct` over the same types without the flag; `count` over any field; every other aggregate, whose rows accept no multi-capable type and whose verdicts therefore do not move; and the skips the rule already had. diff --git a/content/docs/deployment/validating-metadata.mdx b/content/docs/deployment/validating-metadata.mdx index 2383fcf59b9..86affd2db23 100644 --- a/content/docs/deployment/validating-metadata.mdx +++ b/content/docs/deployment/validating-metadata.mdx @@ -210,12 +210,26 @@ no such function and fails at query time. `min`/`max` over the same field are accepted over every type, because it reads no value, and `count_distinct` over every type but the JSON-stored ones (the structured-JSON types and `multiselect` / `checkboxes` / `tags`), whose values no two backends compare -for equality alike. The analytics service refuses the same pair with -`400 DATASET_INVALID` when a query is built; this is the identical verdict, +for equality alike. The check reads the field's declaration as well as its +type: a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared +`multiple: true` holds a list stored as JSON, so `count_distinct` over it is +refused too, although the table accepts the type. The analytics service +refuses the same pair with `400 DATASET_INVALID` when a query is built; this +is the identical verdict, from the identical table, one door earlier. The rule stays silent wherever the field's type cannot be resolved (an object this stack does not define, a dangling field path, an untyped field) rather than guessing. +The same check judges the dataset's **dimensions**, under its own id. A +dimension is a group key, and the analytics service refuses a query grouped by +a JSON-stored column with `400 INVALID_FIELD` before any SQL is built. So a +dimension whose field is declared with a structured-JSON type, or as a +multi-value field (`multiselect` / `checkboxes` / `tags`, or one of the types +above declared `multiple: true`), is refused +(`dimension-json-stored-field-refused`) at `datasets[N].dimensions[M].field`. +Group by a field that stores one value instead. Like the measure check, it +stays silent when the field's type cannot be resolved. + ### 7. Navigation exposing objects nobody can read Navigation and permissions are separate metadata, each valid on its own — so an diff --git a/packages/lint/src/index.ts b/packages/lint/src/index.ts index 0c9de57b252..2495e09adb8 100644 --- a/packages/lint/src/index.ts +++ b/packages/lint/src/index.ts @@ -59,6 +59,7 @@ export type { WidgetBindingFinding, WidgetBindingSeverity } from './validate-wid export { validateDatasetMeasureAggregates, MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, + DIMENSION_JSON_STORED_FIELD_REFUSED, } from './validate-dataset-measure-aggregates.js'; export type { DatasetMeasureAggregateFinding } from './validate-dataset-measure-aggregates.js'; diff --git a/packages/lint/src/validate-dataset-measure-aggregates.test.ts b/packages/lint/src/validate-dataset-measure-aggregates.test.ts index 7c7a969192c..b78fb603d32 100644 --- a/packages/lint/src/validate-dataset-measure-aggregates.test.ts +++ b/packages/lint/src/validate-dataset-measure-aggregates.test.ts @@ -17,12 +17,16 @@ import { AGGREGATE_FIELD_TYPE_COMPATIBILITY, BOOLEAN_VALUE_TYPES, FieldType, + MULTI_CAPABLE_TYPES, NUMERIC_VALUE_TYPES, + STRUCTURED_JSON_TYPES, isAggregateCompatibleWithFieldType, + isMultiValueField, } from '@objectstack/spec/data'; import { runAuthoringRules } from './authoring-rules.js'; import { + DIMENSION_JSON_STORED_FIELD_REFUSED, MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, validateDatasetMeasureAggregates, } from './validate-dataset-measure-aggregates.js'; @@ -388,3 +392,230 @@ describe('measure-aggregate-field-type-refused — reaches the author through th ).toEqual([]); }); }); + +// ─────────────────────────────────────────────────────────────────────────── +// [#20890] The declaration half: `count_distinct` over a field declared +// `multiple: true`. The table is per TYPE and accepts `select`; the declaration +// makes it a list stored as JSON, which the compile leg (`400 DATASET_INVALID`) +// and the engine's `count_distinct` door already refuse. +// ─────────────────────────────────────────────────────────────────────────── + +/** {@link stackWith}, with the measured field declared `multiple: true`. */ +const flaggedStackWith = (aggregate: string, fieldType: string): Record => + stackWith(aggregate, fieldType, { + fields: { name: { type: 'text' }, measured: { type: fieldType, multiple: true } }, + }); + +describe('measure-aggregate-field-type-refused — reads the declaration, not the type alone', () => { + // ⭐ The addendum's pin and its control, one pair. + it('refuses count_distinct over a select declared multiple: true, and accepts it over a single select', () => { + const found = findings(flaggedStackWith('count_distinct', 'select')); + expect(found).toHaveLength(1); + const issue = found[0]; + expect(issue.severity).toBe('error'); + expect(issue.rule).toBe(RULE); + expect(issue.path).toBe('datasets[0].measures[0].aggregate'); + // The DECLARATION is named, flag included, or the author reads the verdict + // as a claim about `select` that the table plainly does not make. + expect(issue.message).toContain('`select` with `multiple: true`'); + expect(issue.message).toContain('none of them with `multiple: true`'); + // The way out is computed from the same predicate: only `count` is left. + expect(issue.hint).toContain('accepts: count.'); + + expect(findings(stackWith('count_distinct', 'select'))).toEqual([]); + }); + + it('refuses count_distinct over every multi-capable type flagged multiple: true, and nothing else moves', () => { + expect(MULTI_CAPABLE_TYPES.size).toBeGreaterThan(0); + for (const fieldType of MULTI_CAPABLE_TYPES) { + expect(findings(flaggedStackWith('count_distinct', fieldType)), `count_distinct(${fieldType}, multiple)`).toHaveLength(1); + expect(findings(stackWith('count_distinct', fieldType)), `count_distinct(${fieldType})`).toEqual([]); + // `count` reads no value, flagged or not. + expect(findings(flaggedStackWith('count', fieldType)), `count(${fieldType}, multiple)`).toEqual([]); + } + }); + + // The whole flagged surface against the spec's two predicates — the table's + // row and `isMultiValueField` — never against a list retyped here. + it('agrees with the table and isMultiValueField on every aggregate × every declared FieldType flagged multiple: true', () => { + let refused = 0; + let accepted = 0; + for (const aggregate of AGGREGATES) { + for (const fieldType of FieldType.options) { + const fires = findings(flaggedStackWith(aggregate, fieldType)).length > 0; + const expected = + !isAggregateCompatibleWithFieldType(aggregate, fieldType) || + (aggregate === 'count_distinct' && isMultiValueField({ type: fieldType, multiple: true })); + expect(fires, `${aggregate}(${fieldType}, multiple)`).toBe(expected); + if (fires) refused++; + else accepted++; + } + } + expect(refused).toBeGreaterThan(50); + expect(accepted).toBeGreaterThan(50); + }); +}); + +// ─────────────────────────────────────────────────────────────────────────── +// [#20890] `dimension-json-stored-field-refused` — the dimension leg. The +// analytics door refuses a query that groups by a JSON-stored column +// (`400 INVALID_FIELD`); this is that verdict where the author is standing. +// ─────────────────────────────────────────────────────────────────────────── + +const DIMENSION_RULE = DIMENSION_JSON_STORED_FIELD_REFUSED; + +/** + * One dataset over one object whose one dimension groups by `grouped`, + * declared as `fieldDef`. `field` overrides the path the dimension names. + */ +const dimensionStack = ( + fieldDef: Record | undefined, + overrides: { field?: unknown; datasetObject?: string } = {}, +): Record => ({ + name: 'analytics_probe', + objects: [ + { + name: 'crm_opportunity', + sharingModel: 'private', + fields: { + name: { type: 'text' }, + grouped: fieldDef ?? { label: 'Untyped' }, + }, + }, + ], + datasets: [ + { + name: 'opportunity_metrics', + object: overrides.datasetObject ?? 'crm_opportunity', + dimensions: [ + { name: 'the_dimension', field: 'field' in overrides ? overrides.field : 'grouped' }, + ], + measures: [{ name: 'n', aggregate: 'count' }], + }, + ], +}); + +const dimensionFindings = (stack: unknown) => + validateDatasetMeasureAggregates(stack).filter((f) => f.rule === DIMENSION_RULE); + +describe('dimension-json-stored-field-refused — a dimension over a JSON-stored field is refused', () => { + // ⭐ The card's pin and its control, one pair. + it('refuses a dimension over a json field, and accepts one over a text field', () => { + const found = dimensionFindings(dimensionStack({ type: 'json' })); + expect(found).toHaveLength(1); + const issue = found[0]; + expect(issue.severity).toBe('error'); + expect(issue.rule).toBe(DIMENSION_RULE); + expect(issue.path).toBe('datasets[0].dimensions[0].field'); + expect(issue.where).toBe('dataset "opportunity_metrics" › dimension "the_dimension"'); + // The author can act without opening the analytics service: the DIMENSION, + // the FIELD, its declared TYPE, the class, and what the door answers. + expect(issue.message).toContain('"the_dimension"'); + expect(issue.message).toContain('"grouped"'); + expect(issue.message).toContain('`json`'); + expect(issue.message).toContain('structured-JSON'); + expect(issue.message).toContain('400 INVALID_FIELD'); + expect(issue.hint).toContain('scalar value'); + + expect(dimensionFindings(dimensionStack({ type: 'text' }))).toEqual([]); + // And a dimension never lands under the measure rule's id. + expect(findings(dimensionStack({ type: 'json' }))).toEqual([]); + }); + + it('refuses a dimension over every member of STRUCTURED_JSON_TYPES', () => { + expect(STRUCTURED_JSON_TYPES.size).toBeGreaterThan(0); + for (const fieldType of STRUCTURED_JSON_TYPES) { + expect(dimensionFindings(dimensionStack({ type: fieldType })), fieldType).toHaveLength(1); + } + }); + + // The analytics door's SECOND predicate, which reads the declaration: the + // same JSON column whether the type is inherently multi or flagged. + it('refuses a dimension over a multi-value field — inherently multi, or flagged multiple: true — and serves the same type unflagged', () => { + for (const fieldType of ['multiselect', 'checkboxes', 'tags']) { + expect(dimensionFindings(dimensionStack({ type: fieldType })), fieldType).toHaveLength(1); + } + const [flagged] = dimensionFindings(dimensionStack({ type: 'select', multiple: true })); + expect(flagged?.message).toContain('`select` with `multiple: true`'); + expect(flagged?.message).toContain('multi-value'); + // The route the door names: filter by one member, never group. + expect(flagged?.hint).toContain('$contains'); + expect(dimensionFindings(dimensionStack({ type: 'select' }))).toEqual([]); + }); + + // The whole surface against the door's two predicates, read from the spec. + it('agrees with STRUCTURED_JSON_TYPES and isMultiValueField on every declared FieldType, flagged and not', () => { + let refused = 0; + let accepted = 0; + for (const fieldType of FieldType.options) { + for (const multiple of [false, true]) { + const def = multiple ? { type: fieldType, multiple: true } : { type: fieldType }; + const fires = dimensionFindings(dimensionStack(def)).length > 0; + const expected = STRUCTURED_JSON_TYPES.has(fieldType) || isMultiValueField({ type: fieldType, multiple }); + expect(fires, `${fieldType}${multiple ? ', multiple' : ''}`).toBe(expected); + if (fires) refused++; + else accepted++; + } + } + expect(refused).toBeGreaterThan(15); + expect(accepted).toBeGreaterThan(50); + }); + + it('judges a dimension over a joined field on the object the leaf lives on', () => { + const joined = (leafType: string) => ({ + name: 'analytics_probe', + objects: [ + { + name: 'crm_opportunity', + sharingModel: 'private', + fields: { name: { type: 'text' }, account: { type: 'lookup', reference: 'crm_account' } }, + }, + { + name: 'crm_account', + sharingModel: 'private', + fields: { name: { type: 'text' }, hq: { type: leafType } }, + }, + ], + datasets: [ + { + name: 'opportunity_metrics', + object: 'crm_opportunity', + include: ['account'], + dimensions: [{ name: 'acct_hq', field: 'account.hq' }], + measures: [{ name: 'n', aggregate: 'count' }], + }, + ], + }); + const found = dimensionFindings(joined('json')); + expect(found).toHaveLength(1); + expect(found[0].message).toContain('account.hq'); + expect(found[0].message).toContain('crm_account'); + expect(dimensionFindings(joined('text'))).toEqual([]); + }); + + it('never hands the predicates a guess — the same skips as the measure leg', () => { + // A base object this stack does not define: `validate-object-references.ts`'s. + expect(dimensionFindings(dimensionStack({ type: 'json' }, { datasetObject: 'not_here' }))).toEqual([]); + // A path that resolves to nothing: `dataset-field-unknown`'s. + expect(dimensionFindings(dimensionStack({ type: 'json' }, { field: 'nope' }))).toEqual([]); + // An untyped field, and a dimension that writes no field / a non-string one. + expect(dimensionFindings(dimensionStack(undefined))).toEqual([]); + expect(dimensionFindings(dimensionStack({ type: 'json' }, { field: undefined }))).toEqual([]); + expect(dimensionFindings(dimensionStack({ type: 'json' }, { field: ['grouped'] }))).toEqual([]); + }); + + it('fires through runAuthoringRules on all three commands, and only on the refused dimension', () => { + const stack = dimensionStack({ type: 'json' }); + for (const command of ['validate', 'build', 'lint'] as const) { + const found = runAuthoringRules(command, { normalized: stack, parsed: stack }).filter( + (f) => f.rule === DIMENSION_RULE, + ); + expect(found, command).toHaveLength(1); + expect(found[0].severity, command).toBe('error'); + } + const control = dimensionStack({ type: 'text' }); + expect( + runAuthoringRules('lint', { normalized: control, parsed: control }).filter((f) => f.rule === DIMENSION_RULE), + ).toEqual([]); + }); +}); diff --git a/packages/lint/src/validate-dataset-measure-aggregates.ts b/packages/lint/src/validate-dataset-measure-aggregates.ts index 4a371f2e651..417ce04e625 100644 --- a/packages/lint/src/validate-dataset-measure-aggregates.ts +++ b/packages/lint/src/validate-dataset-measure-aggregates.ts @@ -4,7 +4,9 @@ * [#16354 — the AUTHORING-TIME leg of the aggregate × field-type contract] * A dataset measure pairs an `aggregate` with a `field`. This rule refuses the * pairs `AGGREGATE_FIELD_TYPE_COMPATIBILITY` (`@objectstack/spec/data`, #16353) - * does not accept, at the moment the author writes them. + * does not accept, at the moment the author writes them. [#20890] It also + * refuses a dataset DIMENSION over a JSON-stored field, which the analytics door + * refuses to group by — see "The dimension leg" below. * * All judgement lives in the SHARED predicate — * `isAggregateCompatibleWithFieldType`, the same call the compile leg makes — @@ -91,11 +93,68 @@ * registry's own definition, so `created_at` reads as the `datetime` it is and * `avg` over it is refused exactly as over an authored field — the pair reaches * the same backend either way. + * + * ## [#20890] The declaration half the per-type table cannot see + * + * The table is per TYPE, so it cannot see `multiple: true`: a `select` (or + * `radio`, `lookup`, `user`, `file`, `image`) flagged `multiple` is a list + * stored as JSON, the very storage the `count_distinct` row refuses `tags` + * for. The compile leg and the engine's `count_distinct` door both ask + * `isMultiValueField` (`@objectstack/spec/data`) beside the row, so this rule + * asks it too, in the same place: {@link acceptsDeclaration} is the row AND + * the declaration, and it decides the verdict and the accepted-aggregates hint + * alike, so the two cannot disagree. Only `count_distinct` can be moved by the + * flag — the `sum` / `avg` / `min` / `max` rows accept no multi-capable type, + * and `count` reads no value. ⛔ No second account of the pair table: the row + * is still the predicate's, the flag is still the spec's helper. + * + * ## [#20890] The dimension leg — `dimension-json-stored-field-refused` + * + * A dataset dimension is a GROUP KEY: it compiles to a cube dimension whose + * `sql` is the dimension's `field` (`dataset-compiler.ts`), and a query that + * selects it groups by that column. The analytics door + * (`service-analytics`' `structured-json-dimension-door.ts`) refuses a grouped + * member whose column is JSON-stored with `400 INVALID_FIELD`, before either + * strategy builds a statement, because a JSON value is no group key the SQL + * dialects share — one grouped each serialized value apart, another refused + * the statement. Its two predicates, in its order, are the ones this rule + * reads, called and never re-listed: + * + * - `STRUCTURED_JSON_TYPES` — `json`, `composite`, `repeater`, `record`, + * `location`, `address`, `vector`; + * - `isMultiValueField` — an inherently multi option type (`multiselect`, + * `checkboxes`, `tags`), or a multi-capable type flagged `multiple: true`. + * The same type without the flag stores one value and is served. + * + * So a dimension over such a field is refused here, where the author is + * standing, instead of on the first dashboard that selects it. There is no + * selection of it the door serves as a group: a dataset's filters name FIELDS, + * not dimensions, and the one ungrouped use of a dimension — a time-dimension + * range with no `granularity` — bounds a date, which a JSON document is not. + * The column is resolved on the object graph exactly as a measure's is (dotted + * paths included, on the object the LEAF lives on) and stays silent under the + * same skips 1, 3 and 4 above, plus a dimension that writes no `field`. + * + * **Cube dimensions are NOT judged here, because lint reads no cube.** No + * authoring rule in this package walks `analyticsCubes`: its one mention is + * `validate-field-consumers.ts`'s list of roots a field-REMOVAL census reads, + * which judges nothing about a cube member. A cube dimension is a different + * position (a record keyed by member name, its column in `sql`) on a different + * metadata type, and the registry entry that carries this rule to the runtime + * door declares `runtimeTypes: ['dataset']` only — so a cube leg would be a new + * walk and a new door, not this rule extended. + * + * The function keeps its name: it is the registry's key (`authoring-rules.ts`), + * and the dimension leg rides on that one entry — gating, on all three + * commands, and at the runtime `dataset` write door. */ import { AGGREGATE_FIELD_TYPE_COMPATIBILITY, + STRUCTURED_JSON_TYPES, isAggregateCompatibleWithFieldType, + isMultiValueField, + type ValueShapeFieldDef, } from '@objectstack/spec/data'; import { @@ -103,6 +162,7 @@ import { isUnjudgeable, recordsOf, resolveFieldPath, + type FieldPathVerdict, type ObjectGraph, } from './object-graph.js'; @@ -115,6 +175,15 @@ import { */ export const MEASURE_AGGREGATE_FIELD_TYPE_REFUSED = 'measure-aggregate-field-type-refused'; +/** + * [#20890] Stable diagnostic id for a dataset dimension whose field is + * JSON-STORED — the spec's own name for the union the analytics door refuses + * to group by (`aggregate-field-type-compatibility.ts`: the structured-JSON + * class, the multi-option types, and a multi-capable type flagged + * `multiple: true`). + */ +export const DIMENSION_JSON_STORED_FIELD_REFUSED = 'dimension-json-stored-field-refused'; + export interface DatasetMeasureAggregateFinding { /** * Always `error`. The pair is decidable from the author's own declarations — @@ -122,13 +191,14 @@ export interface DatasetMeasureAggregateFinding { * number whose value is a property of the SQL dialect. The compile leg * answers the same pair with `400 DATASET_INVALID`, so an advisory here * would only mean the author hears about it later, from someone else's - * dashboard. + * dashboard. A JSON-stored dimension is the same shape one door along: the + * analytics door answers every query grouping by it with `400 INVALID_FIELD`. */ severity: 'error'; - rule: typeof MEASURE_AGGREGATE_FIELD_TYPE_REFUSED; + rule: typeof MEASURE_AGGREGATE_FIELD_TYPE_REFUSED | typeof DIMENSION_JSON_STORED_FIELD_REFUSED; /** Human-readable location, e.g. `dataset "sales" › measure "avg_closed"`. */ where: string; - /** Config path, e.g. `datasets[0].measures[2].aggregate`. */ + /** Config path, e.g. `datasets[0].measures[2].aggregate` or `datasets[0].dimensions[1].field`. */ path: string; message: string; hint: string; @@ -159,20 +229,64 @@ const ACCEPTED_TYPES_BY_AGGREGATE: ReadonlyMap = new ), ); -/** Every aggregate the table accepts for `fieldType` — always non-empty (`count` accepts any type). */ -function aggregatesAccepting(fieldType: string): string[] { +/** + * May `aggregate` be applied to a field DECLARED as `shape`? The table's row + * for the type ({@link isAggregateCompatibleWithFieldType}) AND, for + * `count_distinct`, the declaration half the per-type row cannot see — a + * multi-capable type flagged `multiple: true` is a list stored as JSON + * ({@link isMultiValueField}). Asked the way the compile leg asks it, and the + * one place this file asks it: the verdict and the hint both read it. + */ +function acceptsDeclaration(aggregate: string, shape: ValueShapeFieldDef): boolean { + if (!isAggregateCompatibleWithFieldType(aggregate, shape.type)) return false; + return !(aggregate === 'count_distinct' && isMultiValueField(shape)); +} + +/** Every aggregate that accepts a field declared as `shape` — always non-empty (`count` accepts any type). */ +function aggregatesAccepting(shape: ValueShapeFieldDef): string[] { const accepting: string[] = []; for (const [fn] of ACCEPTED_TYPES_BY_AGGREGATE) { - if (isAggregateCompatibleWithFieldType(fn, fieldType)) accepting.push(fn); + if (acceptsDeclaration(fn, shape)) accepting.push(fn); } return accepting; } +/** The declaration as the words say it: the type, and the flag when the field carries it. */ +function declaredAs(shape: ValueShapeFieldDef): string { + return shape.multiple === true ? `\`${shape.type}\` with \`multiple: true\`` : `\`${shape.type}\``; +} + +/** The declaration the graph holds for a resolved leaf — its type and its `multiple` flag. */ +function shapeOf(verdict: Extract, fieldType: string): ValueShapeFieldDef { + return { type: fieldType, multiple: verdict.meta?.multiple === true }; +} + +/** Which object the words say declares the leaf: the dataset's own, or the joined one. */ +function declarerOf(verdict: Extract, baseObject: string): string { + return verdict.object === baseObject + ? `object "${baseObject}"` + : `object "${verdict.object}" (reached through this dataset's join chain)`; +} + /** - * Refuse every dataset measure whose `aggregate` the field's declared - * `FieldType` cannot carry. Returns findings (empty = clean). Pure - * `(stack) => Finding[]` (ADR-0019): no I/O, and safe on both the - * schema-parsed stack and the raw config the `os lint` path carries. + * [#20890] The class of a column a dimension GROUPS by, or `null` when it + * stores one scalar value — the analytics door's two predicates, in its + * order: {@link STRUCTURED_JSON_TYPES}, then {@link isMultiValueField} (the + * declaration, `multiple` included). ⛔ Never the type alone: a type-only + * reading would refuse `tags` and pass a `select` with `multiple: true`, which + * is the same JSON column. + */ +function groupKeyClassOf(shape: ValueShapeFieldDef): 'structured-json' | 'multi-value' | null { + if (STRUCTURED_JSON_TYPES.has(shape.type)) return 'structured-json'; + return isMultiValueField(shape) ? 'multi-value' : null; +} + +/** + * Refuse every dataset measure whose `aggregate` the field's declaration + * cannot carry, and every dataset dimension whose field is JSON-stored. + * Returns findings (empty = clean), each dataset's dimensions before its + * measures. Pure `(stack) => Finding[]` (ADR-0019): no I/O, and safe on both + * the schema-parsed stack and the raw config the `os lint` path carries. */ export function validateDatasetMeasureAggregates(stack: unknown): DatasetMeasureAggregateFinding[] { const findings: DatasetMeasureAggregateFinding[] = []; @@ -191,6 +305,60 @@ export function validateDatasetMeasureAggregates(stack: unknown): DatasetMeasure const dsName = strName(ds.name) ?? `#${di}`; + // ── [#20890] The dimension leg: a group key over a JSON-stored field ── + recordsOf(ds.dimensions).forEach((dimension, k) => { + // A dimension that writes no `field` has nothing to judge; a non-string + // there is the schema's refusal to give, not this rule's. + const field = strName(dimension.field); + if (!field) return; + + // ── Skip 3: the path does not resolve — `dataset-field-unknown`'s finding ── + const verdict = resolveFieldPath(graph, object, field); + if (!verdict || isUnjudgeable(verdict) || verdict.kind !== 'ok') return; + + // ── Skip 4: the leaf declares no type, so nothing can be asked about it ── + const fieldType = verdict.meta?.type; + if (!fieldType) return; + + const shape = shapeOf(verdict, fieldType); + const cls = groupKeyClassOf(shape); + if (cls === null) return; + + const dimensionName = strName(dimension.name) ?? `#${k}`; + const head = + `dimension "${dimensionName}" groups by field "${field}", which ` + + `${declarerOf(verdict, object)} declares as ${declaredAs(shape)} — `; + const door = + 'The analytics door refuses every query that groups by this dimension with ' + + '`400 INVALID_FIELD` before any SQL is built, so a report or dashboard that selects it gets ' + + 'that refusal instead of an answer.'; + findings.push({ + severity: 'error', + rule: DIMENSION_JSON_STORED_FIELD_REFUSED, + where: `dataset "${dsName}" › dimension "${dimensionName}"`, + path: `datasets[${di}].dimensions[${k}].field`, + message: + cls === 'structured-json' + ? head + + 'a structured-JSON value, which analytics does not group by. A JSON document is no ' + + 'group key the SQL dialects share: one groups each serialized document apart, ' + + `another refuses the statement. ${door}` + : head + + 'a multi-value field, which analytics does not group by. A list of values is no ' + + 'group key the SQL dialects share: one groups each serialized list apart, another ' + + `refuses the statement. ${door}`, + hint: + cls === 'structured-json' + ? 'Group by a field that stores one scalar value: store the part of the document you ' + + 'group on in a field of its own and point this dimension at that field, or remove ' + + 'the dimension.' + : `Filter by one member instead of grouping: a record query on "${verdict.object}" ` + + `with where { "${verdict.field}": { "$contains": VALUE } } counts or lists the records ` + + 'that hold VALUE, one query per member. Point this dimension at a field that stores ' + + 'one value, or remove it.', + }); + }); + recordsOf(ds.measures).forEach((measure, k) => { // ── Skip 2: nothing written in one of the two positions ── const aggregate = strName(measure.aggregate); @@ -209,31 +377,38 @@ export function validateDatasetMeasureAggregates(stack: unknown): DatasetMeasure const fieldType = verdict.meta?.type; if (!fieldType) return; - if (isAggregateCompatibleWithFieldType(aggregate, fieldType)) return; + const shape = shapeOf(verdict, fieldType); + if (acceptsDeclaration(aggregate, shape)) return; + // [#20890] The row accepts the TYPE and the declaration is what refuses: + // a multi-capable field flagged `multiple: true` under `count_distinct`. + const flaggedList = isAggregateCompatibleWithFieldType(aggregate, fieldType); const measureName = strName(measure.name) ?? `#${k}`; - const onObject = - verdict.object === object - ? `object "${object}"` - : `object "${verdict.object}" (reached through this dataset's join chain)`; findings.push({ severity: 'error', rule: MEASURE_AGGREGATE_FIELD_TYPE_REFUSED, where: `dataset "${dsName}" › measure "${measureName}"`, path: `datasets[${di}].measures[${k}].aggregate`, - message: - `measure "${measureName}" applies aggregate "${aggregate}" to field "${field}", which ` + - `${onObject} declares as \`${fieldType}\`. That pair is refused by the aggregate × ` + - `field-type compatibility table in @objectstack/spec, so the number a backend returns ` + - `for it is a property of the SQL dialect rather than of the data — one coerces the ` + - `stored form and answers something plausible, another has no such function and fails ` + - `at query time. "${aggregate}" accepts: ${accepted.join(', ')}.`, + message: flaggedList + ? `measure "${measureName}" applies aggregate "${aggregate}" to field "${field}", which ` + + `${declarerOf(verdict, object)} declares as ${declaredAs(shape)} — a list of values ` + + `stored as JSON. "${aggregate}" COMPARES the stored values for equality, and no two ` + + `backends compare a JSON-stored value alike: one counts every row apart, one compares ` + + `the serialized text, another has no equality for the type and fails at query time. ` + + `"${aggregate}" accepts: ${accepted.join(', ')}, none of them with \`multiple: true\`.` + : `measure "${measureName}" applies aggregate "${aggregate}" to field "${field}", which ` + + `${declarerOf(verdict, object)} declares as \`${fieldType}\`. That pair is refused by the aggregate × ` + + `field-type compatibility table in @objectstack/spec, so the number a backend returns ` + + `for it is a property of the SQL dialect rather than of the data — one coerces the ` + + `stored form and answers something plausible, another has no such function and fails ` + + `at query time. "${aggregate}" accepts: ${accepted.join(', ')}.`, hint: `Either point "${aggregate}" at a field of an accepted type, or aggregate ` + - `"${field}" with one its \`${fieldType}\` type accepts: ` + - `${aggregatesAccepting(fieldType).join(', ')}. ` + + `"${field}" with one its ${declaredAs(shape)} ${shape.multiple === true ? 'declaration' : 'type'} accepts: ` + + `${aggregatesAccepting(shape).join(', ')}. ` + `\`count\` accepts every type because it reads no value, and \`count_distinct\` every ` + - `type but the JSON-stored ones, whose values no two backends compare alike; a ` + + `type but the JSON-stored ones — a field declared \`multiple: true\` among them — whose ` + + `values no two backends compare alike; a ` + `quantity that must be added up or averaged has to be STORED as a ` + `numeric field (a computed column) and aggregated as one. The compile leg refuses ` + `this same pair with \`400 DATASET_INVALID\` before any SQL is emitted, so this is ` +