fix(analytics): the native-SQL strategy applies the engine's aggregate policies — double accumulation, the PostgreSQL boolean cast and the empty-sum fold, hoisted into core (#21042)

.changeset/21042-analytics-native-aggregate-policies.md

@@ -0,0 +1,27 @@
+---
+'@objectstack/core': minor
+'@objectstack/driver-sql': patch
+'@objectstack/service-analytics': patch
+---
+
+fix: the analytics native-SQL path aggregates with the engine's own aggregate policies, so one query answers one number whichever strategy serves it: `sum` / `avg` accumulate in double, a PostgreSQL boolean aggregand is cast, and an all-NULL `sum` answers `0`. The operand policies move from `@objectstack/driver-sql` to `@objectstack/core` (#21042)
+
+Clause-②: yes (widening)
+
+**New exports.** `@objectstack/core` exports the aggregate operand policies, moved here from `@objectstack/driver-sql`, where they were module-private. The driver now imports them and emits byte-identical statements.
+
+- `AGGREGATE_ACCUMULATION`: what each declared aggregate function accumulates in on PostgreSQL and MySQL. `avg` accumulates in double; `sum` accumulates in double over a fractional column; the counts, `min` and `max` take the column as stored.
+- `aggregandColumnClass(shape)`: the one column-class predicate those policies read, over a column's declared `{ type, multiple }`. It answers `'fractional'`, `'integral'`, `'boolean'`, or `undefined` for every other column, a multi-valued one included. The type `AggregandColumnClass` names the three classes.
+- `POSTGRES_BOOLEAN_AGGREGAND_CAST`: the functions whose boolean aggregand is cast to `int` on PostgreSQL. These are `sum`, `avg`, `min` and `max`; the two counts are never cast.
+- `doubleAccumulationOperand(operand, dialect)`: the column's text, parsed as a double, spelled for `'postgres'` or `'mysql'`.
+- `aggregandOperandSql(func, columnClass, dialect, operand)`: the operand an aggregate wraps, with the cast inside the double operand. The type `AggregandSqlDialect` names its dialects (`'sqlite'`, `'postgres'`, `'mysql'`, `'unknown'`).
+
+**What changed.** `POST /api/v1/analytics/query` and `POST /api/v1/analytics/dataset/query` served by `NativeSQLStrategy` (the default on a SQL driver) skipped three policies `SqlDriver.aggregate()` applies. So the ObjectQL strategy and `engine.aggregate` answered differently for the same query. Measured on SQLite and PostgreSQL 16.13:
+
+- On PostgreSQL, `sum` / `avg` over an exact-decimal column, and `avg` over an integer one, added exact decimals. For example, `0.1 + 0.2` answered `0.3` and `11 / 9` answered `1.222222222222222`, where the engine answers `0.30000000000000004` and `1.2222222222222223`. The native statement now accumulates in double, as the driver does.
+- On PostgreSQL, `sum` / `avg` / `min` / `max` over a boolean field answered `500` (`function sum(boolean) does not exist`). The native statement now casts the boolean aggregand to `int`, as the driver does, and answers the numbers the engine answers.
+- On every dialect, a group whose aggregand is NULL in every row, and a measure-scoped `sum` that admits no row, answered `sum` `null` at the cube door. The strategy now folds a `null` answer to `emptyGroupValueFor` (`@objectstack/spec`) for every measure, so that `sum` answers `0`. `avg`, `min` and `max` over nothing stay `null`. The dataset door already answered `0`.
+
+This is no narrowing: each answer moves to the value the platform already declared for the same query.
+
+**What did not move.** `@objectstack/driver-sql`'s statements and answers are unchanged: a move-proof test compiles each aggregate function over each column class on SQLite, PostgreSQL and MySQL, and the statements equal the ones captured before the move. SQLite's native statement is unchanged, because neither operand policy applies there. A host that relays no field declarations to the analytics service, or names no SQL dialect, gets today's native arithmetic.

packages/core/src/utils/aggregate-answer.test.ts

@@ -11,11 +11,30 @@
  * `numeric` to strings), and the large totals are the ones #20335 pinned at the
  * driver door. Each face pins the same presentation through its own door, in
  * its own package, on SQLite and PostgreSQL.
+ *
+ * [#21042] The operand policies, moved here from `driver-sql` so the
+ * analytics native-SQL face applies them too: `AGGREGATE_ACCUMULATION`
+ * (#20387), the column class it and the boolean cast read
+ * (`aggregandColumnClass`), the PostgreSQL boolean-aggregand cast (#11635) and
+ * the double operand. The SQL text below is the text `driver-sql` emitted
+ * before the move (its move proof, `sql-driver-21042-aggregate-policy-move
+ * .test.ts`, pins the whole statements); each face pins the answers through
+ * its own door.
  */
 
 import { describe, it, expect } from 'vitest';
-import { AggregationFunction } from '@objectstack/spec/data';
-import { AGGREGATE_ANSWER_KIND, presentAsNumber } from './aggregate-answer';
+import { AggregationFunction, FieldType } from '@objectstack/spec/data';
+import {
+  AGGREGATE_ACCUMULATION,
+  AGGREGATE_ANSWER_KIND,
+  POSTGRES_BOOLEAN_AGGREGAND_CAST,
+  aggregandColumnClass,
+  aggregandOperandSql,
+  doubleAccumulationOperand,
+  presentAsNumber,
+  type AggregandColumnClass,
+  type AggregandSqlDialect,
+} from './aggregate-answer';
 
 describe('[#20889] AGGREGATE_ANSWER_KIND — what each declared aggregate function answers', () => {
   it('has exactly one row per declared aggregate function', () => {
@@ -74,3 +93,148 @@ describe("[#20889] presentAsNumber — the 'number' presenter", () => {
     expect(presentAsNumber(date)).toBe(date);
   });
 });
+
+describe('[#20387, #21042] AGGREGATE_ACCUMULATION — what each declared aggregate function accumulates in', () => {
+  it('has exactly one row per declared aggregate function', () => {
+    expect(Object.keys(AGGREGATE_ACCUMULATION).sort()).toEqual([...AggregationFunction.options].sort());
+  });
+
+  it('avg accumulates in double, sum in double over a fractional column, the rest as stored', () => {
+    expect(AGGREGATE_ACCUMULATION).toEqual({
+      count: 'as-stored',
+      count_distinct: 'as-stored',
+      sum: 'double-over-fractional',
+      avg: 'double',
+      min: 'as-stored',
+      max: 'as-stored',
+    });
+  });
+});
+
+describe('[#11635, #21042] POSTGRES_BOOLEAN_AGGREGAND_CAST — the functions a PostgreSQL boolean aggregand is cast for', () => {
+  it('has exactly one row per declared aggregate function', () => {
+    expect(Object.keys(POSTGRES_BOOLEAN_AGGREGAND_CAST).sort()).toEqual([...AggregationFunction.options].sort());
+  });
+
+  it('sum, avg, min and max cast; the two counts never do', () => {
+    expect(POSTGRES_BOOLEAN_AGGREGAND_CAST).toEqual({
+      count: false,
+      count_distinct: false,
+      sum: true,
+      avg: true,
+      min: true,
+      max: true,
+    });
+  });
+});
+
+describe('[#20387, #11635, #21042] aggregandColumnClass — the one column-class predicate over a declared shape', () => {
+  const FRACTIONAL = ['number', 'currency', 'percent', 'slider', 'progress', 'summary'];
+  const INTEGRAL = ['rating'];
+  const BOOLEAN = ['boolean', 'toggle'];
+
+  it('classes every declared field type: fractional, integral, boolean, or none', () => {
+    for (const type of FieldType.options) {
+      const want: AggregandColumnClass | undefined = FRACTIONAL.includes(type)
+        ? 'fractional'
+        : INTEGRAL.includes(type)
+          ? 'integral'
+          : BOOLEAN.includes(type)
+            ? 'boolean'
+            : undefined;
+      expect(aggregandColumnClass({ type }), type).toBe(want);
+    }
+  });
+
+  it('every member of the three classes is a declared field type, so none of them is a typo', () => {
+    for (const type of [...FRACTIONAL, ...INTEGRAL, ...BOOLEAN]) expect(FieldType.options, type).toContain(type);
+  });
+
+  it("classes driver-sql's internal column aliases: float is fractional, integer and int integral", () => {
+    expect(aggregandColumnClass({ type: 'float' })).toBe('fractional');
+    expect(aggregandColumnClass({ type: 'integer' })).toBe('integral');
+    expect(aggregandColumnClass({ type: 'int' })).toBe('integral');
+  });
+
+  it('a multi-valued column (a JSON list) is in no class', () => {
+    expect(aggregandColumnClass({ type: 'select', multiple: true })).toBeUndefined();
+    expect(aggregandColumnClass({ type: 'tags' })).toBeUndefined();
+    expect(aggregandColumnClass({ type: 'lookup', multiple: true })).toBeUndefined();
+  });
+
+  it('`multiple` on a type that cannot be multi-valued changes nothing, as storage ignores it', () => {
+    expect(aggregandColumnClass({ type: 'number', multiple: true })).toBe('fractional');
+    expect(aggregandColumnClass({ type: 'rating', multiple: true })).toBe('integral');
+    expect(aggregandColumnClass({ type: 'boolean', multiple: true })).toBe('boolean');
+  });
+
+  it('no declaration is no class', () => {
+    expect(aggregandColumnClass(undefined)).toBeUndefined();
+    expect(aggregandColumnClass(null)).toBeUndefined();
+    expect(aggregandColumnClass({})).toBeUndefined();
+    expect(aggregandColumnClass({ type: 'string' })).toBeUndefined();
+  });
+});
+
+describe('[#20387, #21042] doubleAccumulationOperand — the column text, parsed as a double', () => {
+  it('PostgreSQL and MySQL each spell it their way', () => {
+    expect(doubleAccumulationOperand('"amount"', 'postgres')).toBe('cast(cast("amount" as text) as double precision)');
+    expect(doubleAccumulationOperand('`amount`', 'mysql')).toBe('cast(cast(`amount` as char) as double)');
+  });
+});
+
+describe('[#20387, #11635, #21042] aggregandOperandSql — the operand each face aggregates', () => {
+  const X = 'COL';
+  const pg = (f: AggregationFunction, c: AggregandColumnClass | undefined) => aggregandOperandSql(f, c, 'postgres', X);
+  const my = (f: AggregationFunction, c: AggregandColumnClass | undefined) => aggregandOperandSql(f, c, 'mysql', X);
+  const PG_DOUBLE = (x: string) => `cast(cast(${x} as text) as double precision)`;
+  const MY_DOUBLE = (x: string) => `cast(cast(${x} as char) as double)`;
+
+  it('PostgreSQL: sum over a fractional column and avg over every class accumulate in double', () => {
+    expect(pg('sum', 'fractional')).toBe(PG_DOUBLE(X));
+    expect(pg('avg', 'fractional')).toBe(PG_DOUBLE(X));
+    expect(pg('avg', 'integral')).toBe(PG_DOUBLE(X));
+    expect(pg('sum', 'integral'), 'an integer total stays exact').toBe(X);
+  });
+
+  it('PostgreSQL: a boolean aggregand is cast to int for sum / avg / min / max, inside the double operand', () => {
+    expect(pg('sum', 'boolean')).toBe(`cast(${X} as int)`);
+    expect(pg('avg', 'boolean')).toBe(PG_DOUBLE(`cast(${X} as int)`));
+    expect(pg('min', 'boolean')).toBe(`cast(${X} as int)`);
+    expect(pg('max', 'boolean')).toBe(`cast(${X} as int)`);
+  });
+
+  it('PostgreSQL: the counts and min / max over a numeric column take the column as stored', () => {
+    for (const c of ['fractional', 'integral', 'boolean'] as const) {
+      expect(pg('count', c), `count ${c}`).toBe(X);
+      expect(pg('count_distinct', c), `count_distinct ${c}`).toBe(X);
+    }
+    expect(pg('min', 'fractional')).toBe(X);
+    expect(pg('max', 'integral')).toBe(X);
+  });
+
+  it('MySQL: the same accumulation, and no boolean cast (a boolean is tinyint(1) there)', () => {
+    expect(my('sum', 'fractional')).toBe(MY_DOUBLE(X));
+    expect(my('avg', 'integral')).toBe(MY_DOUBLE(X));
+    expect(my('avg', 'boolean')).toBe(MY_DOUBLE(X));
+    expect(my('sum', 'boolean')).toBe(X);
+    expect(my('min', 'boolean')).toBe(X);
+    expect(my('sum', 'integral')).toBe(X);
+  });
+
+  it('a column in no class is aggregated as stored on every dialect', () => {
+    for (const d of ['postgres', 'mysql', 'sqlite', 'unknown'] as const) {
+      for (const f of AggregationFunction.options) expect(aggregandOperandSql(f, undefined, d, X), `${d} ${f}`).toBe(X);
+    }
+  });
+
+  it('SQLite and an unnamed dialect apply neither policy: SQLite already adds doubles and stores booleans as 0 / 1', () => {
+    for (const d of ['sqlite', 'unknown'] as const satisfies readonly AggregandSqlDialect[]) {
+      for (const f of AggregationFunction.options) {
+        for (const c of ['fractional', 'integral', 'boolean'] as const) {
+          expect(aggregandOperandSql(f, c, d, X), `${d} ${f} ${c}`).toBe(X);
+        }
+      }
+    }
+  });
+});