feat(spec)!: an analytics cube member's sql is a column reference, and the showcase done rate moves to its dataset (#20943)

.changeset/20943-cube-member-sql-column-reference.md

@@ -0,0 +1,108 @@
+---
+'@objectstack/spec': minor
+'@objectstack/service-analytics': patch
+---
+
+feat(spec)!: an analytics cube member's `sql` is a column reference — a SQL expression there is refused at parse, and a derived value is declared on an ADR-0021 dataset (#20943)
+
+Clause-②: yes (narrowing)
+
+**BREAKING** — shipped as `minor` under the launch-window convention
+(`check-changeset-no-major` refuses `major` until GA; breaking-ness is carried by
+this banner, the `(narrowing)` arm above and the ADR-0087 disposition below,
+never by the level).
+
+`MetricSchema.sql` and `DimensionSchema.sql` — the `sql` of every member in an
+analytics cube's `measures` and `dimensions` — admit a column reference only: a
+field of the cube's object (`amount`), a relationship path of bare identifiers
+ending in one (`account.amount`, `account.owner.region`), or `'*'` for a count.
+Any other value — a `CASE WHEN …`, an aggregate or a ratio of aggregates, a quoted
+or `$`-prefixed spelling, an empty string — is refused at parse with a
+prescription. This is ADR-0021's "zero raw SQL / zero raw expressions" carried
+from the dataset layer to the cube members it compiles to (maintainer ruling D on
+the card): an expression names no single field, so no platform check can judge
+which fields it reads, and the two analytics strategies never agreed on it — the
+raw-SQL path ran it verbatim and the ObjectQL path refused it. The rule is a
+`pattern` in the published JSON Schema too, so a document validated against
+`json-schema/**` is judged as the parse judges it.
+
+## FROM → TO
+
+A derived value moves to an ADR-0021 dataset over the same object. A conditional
+count or sum is a dataset measure with its own structured `filter`; a ratio, sum,
+difference or product of measures is `derived: { op, of: [...] }` over measures
+named in the same dataset.
+
+```
+FROM  defineCube({ name: 'delivery', sql: 'task', measures: {
+        done_rate: { label: 'Done Rate (%)', type: 'number',
+                     sql: "SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) * 100.0 / COUNT(*)" },
+      } })
+      -> parsed; the expression ran verbatim on one strategy and was refused on the other
+TO    -> ZodError at measures.done_rate.sql (invalid_format): `measures.<metric>.sql` is a
+         column reference: a field of the cube's object (`amount`), a relationship path ending
+         in one (`account.amount`), or `'*'` for a count. A SQL expression there was retired …
+
+      defineDataset({ name: 'task_metrics', label: 'Task Metrics', object: 'task',
+        dimensions: [/* … */],
+        measures: [
+          { name: 'task_count', aggregate: 'count' },
+          { name: 'done_count', aggregate: 'count', filter: { status: 'done' } },
+          { name: 'done_rate', derived: { op: 'ratio', of: ['done_count', 'task_count'] }, format: '0.0%' },
+        ] })
+```
+
+**Mind the scale.** A `derived` ratio is a 0–1 fraction. An expression that
+multiplied by 100 returned percentage points; pair the ratio with a `%` numeral
+pattern (the server marks a ratio column's percent scale as a fraction) and
+re-check any consumer that read the old number raw.
+
+**A dimension that bucketed a column with a CASE expression** has no expression
+form in the cube layer or the dataset layer: group by the column itself, or keep
+the bucket as a field of the object and name that field.
+
+**The one-line fix:** parse each cube; every refusal at `…sql` is one member to
+move — replace it with the column it aggregates, or move the derived value to a
+dataset measure as above, and point the dashboards, reports and queries that named
+`<cube>.<member>` at the dataset measure.
+
+**What an author who still writes it sees.** `CubeSchema`, `defineCube()`,
+`defineStack({ analyticsCubes })` (`STACK_SCHEMA_INVALID` / 422) and the
+`analytics_cube` write door refuse the member at its `sql` path with the
+prescription. `tsc` does not: the key's type is still `string`.
+
+## The retirement kit
+
+- **Schema.** `MetricSchema.sql` / `DimensionSchema.sql` carry the pattern and
+  their prescriptions (`data/analytics.zod.ts`). A column reference parses
+  byte-identically to before. The retired metric `filters` guidance and the
+  analytics query's `filters` guidance no longer offer "fold the condition into
+  the metric's own `sql` expression" as a live channel; the `metric-filters-removed`
+  conversion summary and its D3 entry and step-18 rationale fragment say the same.
+- **ADR-0087.** The D3 entry `cube-member-sql-expression-retired`, with its
+  step-18 rationale fragment. No D2 conversion — an expression has no mechanical
+  rewrite into a dataset — and no `RETIRED_KEYS_BY_MAJOR` row: no key left the
+  shape, so the authorable-surface, api-surface and JSON-schema manifest
+  ratchets are unchanged.
+- **Liveness.** The `analytics_cube` ledger rows `measures.sql` and
+  `dimensions.sql` stay `live`, re-verified, with the narrowing recorded.
+- **Docs.** The `data/analytics` reference page is regenerated.
+- **Example.** The showcase cube's `done_rate` expression member moves to the
+  `showcase_task_metrics` dataset as `done_count` (a count filtered on
+  `status: 'done'`) and `done_rate` (`ratio` over `done_count` and `task_count`,
+  format `0.0%`).
+- **`@objectstack/service-analytics`** (README only): its query-body section no
+  longer tells a reader to fold a per-metric condition into the metric's own
+  `sql` expression. The runtime is unchanged: its expression branches remain for
+  a cube that reaches the service without meeting the parse, and their deletion
+  is a separate change.
+
+## Reach, measured
+
+- This repository: one authored expression member (the showcase `done_rate`),
+  moved here. Test fixtures in `@objectstack/service-analytics` that build
+  expression members WITHOUT the parse keep exercising the runtime's expression
+  branches, unchanged.
+- Out-of-repo authored cubes: NOT MEASURED.
+
+<!-- adr-0087: registered cube-member-sql-expression-retired -->

content/docs/references/data/analytics.mdx

@@ -147,7 +147,7 @@ Type: `[string, string]`
 | **label** | `string` | ✅ | Human readable label |
 | **description** | `string` | optional |  |
 | **type** | `Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct' \| 'number' \| 'string' \| 'boolean'>` | ✅ |  |
-| **sql** | `string` | ✅ | SQL expression or field reference |
+| **sql** | `string` | ✅ | Column reference: a field of the cube's object ("amount"), a relationship path ending in one ("account.amount"), or "*" for a count. Never a SQL expression: a derived value is declared on an ADR-0021 dataset (a measure-scoped filter, or derived: `{ op, of }`). |
 | **format** | `string` | optional | Display format for this measure's result column: a numeral pattern such as "$0,0.00" or "0.0%". Relayed verbatim as fields[].format on POST /analytics/query results, and on the measure by GET /analytics/meta. |
 
 ### Nested Shape: `Cube.dimensions[string]`
@@ -158,7 +158,7 @@ Type: `[string, string]`
 | **label** | `string` | ✅ | Human readable label |
 | **description** | `string` | optional |  |
 | **type** | `Enum<'string' \| 'number' \| 'boolean' \| 'time' \| 'geo'>` | ✅ |  |
-| **sql** | `string` | ✅ | SQL expression or column reference |
+| **sql** | `string` | ✅ | Column reference: a field of the cube's object ("status") or a relationship path ending in one ("account.industry"). Never a SQL expression. |
 | **granularities** | `Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>[]` | optional | For a time dimension. A single interval is its default bucket: a query that groups by this dimension without stating a granularity is bucketed at it. Two or more intervals state no default. A granularity the query states always wins, listed or not. |
 
 ### Nested Shape: `Cube.joins[string]`
@@ -191,7 +191,7 @@ Type: `[string, string]`
 | **label** | `string` | ✅ | Human readable label |
 | **description** | `string` | optional |  |
 | **type** | `Enum<'string' \| 'number' \| 'boolean' \| 'time' \| 'geo'>` | ✅ |  |
-| **sql** | `string` | ✅ | SQL expression or column reference |
+| **sql** | `string` | ✅ | Column reference: a field of the cube's object ("status") or a relationship path ending in one ("account.industry"). Never a SQL expression. |
 | **granularities** | `Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>[]` | optional | For a time dimension. A single interval is its default bucket: a query that groups by this dimension without stating a granularity is bucketed at it. Two or more intervals state no default. A granularity the query states always wins, listed or not. |
 
 
@@ -220,7 +220,7 @@ Type: `[string, string]`
 | **label** | `string` | ✅ | Human readable label |
 | **description** | `string` | optional |  |
 | **type** | `Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct' \| 'number' \| 'string' \| 'boolean'>` | ✅ |  |
-| **sql** | `string` | ✅ | SQL expression or field reference |
+| **sql** | `string` | ✅ | Column reference: a field of the cube's object ("amount"), a relationship path ending in one ("account.amount"), or "*" for a count. Never a SQL expression: a derived value is declared on an ADR-0021 dataset (a measure-scoped filter, or derived: `{ op, of }`). |
 | **format** | `string` | optional | Display format for this measure's result column: a numeral pattern such as "$0,0.00" or "0.0%". Relayed verbatim as fields[].format on POST /analytics/query results, and on the measure by GET /analytics/meta. |
 
 

examples/app-showcase/src/data/analytics/showcase.cube.ts

@@ -35,14 +35,12 @@ export const DeliveryCube = defineCube({
       type: 'avg',
       sql: 'estimate_hours',
     },
-    done_rate: {
-      label: 'Done Rate (%)',
-      type: 'number',
-      sql: "SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) * 100.0 / COUNT(*)",
-      // A numeral pattern, the vocabulary `fields[].format` documents: `%` marks a
-      // percent, `.0` one decimal. The value above is in percentage points (0-100).
-      format: '0.0%',
-    },
+    // No `done_rate` here: a member's `sql` names a column, never a SQL
+    // expression (ADR-0021 "zero raw expressions", carried to the cube layer by
+    // #20943). The done rate this cube used to compute with a CASE expression
+    // is declared where the platform can judge every field it reads — the
+    // `showcase_task_metrics` dataset (src/ui/datasets/chart-gallery.dataset.ts),
+    // as a filtered count over a count.
   },
   dimensions: {
     status: {

examples/app-showcase/src/system/translations/index.ts

@@ -348,6 +348,18 @@ export const ShowcaseTranslationBundle = {
         },
       },
     },
+    // Translated at birth, same rule as the widgets above: the done rate moved
+    // here from the delivery cube (#20943) as two new dataset measures, born
+    // under the ratchet. The task dataset's older labels predate it and stay in
+    // the frozen baseline.
+    datasets: {
+      showcase_task_metrics: {
+        measures: {
+          done_count: { label: 'Done Tasks' },
+          done_rate: { label: 'Done Rate' },
+        },
+      },
+    },
   },
   'zh-CN': {
     objects: {
@@ -1018,6 +1030,16 @@ export const ShowcaseTranslationBundle = {
         },
       },
     },
+    // The done rate's two measures, translated at birth (#20943) — see the `en`
+    // block. The other task-dataset labels stay in the frozen baseline.
+    datasets: {
+      showcase_task_metrics: {
+        measures: {
+          done_count: { label: '已完成任务' },
+          done_rate: { label: '完成率' },
+        },
+      },
+    },
     // Page component copy became declared surface with `pages.<name>.components`
     // (#6080), so these keys are born under the ratchet: leaving any of them
     // untranslated widens the frozen baseline and fails `check-i18n-coverage`.

examples/app-showcase/src/ui/datasets/chart-gallery.dataset.ts

@@ -24,6 +24,20 @@ export const ShowcaseTaskDataset = defineDataset({
     { name: 'est_hours', label: 'Estimated Hours', aggregate: 'sum', field: 'estimate_hours', format: '0.0' },
     { name: 'avg_estimate', label: 'Avg Estimate', aggregate: 'avg', field: 'estimate_hours', format: '0.0' },
     { name: 'avg_progress', label: 'Avg Progress', aggregate: 'avg', field: 'progress', format: '0.0' },
+    // The done rate, moved here from the `showcase_delivery` cube (#20943): a
+    // cube member's `sql` is a column reference, and this is the declared
+    // form of what its CASE expression computed — a count scoped by its own
+    // structured filter, over the unfiltered count. `ratio` yields a 0–1
+    // fraction (the cube's expression multiplied by 100), which the `%`
+    // pattern displays as a percentage; the server annotates the column's
+    // scale from the operator, as it does for `paid_rate`.
+    { name: 'done_count', label: 'Done Tasks', aggregate: 'count', filter: { status: 'done' } },
+    {
+      name: 'done_rate',
+      label: 'Done Rate',
+      derived: { op: 'ratio', of: ['done_count', 'task_count'] },
+      format: '0.0%',
+    },
   ],
 });
 

examples/app-showcase/test/gap-fill.test.ts

@@ -4,9 +4,12 @@ import { existsSync } from 'node:fs';
 
 import { describe, it, expect } from 'vitest';
 import { SchemaRegistry } from '@objectstack/objectql';
+import { CubeSchema } from '@objectstack/spec/data';
+import { DatasetSchema } from '@objectstack/spec/ui';
 
 import stack from '../objectstack.config.js';
 import { DeliveryCube } from '../src/data/analytics/showcase.cube.js';
+import { ShowcaseTaskDataset } from '../src/ui/datasets/chart-gallery.dataset.js';
 import { AccountExtension } from '../src/data/extensions/account.extension.js';
 import { Account } from '../src/data/objects/account.object.js';
 
@@ -26,7 +29,7 @@ describe('showcase gap fill — analytics cube', () => {
   it('declares measures and dimensions over the delivery backbone', () => {
     expect(DeliveryCube.sql).toBe('showcase_task');
     expect(Object.keys(DeliveryCube.measures ?? {})).toEqual(
-      expect.arrayContaining(['count', 'total_estimate_hours', 'avg_estimate_hours', 'done_rate']),
+      expect.arrayContaining(['count', 'total_estimate_hours', 'avg_estimate_hours']),
     );
     expect(Object.keys(DeliveryCube.dimensions ?? {})).toEqual(
       expect.arrayContaining(['status', 'priority', 'due_date']),
@@ -63,6 +66,36 @@ describe('showcase gap fill — analytics cube', () => {
   });
 });
 
+describe('showcase gap fill — the done rate is a dataset measure, not a cube expression (#20943)', () => {
+  it('the cube declares no expression member: the shipped literal passes the narrowed member `sql` contract', () => {
+    // A cube member's `sql` names a column (ADR-0021 "zero raw expressions",
+    // carried to the cube layer). `defineCube` already parsed this file at
+    // import; re-parsing here pins that the shipped literal still passes the
+    // narrowed contract, member by member.
+    expect(CubeSchema.safeParse(DeliveryCube).success).toBe(true);
+    expect(Object.keys(DeliveryCube.measures ?? {})).not.toContain('done_rate');
+  });
+
+  it('the task dataset carries it as a filtered count over the unfiltered count, and parses', () => {
+    const measures = new Map(ShowcaseTaskDataset.measures.map((m) => [m.name, m]));
+    // The numerator: a count scoped by its OWN structured filter — the half of
+    // the old CASE expression the platform can now judge (it names `status`).
+    expect(measures.get('done_count')).toMatchObject({ aggregate: 'count', filter: { status: 'done' } });
+    expect(measures.get('done_count')?.field).toBeUndefined();
+    // The denominator is the dataset's existing unfiltered count.
+    expect(measures.get('task_count')).toMatchObject({ aggregate: 'count' });
+    expect(measures.get('task_count')?.filter).toBeUndefined();
+    // The rate: a ratio of the two, a 0–1 fraction shown through the `%` pattern.
+    expect(measures.get('done_rate')).toMatchObject({
+      derived: { op: 'ratio', of: ['done_count', 'task_count'] },
+      format: '0.0%',
+    });
+    // `defineDataset` is an identity helper, so the parse is asserted here: the
+    // strict shape and its cross-measure refinement (every `of` name declared).
+    expect(DatasetSchema.safeParse(ShowcaseTaskDataset).success).toBe(true);
+  });
+});
+
 describe('showcase gap fill — named import mapping (#2611)', () => {
   it('is wired into the stack definition', () => {
     const mappings = (stack as { mappings?: Array<{ name: string }> }).mappings ?? [];

packages/services/service-analytics/README.md

@@ -104,9 +104,10 @@ is rejected rather than dropped.
 
 There is no `filters` key and no `aggregations` key. `filters` is rejected at the REST
 door with a 400 naming `where`. There is no per-metric filter key either — the cube
-metric's `filters` was removed (#10414: no strategy ever read it); fold a per-metric
-condition into the metric's own `sql` expression, or use an ADR-0021 dataset measure's
-structured `filter`.
+metric's `filters` was removed (#10414: no strategy ever read it). A measure that counts
+or sums only some rows is an ADR-0021 dataset measure with its own structured `filter`;
+a cube member's `sql` is a column reference (a field, a relationship path ending in one,
+or `'*'`), never a SQL expression (#20943).
 
 ```typescript
 const revenueByStatus = await analytics.query({

packages/services/service-analytics/src/__tests__/cube-authored-format-granularity.test.ts

@@ -274,13 +274,19 @@ describe('analytics_cube.dimensions.granularities — the declared single granul
   });
 
   it('DECLARED NARROWING: a custom-SQL measure grouped by a declared-default dimension gets the refusal a stated granularity gets', async () => {
-    const withExpression: Cube = CubeSchema.parse({
+    // NOT parsed: since #20943 the cube contract admits a column reference
+    // only, so `CubeSchema` refuses this expression member at every authoring
+    // door. The engine path's refusal it pins is still owed to a cube that
+    // reaches the service without meeting that parse (a host registering one
+    // in-process), so the member is built directly on the parsed cube.
+    const withExpression: Cube = {
       ...authored,
       measures: {
         ...authored.measures,
         done_rate: { label: 'Done', type: 'number', sql: "SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END) * 1.0 / COUNT(*)" },
       },
-    });
+    };
+    expect(CubeSchema.safeParse(withExpression).success).toBe(false);
     const aggregated: string[] = [];
     const sqls: string[] = [];
     const service = new AnalyticsService({