Repository navigation
fix(core,driver-sql,service-analytics): the analytics native-SQL path answers measures declared number as numbers (#20889) - #21040
Conversation
…mbers on the analytics native-SQL path (red) Red-first pins for #20889. On PostgreSQL the analytics native-SQL path answers count / count_distinct / sum / avg and numeric min / max as strings while fields[] declares number; SQLite answers numbers. - core: aggregate-answer.test.ts pins AGGREGATE_ANSWER_KIND and the 'number' presenter (red: the module does not exist yet). - service-analytics: the cube read and the dataset door over a real engine and SqlDriver, SQLite and a named-skip PostgreSQL cell. - rest: POST /api/v1/analytics/dataset/query on both cells, and the lenient row_count pin (number or string) tightened to a number. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
… answers measures declared number as numbers (#20889) On PostgreSQL, NativeSQLStrategy returned the SQL client's rows unshaped, so count / count_distinct / sum / avg and numeric min / max came back as strings while fields[] declared number. driver-sql's own aggregate() has presented those answers since #20335, through a private table and presenter. - core: utils/aggregate-answer.ts receives AGGREGATE_ANSWER_KIND (its docblock moved byte-identical) and the 'number' presenter, presentAsNumber (the body of presentReadValue's 'number' arm). Exported from the root beside compensatedSum. - driver-sql: the definition becomes an import; presentReadValue's 'number' arm calls presentAsNumber. No behaviour change. - service-analytics: NativeSQLStrategy.execute presents each measure column keyed on its declared aggregate function, and min / max only over a column declared numeric (declaredFieldType). Expression measures are untouched. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift CheckThis PR changes 3 package(s): 6 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:
⛔ 3 release-owned page(s) also name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 35 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 57fed1f5f61445066b13c8b00611ecfa264fe3d0 && git checkout 57fed1f5f61445066b13c8b00611ecfa264fe3d0
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 2f2fa11d756f665a4c06160480c1dce15b9d67a4 e827c237d1a18f469e33955474d81e8c3dcc9e56 && git checkout -B drift-repro 2f2fa11d756f665a4c06160480c1dce15b9d67a4 && git merge --no-ff e827c237d1a18f469e33955474d81e8c3dcc9e56
node scripts/docs-audit/affected-docs.mjs --json 2f2fa11d756f665a4c06160480c1dce15b9d67a4
|
Contract reviewServed-tier: Inputs: card #20889 (body and all 8 comments, the route ruling ① Derived judgments
② Semver level
③ Boundary flagsThe build report's Dev flags, from the acceptance notes and
Implemented-by: VERDICT: PASS |
…n reads the object its lookup field declares (objectstack-ai#20986) (objectstack-ai#21088) Fixes objectstack-ai#20986 Clause-②: yes (narrowing) ## What this changes A relationship-path hop the cube declares no join for now reads the object its lookup field DECLARES as its target, the field's `reference`. Before, such a hop fell back to its ALIAS, the lookup field's own name. An inferred cube declares no join, so a dotted path through a lookup named differently from its target (`owner` referencing a person object) named no object at all. The door admitted, and refused, the field's name as if it were an object, and the strategies joined and read a table of that name. One resolver, `service-analytics/src/hop-object.ts` (`resolvePathHops`), names a hop's object in three tiers, first answer wins: 1. the cube's declared join at that path, keyed by the path with its dots as `__`. An authored cube that declares its join keeps it; 2. the relationship field's declared `reference`, asked of the host's `relationshipResolver`. `AnalyticsServicePlugin` already wires it from the data engine's object schema, through `referenceCarrierOf`, for dataset compilation; 3. the alias itself, when the host cannot answer. Every reader takes its answer from there, with the same function: the door's field gate and its admitted and scoped set (`fieldsOfColumnSql`, `queryObjects`), and both strategies through the context's new `relationshipReference`, which the service sets to the very function its own gate uses. There is no second resolution. - `NativeSQLStrategy`: each registered join records the object it reads (`StatementJoins`), and the read scope applied to the alias is that recorded object's. The join table and the scoped object are one value. - `ObjectQLStrategy`: the cross-object plan and the FK-expand read (and its read scope) take the hop's object from the resolver. A lookup whose declared target is the base object itself, a self-reference such as `parent`, still plans as a cross-object hop. ## Per site (H2), file:line on base `8f784959cf` and on head `e75208968e` | site | base | role | head | |---|---|---|---| | `analytics-service.ts` `fieldsOfColumnSql` (via `namedQueryFields`) | `:428` `joins?.[alias]?.name`, else alias | the field gate, AND the admitted and scoped set (`queryObjects` `:1612`, `assertFieldsReadable` `:1682`) | `:432` `resolvePathHops`; callers `:1652`, `:1722` pass `this.hopReference` | | `analytics-service.ts` `cubeObjects` | `:1634` `j?.name ?? alias` | admits and scopes the DECLARED joins only | unchanged `:1674`; `?? alias` is unreachable, `CubeJoin.name` is required by the spec | | `native-sql-strategy.ts` `crossFieldComparisonIn` fallback | `:364` `cube.joins?.[alias]?.name ?? alias` | reads scopes for the decline, declared joins only, only for a context without the door's set | unchanged `:388`; the door always passes its set | | `native-sql-strategy.ts` `generateSql` read-scope loop | `:686` `cube.joins?.[alias]?.name ?? alias` | scopes each joined alias | `:751` `join.object`, the object the join recorded | | `native-sql-strategy.ts` `qualifyAndRegisterJoin` | `:848` `cube?.joins?.[alias]?.name ?? alias` | joins | `:904` `resolvePathHops` | | `native-sql-strategy.ts` `resolveStorageTarget` | `:1044` `cube.joins?.[joinAlias(relPath)]?.name ?? relPath` | reads (declared type, temporal coercion) | `:1104` `columnObjectOf` | | `native-sql-strategy.ts` `canHandle` external decline | `:169` declared join targets | reads (routing), declared joins only | unchanged `:193`; see Acceptance notes | | `objectql-strategy.ts` `isCrossObjectField` | `:724` `cube.joins?.[alias]?.name ?? alias` | reads (plans the FK-expand or the refusal) | `:736` `resolvePathHops` | | `objectql-strategy.ts` `planCrossObject` cross dimension | `:1054` `refObject: cube.joins?.[alias]?.name ?? alias` | reads and scopes (the FK-expand `executeAggregate` and its `getReadScope`) | `:1070` `resolvePathHops` | | `objectql-strategy.ts` `resolveStorageTarget` | `:1456` `... ?? relPath` | reads (declared type for the lowering and the echo) | `:1478` `columnObjectOf` | | `structured-json-dimension-door.ts` `columnOf` | `:192` declared join only, stands down otherwise | judges a grouped column's class | unchanged; outside this claim, see Acceptance notes | | `dataset-compiler.ts` | `:648` `relationshipResolver` per `include` | joins (compiled, declared) | unchanged: already tier 2's source, read into tier 1 | **H3, where the declared reference is reachable.** At the door, `AnalyticsServiceConfig.relationshipResolver`. `plugin.ts` answers it from `engine.getObject(base).fields[rel]` for a `lookup` or `master_detail` field, through `referenceCarrierOf`. At the strategies it was not reachable on base: `StrategyContext` carried a field's type and value shape (`declaredFieldType`, `declaredValueShape`), never its reference. It is now, as `DatasetScopedStrategyContext.relationshipReference`, which is package-internal. The context type is not exported. The member's one hit in the built `dist/index.d.ts` is prose in the doc comment of the private `hopReference` field, against 19 hits for `StrategyContext`. A host returning a `RelationshipTarget` is read by its `object`. **H4, the divergence check, both before and after.** On base, every site resolved the alias (the reads site resolved a dotted `relPath`), so the object admitted and the object joined were the same name. After, the six changed sites call `resolvePathHops` with the same function. The native join and the scope applied to it are one recorded value. The unchanged sites read declared joins only (tier 1), which the resolver returns verbatim. No site uses the alias where another uses the reference, so there is no `security` stop. Measured on the route pin: the security spy shows `canReadObject` asked only for the base object and the target. ## Per row (H1), measured in the shipped composition Fixture: real `SecurityPlugin` and `AnalyticsServicePlugin` over `ObjectQL` on SQLite. A member who may read everything except two objects. The "before" column was read at base `8f784959cf`. The decoy row's "before" was read on the ablation leg below, whose resolver is the base one. The "after" column was read at the fix commit `bf58903995`. The member is the caller. At head `e75208968e`, the two pins re-ran green over these rows: the readable dimension and filter, the unreadable target, the self-reference, and the controls. | strategy | lookup | target | position | before | after | |---|---|---|---|---|---| | native | named differently (`owner`) | readable | dimension | 403 naming `owner` (system caller: 500 `DATABASE_ERROR`) | rows, equal to a declared join's | | native | named differently | readable | filter member | 403 naming `owner` (system: 500) | rows, equal to a declared join's and to the nested form `{ owner: { region } }` | | native | named differently (`keeper`) | unreadable | dimension, filter | 403 naming `keeper` | 403 naming the target | | native | named after its target (control) | readable / unreadable | dimension | rows / 403 naming it | unchanged | | native | self-reference (`parent`) | the base object | dimension, filter | 403 naming `parent` | rows | | native | named after ANOTHER object (decoy) | unreadable | dimension | answered from the other object's table | 403 naming the target | | ObjectQL | named differently | readable | dimension | 403 naming `owner` (system: 500) | rows (FK-expand), equal to a declared join's | | ObjectQL | named differently | readable | filter member | 403 naming `owner` (system: 400) | 400 `INVALID_FIELD`, the strategy's own capability refusal, equal to a declared join's | | ObjectQL | named differently | unreadable | dimension, filter | 403 naming `keeper` | 403 naming the target | | ObjectQL | named after its target (control) | readable / unreadable | dimension | rows / 403 naming it | unchanged | | ObjectQL | self-reference | the base object | dimension | 403 naming `parent` | rows (FK-expand) | | ObjectQL | named after another object (decoy) | unreadable | dimension | answered from the other object | 403 naming the target | **`Clause-②: yes (narrowing)`, measured.** Widening: the first rows move from refused to served. Narrowing: the decoy rows move from answered to `403`. A lookup whose name is also the name of ANOTHER object used to read that other object's rows, by the ids of the records the field points to. The dispatch expected a plain `yes`; the measured grammar adds the arm. Line 2 reads `declared · yes · narrowing` through `scripts/pm/clause2-line.mjs`'s `readClause2Line`. The changeset is `minor`, carries the BREAKING banner, and carries its ADR-0087 marker (`not-required (no-migration-prescription)`). ## Pins, red first, and the ablation - **Unit pin** `service-analytics/src/__tests__/hop-object-reference-resolution.test.ts` (28 cases), on both strategies. It covers five positions: an inferred dimension, a filter member, a time-dimension window, an authored member over an undeclared relationship, and a two-hop path's second hop. For each, an unreadable target is refused by name before anything runs, on the read and on the echo. It also pins: - admission and the read scope ask one set, carrying the target and never the field's name; - the field gate judges the column on the target; - the native join and its scope name the target; - the native strategy asks the target's declared column type; - the ObjectQL FK-expand reads the target under its scope; - the self-reference on both strategies; - and two controls: a lookup named after its target, and a declared join is kept. - **Route pin** `packages/rest/src/analytics-hop-object-reference.test.ts` (10 cases), once per strategy. Every answer is compared with the same question through a DECLARED join and checked absolutely. It covers: - readable dimension: rows, with only the base and the target admitted (security spy); - readable filter: rows on native, equal to the nested form; ObjectQL's own 400; - unreadable dimension and filter: 403 naming the target, on the read and the echo; - the control. - **Red first.** Pins committed at `f31ca819e2`, before the fix at `bf58903995`: unit 24 failed / 4 passed (the 4 are the controls), route 8 failed / 2 passed (the controls). Every failure sits at the assertion naming the target, never at a reference assertion. - **Ablation A1, the resolver falls back to the alias again.** Predicted in writing before the run: exactly the red set above. - `scripts/ablation-replace.mjs` replaced tier 2's `const reference = referenceOf?.(from, field);` with `const reference = undefined as string | undefined;`: anchor 1 to 0, blob `489bc64f35f0` to `a783c0f62a1c`. The package was rebuilt, and `ablation-dist-preflight --absent` found the marker gone from all 6 built files. The pristine build carried it in `dist/index.js` and `dist/index.cjs`, 1 each. - Result: unit 24 failed / 4 passed, route 8 failed / 2 passed. The failing names are identical, as sorted lists, to the pins-first red set. - Restore: by absolute path, under an EXIT/INT/TERM trap. The blob equals the HEAD blob (`489bc64f35f0`), `git diff HEAD` is empty, and porcelain is 0. After a rebuild, the preflight found the marker present in 2 built files, and the tree clean. The pins then read unit 28/28, route 10/10. ## Tests and gates, head `e75208968e` Each command below ran under `os-verify-lock`, in ONE locked sequential script, on head `e75208968e`. The script printed `git rev-parse --short HEAD` at its start and its end, with a clean porcelain both times. Each exit code was captured before any pipe. - **Build:** `turbo run build --filter=!@objectstack/docs --concurrency=1`, 72 of 72 tasks, exit 0. - **Gate union:** `dispatch-gates --commands --repo objectstack-ai/objectstack` derives 62 commands from this diff's 8 paths, the same 62 as on the first merge. All 62 exit 0. - `dispatch-gates --ran`: `62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN`. - Verdict lines include `check:nul-bytes` (OK, 9671 files, no raw control bytes), `check:cross-package-test-inputs` ("29 package(s) read outside themselves, all declared"), `check:engine-double-contract`, `check:dual-build-cjs-loads`, `check:type-check-debt` (no entry above its count), `check-adr-0087-registration` (the `no-migration-prescription` exemption read from this changeset) and `check-empty-changeset`. - `check-changeset-no-major`'s clause-② axis reads the PR payload, so it is NOT APPLICABLE locally and is CI's. - **The four ⛔ roster families the derivation names, run anyway:** `check-changeset-fixed`, `check:authz-resolver`, `check:error-code-casing` and `check:filter-alias-parity`. All exit 0. - **`@objectstack/service-analytics`:** `vitest run --maxWorkers=2`: 155 files, 3526 passed, 10 skipped. `typecheck` exits 0, and `--listFiles` counts 152 of the 152 `__tests__` files, the unit pin among them, in the program. - **The rest analytics pins:** every `src/analytics-*.test.ts` plus `data-nested-relation-permission.test.ts`, 20 files: 243 passed, 8 skipped. `@objectstack/rest` `typecheck` exits 0, with `check:test-typecheck` OK; the route pin is in `tsconfig.test.json`'s program (`--listFiles`). - **The client census** `envelope-caller-census.test.ts`: 20 of 20. Neither pin calls the censused spelling: the pins call `service.query(`, never `analytics.query(`. - **ESLint, a proven narrowing.** - The population is the 7 changed `.ts` files (`git diff --name-only 2821e9f...HEAD`). Per eslint's own config, `isPathIgnored` is false for all 7, and `parserOptions.project` and `projectService` are unset for all 7. - `lintFiles` gives 7 results, 0 errors, 0 warnings. - With no type-aware lint, this diff cannot move an untouched file's verdict. - **On the first merge (`cff14ac80f`)**, the same union was 62 of 62 exit 0, and the four rosters were green. Analytics was 154 files / 3521 passed, and the rest pins 19 files / 242 passed. ## Surface Within the claim's regions: `fieldsOfColumnSql` and `queryObjects` in `analytics-service.ts`; the hop-object sites in both strategies; the new module; the pins; and the changeset. ⛔ Not touched: the native `execute`, `buildFilterClause`, `convertFilter`, `packages/objectql/src/**` or `packages/spec/src/**`. Four further lines carry the resolver to the strategies, outside the claim's listed regions: - `analytics-service.ts`: the `hopReference` class field, `baseCtx.relationshipReference`, the `assertFieldsReadable` call to `namedQueryFields`, and the `relationshipResolver` config doc; - `strategies/types.ts`: one member, `relationshipReference`, on the package-internal context type. No in-flight sibling PR touches them. `main` was merged twice, with no rebase: `5dbeb7d7b7` brought PR objectstack-ai#21036 (`convertFilter`), and `2821e9f15b` brought PR objectstack-ai#21040 (`execute`) and PR objectstack-ai#21037. Both merges were clean, and the branch delta against `main` stayed the same 8 files. ## Acceptance notes - **The ObjectQL strategy still refuses a dotted FILTER or time window through any relationship** with its own `400 INVALID_FIELD`, as it does through a declared join. The engine serves the nested form there. Lowering the dotted filter onto the nested form for the engine belongs to `convertFilter`, outside this claim. The dimension position is served (FK-expand). - **The grouped-column door** (`structured-json-dimension-door.ts` `columnOf`) judges declared joins only and stands down on a path the cube does not declare. A multi-value or structured-JSON column reached through an undeclared path is not judged there. That was already true for a lookup named after its target, and is now reachable for one named differently. NOT MEASURED. Outside this claim. - **The native external-object decline** (`canHandle`) reads declared joins only, so an undeclared hop to a federated object is not declined. That holds for same-named lookups on base, and now for differently named ones. NOT MEASURED: there is no federated fixture. - **The ObjectQL SQL echo for a self-reference** prints an unaliased self-join. It describes a statement that is not the FK-expand the query runs. NOT MEASURED by a pin; it affects the echo only. - **A null FK** buckets as `(restricted)` on the ObjectQL FK-expand and as null on native. This is pre-existing, and the same for a lookup named after its target. - **A hop no tier answers** (a field that declares no reference, or a host with no resolver) reads the alias, as before. For a multi-hop path that fallback is now spelled as the join alias (`a__b`) on the two reads sites, where it was the dotted path. Either spelling names no object. - **`relationshipResolver`'s warn** for a field whose `reference` is not a string is now reachable per query hop, not only at dataset compilation. NOT MEASURED. - **The dataset door** with a selection naming an undeclared, differently named path is NOT MEASURED. The route pin covers the cube read and the echo. - **Drivers.** SQLite only. The resolution is computed before any driver. - **CI is not awaited.** These lanes are CI's: the test shards, dogfood, temporal conformance, build core, the workspace type-check lanes, and the wide-population and roster families the derivation lists outside its total. --- _Generated by [Claude Code](https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
…e policies — double accumulation, the PostgreSQL boolean cast and the empty-sum fold, hoisted into core (objectstack-ai#21042) (objectstack-ai#21209) Fixes objectstack-ai#21042 Clause-②: yes (widening) ## What this changes The analytics native-SQL strategy (`NativeSQLStrategy`, the default on a SQL driver) skipped three aggregate policies that `SqlDriver.aggregate()` applies. So one route answered different numbers, or a `500`, depending on which strategy served it. This PR follows the route ruling on objectstack-ai#21042 (comment `5925613967`): the operand policies are hoisted into `@objectstack/core`, beside `AGGREGATE_ANSWER_KIND`, and both faces read them from there. - **`packages/core/src/utils/aggregate-answer.ts`** (`@objectstack/core`, `minor`). It takes: - `AGGREGATE_ACCUMULATION`, moved from `driver-sql` with its docblock (the docblock and the table are byte-identical to the base, apart from the `export` keyword); - `aggregandColumnClass({ type, multiple })`, the one column-class predicate (`'fractional'`, `'integral'`, `'boolean'`, or none); - `POSTGRES_BOOLEAN_AGGREGAND_CAST`, the objectstack-ai#11635 cast, as a `Record` over `AggregationFunction`: `sum` / `avg` / `min` / `max` are cast, the counts never are; - `doubleAccumulationOperand(operand, dialect)`, the double operand with the dialect as a parameter; - `aggregandOperandSql(func, columnClass, dialect, operand)`, the one composition both faces emit (the cast inside, the double operand around it). No `index.ts` line was added: the module is already exported. - **`packages/drivers/driver-sql/src/sql-driver.ts`** (`patch`), in the ruled regions only. The `AGGREGATE_ACCUMULATION` table becomes a pointer plus an import. `isFractionalNumericType` is deleted. The two registry fills fill `fractionalNumericFields` through the predicate. `accumulatesInDouble` / `doubleAccumulationOperand` are replaced by `aggregandColumnClassOf`, which maps the driver's registries onto the predicate's classes. In `aggregate()`, the private boolean-cast condition and the accumulation call become one `aggregandOperandSql(funcName, class, this.dialectName, '??')`. - **`packages/services/service-analytics/src/strategies/native-sql-strategy.ts`** (`patch`), in two places. - **`resolveMeasureSql`** wraps the column it hands `AGGREGATE_SQL` / `CONDITIONAL_AGGREGATE_SQL` in `aggregandOperandSql`. The column class comes from the declaration the host already relays (`declaredValueShape`), on the object the column lives on (`columnObjectOf`, the one hop resolver). The dialect comes from `sqlDialect`, which the strategy already reads. - **The `execute` shaping point** that PR objectstack-ai#21040 added now folds a `null` measure answer to `emptyGroupValueFor(measure.type)` (`@objectstack/spec`). It does this for every measure, measure-scoped ones included, before the number presenter, in `driver-sql`'s order. The dataset door's `DatasetExecutor` fill stays, and it is idempotent on a folded row. - The one line outside those two regions is the `generateSql` call site, which now passes `ctx` to `resolveMeasureSql`. - No native copy of any policy, and no runtime hook asks the driver: the rejected (C) route was not taken. `canHandle`, `buildFieldMeta`, the hop-object sites, the filter / text-match rendering, `analytics-service.ts` and `field-read-admission.ts` are untouched. ## The card's table, before and after Measured through `AnalyticsService.query` (the cube door, which `POST /api/v1/analytics/query` relays verbatim) and `AnalyticsService.queryDataset` (the dataset door), on `AnalyticsServicePlugin` over a real ObjectQL engine and `SqlDriver`. **Native** is the plugin's own composition (`NativeSQLStrategy` answered, with one raw statement and no engine aggregate). **ObjectQL** is the same composition narrowed to `engine.aggregate`. "Before" is the strategy file at the base `d34aa58a2a`; "after" is this branch at `61aab5013a`. Neither merge since then touches the aggregate code paths, and the pins below are green at `ef1f9d8484`. Fixture: - group `f`: `frac` (a `number` column) holds 0.1 and 0.2, and `flag` holds true and false; - group `i`: `stars` (a `rating` column) holds seven 1s and two 2s, and `flag` holds 7 trues and 2 falses; - group `n`: every aggregand is NULL in all three rows. The measure-scoped measures filter on `tag = x`, which only group `f` holds. **PostgreSQL 16.13** (a private local server; the ObjectQL column is the same before and after): | measure | group | door | native before | native after | ObjectQL | |:--|:--|:--|:--|:--|:--| | `sum(frac)` | f | cube, dataset | `0.3` | `0.30000000000000004` | `0.30000000000000004` | | `avg(frac)` | f | cube, dataset | `0.15` | `0.15000000000000002` | `0.15000000000000002` | | `avg(stars)`, an integer column | i | cube, dataset | `1.222222222222222` | `1.2222222222222223` | `1.2222222222222223` | | `sum(flag)` | i | cube, dataset | `500 DATABASE_ERROR` | `7` | `7` | | `avg(flag)` | i | cube, dataset | `500 DATABASE_ERROR` | `0.7777777777777778` | `0.7777777777777778` | | `min(flag)` / `max(flag)` | i | cube, dataset | `500 DATABASE_ERROR` | `0` / `1` | `0` / `1` | | `sum(frac)`, all-NULL group | n | cube | `null` | `0` | `0` | | `sum(frac)`, all-NULL group | n | dataset | `0` (executor fill) | `0` | `0` | | `sum(flag)`, all-NULL group | n | cube | `500 DATABASE_ERROR` | `0` | `0` | | `avg(frac)`, all-NULL group | n | cube, dataset | `null` | `null` | `null` | | measure-scoped `sum(frac)`, no admitted row | i | cube | `null` | `0` | `0` | | measure-scoped `sum(frac)`, no admitted row | i | dataset | `0` (executor fill) | `0` | `0` | | measure-scoped `avg(frac)`, no admitted row | i | cube | `null` | `null` | `null` | | `count` control | n | cube, dataset | `3` | `3` | `3` | | measure-scoped `count` control | i | cube, dataset | `0` | `0` | `0` | **SQLite** (better-sqlite3): accumulation and the boolean answers already agreed on every face (`0.30000000000000004`, `0.15000000000000002`, `1.2222222222222223`, `7`, `0.7777777777777778`, `0` / `1`). The fold is the policy that diverged there: | measure | group | door | native before | native after | ObjectQL | |:--|:--|:--|:--|:--|:--| | `sum(frac)` / `sum(stars)` / `sum(flag)`, all-NULL group | n | cube | `null` | `0` | `0` | | `sum(frac)` / `sum(stars)` / `sum(flag)`, all-NULL group | n | dataset | `0` (executor fill) | `0` | `0` | | measure-scoped `sum(frac)`, no admitted row | i, n | cube | `null` | `0` | `0` | | measure-scoped `sum(frac)`, no admitted row | i, n | dataset | `0` (executor fill) | `0` | `0` | After the fix, the native and ObjectQL faces **differ in 0 of 144 cells** (2 drivers × 2 doors × 12 measures × 3 groups). **MySQL is NOT MEASURED**: there is no MySQL server in this container. The MySQL operand text is pinned offline: by `core`'s `aggregate-answer.test.ts`, and by the `driver-sql` move proof for the driver's own statements. ## The move proof `driver-sql`'s aggregate statements were dumped at the base, before any consumer changed. The dump covered `SqlDriver.aggregate()` for every function (`count`, `count_distinct`, `sum`, `avg`, `min`, `max`, and `count(*)`), aliased and unaliased, over 23 columns: every fractional, integral and boolean type, the `float` / `integer` / `int` aliases, multi-valued and untyped columns, and text / date / lookup / formula. It ran on SQLite, PostgreSQL and MySQL, through both registration paths (`registerObjectMetadata` and `registerExternalObject`), offline (knex `toSQL()`). The policies were then hoisted, `driver-sql` was switched to the imports, and the same dump was run again: - base dump: 1668 entries, 0 errors, md5 `8eee668372a28a7568f3eb1cc5a2bc9b`; - after dump (at `bc8aa0cc2f`): 1668 entries, md5 `8eee668372a28a7568f3eb1cc5a2bc9b`. `cmp` printed nothing: the two dumps are **byte-identical**. `sql-driver.ts` and `aggregate-answer.ts` are unchanged between `bc8aa0cc2f` and `61aab5013a`. The committed move-proof pin, `packages/drivers/driver-sql/src/sql-driver-21042-aggregate-policy-move.test.ts`, holds the captured expressions for one column of each class, on each dialect and through each registration path. It passed at the base (`192fc0010b`: 54 / 54) and passes after (54 / 54). ## Pins (committed red first, then the fix) | file | at the pins commit (`192fc0010b`, base code) | after | |:--|:--|:--| | `core` `aggregate-answer.test.ts` | 16 red (the exports did not exist) | 22 / 22 | | `service-analytics` `native-sql-aggregate-policies.test.ts` (each measure on both faces at both doors, against the engine's arithmetic; SQLite and live PostgreSQL cells) | SQLite: the cube-door folds red. PostgreSQL: accumulation, boolean `500`, folds red | 49 / 49 | | `service-analytics` `cube-measure-field-type-door.test.ts`, **the lifted skip** | PostgreSQL native `max(boolean)` red (`500`) | 23 / 23 | | `rest` `analytics-dataset-aggregate-policies-door.test.ts` (the route, both strategies) | PostgreSQL: 7 red (accumulation and booleans). SQLite green (that door already folded) | 19 / 19 | | `driver-sql` move proof | 54 / 54 | 54 / 54 | **The lifted skip:** `it.skipIf(cell.id === 'pg' && face === 'native')` in `cube-measure-field-type-door.test.ts` (from PR objectstack-ai#21128) is gone. Its comment now says why the cell runs on every cell and face. The PostgreSQL native `max_flag` cell answers `1`. `native-sql-measure-number-presentation.test.ts` gets a comment-only edit: its header said the native statement does not carry objectstack-ai#20387's accumulation, and it now points at the new pin. ## Ablations There was one ablation per policy, each predicted in writing before it ran. Each mutation was planted through `scripts/ablation-replace.mjs`, which checks that the anchor hit and that the blob changed. Each mutation was confirmed in the built `dist/` (`ablation-dist-preflight.mjs`: marker present). Each restore ran by absolute path (`git checkout HEAD`), and the file's blob was proven equal to its `HEAD` blob with `git diff HEAD` empty. After each restore the package was rebuilt, and the marker was proven absent from `dist/` with the tree clean. Every prediction held exactly. | ablation | mutation | predicted red | observed red | |:--|:--|:--|:--| | A1 accumulation | `core` `accumulatesInDouble`'s dialect gate never admits PostgreSQL or MySQL | `core` 3; `driver-sql` move proof 24 (pg and mysql × both fills × the six numeric / boolean columns); `service-analytics` 10, PostgreSQL only (both doors × `sum` / `avg(frac)`, `avg(stars)`, measure-scoped `sum` / `avg`); `rest` 3, PostgreSQL only | the same 3 / 24 / 10 / 3; SQLite cells green; `avg(flag)` green as predicted | | A2 boolean cast | `core` `aggregandOperandSql` never casts | `core` 1; move proof 4 (pg × both fills × `boolean` / `toggle`); `service-analytics` 8, PostgreSQL only; the lifted cell, PostgreSQL native and ObjectQL, 2; `rest` 4, PostgreSQL only | the same 1 / 4 / 8 / 2 / 4 | | A3 fold | the native shaping point never folds | `service-analytics` 8, cube door only (SQLite and PostgreSQL × the three all-NULL `sum`s and the measure-scoped `sum`); everything else green, the `rest` dataset-door route included, because the executor fill folds there | the same 8; `rest` 19 / 19 green | A1 and A2 show one policy reaching both faces. Each one turned `driver-sql`'s own statements red. Under A2 the ObjectQL face's `max(boolean)` cell failed too, with the driver's refusal. Under A1 the ObjectQL face answered the same exact decimal as the native face for the plain measures: the failing assertion was the engine's number, while native and ObjectQL still agreed. A **reverse type check** also ran. Passing a dialect the new type rejects (`'oracle'`) to `aggregandOperandSql` turned `service-analytics`' typecheck red (`TS2345 ... not assignable to parameter of type 'AggregandSqlDialect'`), which shows the rebuilt `core` `.d.ts` was read. The file was restored byte-identical. ## Verification (at `ef1f9d8484`, after merging `origin/main` at `cb45469e67`, which carries PR objectstack-ai#21170 and PR objectstack-ai#21173) Everything below ran as one locked script, at `ef1f9d8484`, with each exit code captured before any pipe. The live PostgreSQL 16.13 server ran at `timezone = Asia/Shanghai`, and the `driver-sql` suite ran under `TZ=America/New_York`, which are its own non-vacuity preconditions. - **Refresh after the merge:** `pnpm turbo run build --filter='!@objectstack/docs' --concurrency=1` exit 0, and `pnpm --filter @objectstack/spec check:generated` exit 0. - **Gates:** `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 67 commands from the real diff (10 paths). All 67 exited 0. Reconciliation with `--ran` and the recorded exit codes printed: `Run reconciliation — 67 derived, 67 run, 0 NOT-MEASURED, 0 UNRUN.` - **Typecheck:** `pnpm --filter … typecheck` exit 0 for `@objectstack/core`, `@objectstack/driver-sql`, `@objectstack/service-analytics` and `@objectstack/rest`. - **Tests, every vitest project of every touched package** (`vitest run --maxWorkers=2`, with `OS_TEST_POSTGRES_URL` set so the live cells ran): | package / project | files | tests | |:--|:--|:--| | `core` local | 74 passed | 2120 passed | | `core` repo | 3 passed | 48 passed | | `driver-sql` | 216 passed, 3 skipped | 4300 passed, 96 skipped | | `service-analytics` | 161 passed | 3765 passed | | `rest` local | 256 passed | 5027 passed, 127 skipped | | `rest` repo | 5 passed | 179 passed, 1 skipped | - **The five pin files, run explicitly:** move proof 54 / 54, `core` 22 / 22, policies 49 / 49 (24 SQLite + 24 PostgreSQL + the oracle), field-type door 23 / 23, `rest` route 19 / 19 (9 SQLite + 9 PostgreSQL + the oracle). - **Lint, narrowed and proven:** `eslint --no-inline-config --format json` over the 9 changed code files answered 9 results, 0 errors and 0 warnings. The population is read from eslint's own config: `ESLint.isPathIgnored` answers `false` for each of the 9. The narrowing excludes nothing that could move, because `eslint.config.mjs` never enables type-aware linting (no `parserOptions.project`, no typed rules), so this diff cannot change the verdict on an untouched file. The repo-wide `pnpm lint` is CI's. - **The `driver-sql` live preconditions:** the first gate run used a private server at UTC, and the suite's four timezone non-vacuity cells failed by design (`… start it with timezone=Asia/Shanghai`). With the server at `Asia/Shanghai` and the process at `America/New_York` the suite is green, as listed above. - **`main` after the final merge:** `origin/main` moved 4 commits past `cb45469e67` before this PR opened (objectstack-ai#21149, objectstack-ai#21188, objectstack-ai#21195, objectstack-ai#21192). None of them touches `core`, `driver-sql`, `service-analytics` or the analytics `rest` tests. The one `packages/spec` file in the analytics area, `ui/dataset.zod.ts`, changes a comment only. They are not merged here; CI runs on the merge ref. ## Acceptance notes - **MySQL is NOT MEASURED** (no server in this container). The MySQL operand text is pinned offline in `core` and in the `driver-sql` move proof. - **The live PostgreSQL cells are not run in CI.** No CI step sets `OS_TEST_POSTGRES_URL` for `service-analytics` or `rest`. The cells above ran against a private PostgreSQL 16.13 started for this run and removed afterwards. In CI the SQLite cells run, and so do the offline `core` / move-proof pins. - **Phase 0's note on the dataset door, which is no divergence:** a measure-scoped `avg` is **absent** from a row its supplementary query reported no row for. That is `x_avg_frac` for groups `i` and `n`, on both strategies and before and after. It is not `null`. The new service pin holds this cell only to "both faces agree", not to a value. Relatedly, a dataset-door selection made only of measure-scoped measures reports only the groups their filter admits, so the pin asks each one beside the base count. - **Residual, as stated in the ruling:** a host that relays no field declarations (`declaredValueShape`), or names no SQL dialect, gets no column class or no policy. It keeps the native arithmetic it had, and a PostgreSQL boolean `sum` there still answers `500`. The plugin's own composition wires both. - **How `driver-sql` reads the class:** it reads its own registries rather than calling the predicate per column. `fractionalNumericFields` is filled by the predicate. `booleanFields` and `numericFields` are filled by the driver's coercion rules, whose populations equal the predicate's `'boolean'` and `'fractional'` ∪ `'integral'` classes. The move proof pins that equality per column class. Asking the predicate per column through the driver's `valueShapeFields` would retire `fractionalNumericFields`, but its declaration and shard-alias regions are outside the ruled surface, so this PR does not do it. - **The scan-order residual is unchanged.** On PostgreSQL and MySQL the double sums are added without compensation (`AGGREGATE_ACCUMULATION`'s docblock), so three or more fractions can still differ in the last place from SQLite and the rows path. The pins use two addends. - objectstack-ai#21129 (the presenter for a relationship-path `min` / `max`) is not addressed here. --- _Generated by [Claude Code](https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Fixes #20889
Clause-②: yes (widening)
What changed
On PostgreSQL, the analytics native-SQL path answered every measure its response declares
numberas a string:count: "2",sum: "500.000000000000000000000000000000". SQLite answered numbers.driver-sql's ownaggregate()has presented these answers since #20335, but through a private table and presenter thatservice-analyticscould not reach. The seat's route ruling (5922780640) moves that one rule into@objectstack/core, and both faces now call it.@objectstack/core(newsrc/utils/aggregate-answer.ts, one export line inindex.tsbesidecompensatedSum):AGGREGATE_ANSWER_KIND, with its precision-policy docblock, moved fromsql-driver.ts. The text is byte-identical apart from theexportkeyword; the proof is below.presentAsNumber(value), the body ofpresentReadValue's'number'arm as a named function. A stringNumber()reads as a number becomes that number. Any other value is returned as given.@objectstack/driver-sql,sql-driver.ts, the two ruled regions only:AGGREGATE_ANSWER_KINDdefinition becomes an import, and a pointer comment marks where it was;presentReadValue's'number'arm returnspresentAsNumber(value).The import is a separate statement beside the
AggregationFunctionimport (inserted after base line 23; line 27 at HEAD). It does NOT join the@objectstack/coreimport block at base lines 94-100, which PR fix(plugin-security,driver-sql,driver-turso): lower type-blind at the RLS seam without a guard, then delete the F1/F2 whole-day and NOT-rewrite copies (#5930 step 4, group 2) #20988 rewrites. That hunk starts at base line 91, 67 lines below the insertion. No behaviour change: the full driver-sql suite is green on SQLite and PostgreSQL (below).@objectstack/service-analytics,NativeSQLStrategy.executeonly. After the raw statement runs, each measure column is presented by the measure's DECLARED aggregate function (lookupMember(cube, m, 'measure').type), never by whether a value looks numeric:count,count_distinct,sumandavg(the table's'number'kind) always takepresentAsNumber;min/max(the'column'kind) take it only whendeclaredFieldType(object, column)is inNUMERIC_VALUE_TYPES. That is the ruledriver-sql'sreadPresentationKindapplies to a declared numeric column. A host that cannot answer, or a relationship-path column, leaves the value as given;number/string/boolean) are left as they are (ruling Q3).One point covers both doors: the cube read (
POST /api/v1/analytics/query) and the dataset door (POST /api/v1/analytics/dataset/query, throughDatasetExecutor)..changeset/20889-analytics-native-measure-number.md:@objectstack/coreminor,@objectstack/driver-sqland@objectstack/service-analyticspatch,Clause-②: yes (widening)for the two new root exports.Per-measure answers, before and after
Fixture:
textcategory;ratingstars (an integer column);numberamount and frac (the exact-decimal column);currencyprice. Groupaholds 2 rows.7fa67dada3, throughAnalyticsService.query(what the route relays verbatim). The red pins at719b98795eread the same strings.e827c237d1through the realdispatcher-pluginmount ofPOST /api/v1/analytics/query, by a scratch probe that was deleted after the run.count*22"2"2count_distincttext)22"2"2sumrating, integer)77"7"7avgrating, integer)3.53.5"3.5000000000000000"3.5sumnumber, decimal)500500"500.000000000000000000000000000000"500avgnumber, decimal)250250"250.000000000000000000000000000000"250minnumber)100100"100.000000000000000000000000000000"100maxcurrency)20.520.5"20.500000000000000000000000000000"20.5maxtext), the control"y""y"(left as the client gave it)sum900719925474099390071992547409929007199254740992"9007199254740993.000000000000000000000000000000"9007199254740992row_count(count)*22"2"2filtered_count, a group the statement answered*11"1"1numberSUM(amount) / 2250250"250.000000000000000000000000000000"fields[]declarednumberfor every measure on both dialects, before and after. On the ObjectQL face of the same route, every answer was already a number on both dialects, and it is unchanged.Move proof
move-proof.py(scratch) reads the block from BASE2f2fa11d75'ssql-driver.tsand from HEAD'saggregate-answer.ts:The table and its docblock are byte-identical apart from
export. The presenter's eight body lines, comment included, are identical once de-indented. In core, a new file header says why the rule lives there. The moved docblock still speaks ofSqlDriver.aggregateandformatOutputin the driver's voice, and the header says so.Pins (red first), the ablation, and the local PostgreSQL run
719b98795e(pins only, no fix), on a private PostgreSQL 16.13 (initdb, 127.0.0.1, trust):@objectstack/coreaggregate-answer.test.ts: 1 file failed,Cannot find module './aggregate-answer'.service-analyticsnative-sql-measure-number-presentation.test.ts: 5 failed, all PostgreSQL; 5 passed, all SQLite. First failure:a row_count is a number, never "2": expected 'string' to be 'number'.rest, the newanalytics-dataset-measure-number-door.test.tsand the tightenedanalytics-dataset-json-dimension-door.test.ts: 2 failed, both PostgreSQL; 6 passed. Failures:a row_count is a number, never "2"andexpected [ [ 'x', '2' ], [ 'y', '1' ] ] to deeply equal [ [ 'x', 2 ], [ 'y', 1 ] ].e827c237d1: core 5/5; service-analytics 10/10 (SQLite and PostgreSQL); rest 8/8 (SQLite and PostgreSQL).scripts/ablation-replace.mjs(WRAP mode, with its restore trap) on the absolute path. The guardif (numberMeasures.length > 0 && Array.isArray(rows))gained&& String(numberMeasures) === 'ABLATION-20889', so it never holds. Anchor count went 1 to 0, blob3bce7bb073to8f36f62dfe.service-analyticswas rebuilt, andablation-dist-preflight.mjsfound the marker in 2 built files.3bce7bb073),git diff HEADis 0 bytes, and porcelain is 0 lines. Then a rebuild,ablation-dist-preflight.mjs --absent(marker absent from all 6 built files, tree clean), and a re-run: all green.driver-sql's full suite ate827c237d1, SQLite and live PostgreSQL,TZ=America/New_York, with the server atAsia/Shanghaias CI runs it: 210 files passed, 3 skipped; 4061 tests passed, 95 skipped.sql-driver-20335-aggregate-numeric-presentation.test.ts15 tests (1 skipped, the MySQL cell), andsql-driver-20387-aggregate-double-accumulation.test.ts11 (1 skipped). A first run on a UTC server failed only the four suites whose precondition asserts a non-UTC server.OS_TEST_POSTGRES_URLforservice-analyticsorrest, so their PostgreSQL cells are named skips there (skipped: set OS_TEST_POSTGRES_URL to run this cell). The core pin (string to number) and every SQLite cell run in CI. The private cluster was stopped and its data directory removed after the runs.Gates
All at HEAD
e827c237d1, as ONE sequential script under the shared verify lock, each exit code captured before any pipe.node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstacknames 67.--ranreconciles:67 derived famil(ies) accounted for — 65 run, 2 NOT-MEASURED, 0 unrun. All 65 exit 0.check:dual-build-cjs-loadsandcheck:type-check-debt. Reason: both exit 3,PREREQUISITE NOT MET. They read the built output of every package, and this run built only the dependency closure it tests. CI'slint.ymlbuilds the full closure before them.check-changeset-fixed.mjs,check:authz-resolver,check:error-code-casing,check:filter-alias-parity. All exit 0.@objectstack/coretest: 62 files, 1803 tests passed. Typecheck exit 0, and itscheck:test-typecheckcovers the new test.@objectstack/driver-sql: typecheck exit 0; the full suite as above.@objectstack/service-analyticsfull suite withOS_TEST_POSTGRES_URLset: 151 files, 3468 tests passed. Typecheck exit 0;tsc --listFilesincludes the new pin.@objectstack/rest: the two pin files, SQLite and PostgreSQL, 8/8. Typecheck exit 0, withcheck:test-typecheckovertsconfig.test.json.pnpm lintis CI's):eslint --no-inline-config --format jsonover the 8 touched.tsfiles. From the JSON: 8 files, 0 errors, 0 warnings, 0 ignored.eslint.config.mjs'sfiles: ['**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}']minusNEVER_LINTED, which contains all 8.eslint.config.mjs:326-328, noparserOptions.project), and this diff does not touch the config, so no untouched file's verdict can move.Acceptance notes
DECIMALstring would be presented the same way. Separately, a read-only inference, also unmeasured:plugin.ts:500returnsengine.execute's result as the rows whenever it is an array, andknex.rawon mysql2 resolves to a[rows, fields]pair, so on MySQL the native path may read that pair as its rows.formatOutput's inline copy of the presenter (sql-driver.ts, the numeric pass at about:20303) is not folded ontopresentAsNumberhere, per the ruling. It is the same rule written a second time, againstpresentReadValue's "never a re-derivation" docblock. Fordomain:engine.sql-driver-20335-aggregate-numeric-presentation.test.ts:31andsql-driver-20387-aggregate-double-accumulation.test.ts:7sayAGGREGATE_ANSWER_KINDlives insql-driver.ts. These are driver-sql test docblocks outside this card's two regions. Fordomain:engine, or whoever next touches those files.type: 'number' | 'string' | 'boolean') are untouched:fields[]still declaresnumberwhile PostgreSQL answers"250.000…", text or a JS boolean. [Decision] analytics field gate (#20917): an authored cube member whosesqlis an expression — keep the stand-down, judge its identifiers, refuse it, or retire expressions #20943 (PR feat(spec)!: an analytics cube member's sql is a column reference, and the showcase done rate moves to its dataset (#20943) #20998) retires SQL expressions in cube membersql.min/maxover a relationship-path column (account.revenue) stay as the client gave them.declaredFieldTypelooks a column up on the base object only, the same "cannot answer, do not block" tiermeasureResultTypetakes.0.1 + 0.2answers0.3on PostgreSQL native, against0.30000000000000004elsewhere). It was measured once through the route, as the ruling asked, and is reported to the seat. No pin enshrines either side of it.analytics.query(call sites (envelope-caller-census.test.ts) is unchanged: the new pins drive the route or call the service under another name, so no ledger row or declared input moved.Generated by Claude Code