Repository navigation
Commit 00a92e1
Fixes #20807
Clause-②: no (narrowing)
## What this changes
A dimension that GROUPS an analytics query, and whose column is a
declared **structured-JSON** field (`json`, `composite`, `repeater`,
`record`, `location`, `address`, `vector`), is now refused
`INVALID_FIELD` / 400 at the analytics door, naming the member the
caller wrote, before either strategy builds anything. "Groups" means a
`dimensions` entry, or a `timeDimensions` entry that carries a
`granularity`. It holds on `POST /api/v1/analytics/query`, its dry run
`POST /api/v1/analytics/sql`, and `POST
/api/v1/analytics/dataset/query`, on every driver.
The words, as `POST /api/v1/analytics/query` returns them for a cube
dimension:
```text
Dimension 'meta' on cube 'json_dim_ledger' groups by field 'meta', which object 'analytics_json_dim_ledger' declares as json — a structured-JSON value, which analytics does not group by. The query was NOT run. Group by a field that stores one scalar value: store the part you group on in a field of its own and group by that field. A JSON document is no group key the SQL dialects share: one grouped each serialized document apart, another refused the statement.
```
For a dataset dimension over an included relationship, the same words
say `groups by field 'account.hq', whose column 'hq' the joined object
'X' declares as json`. The thrown error is `invalidMemberError`'s
envelope (`code: 'INVALID_FIELD'`, `status: 400`, `member`, `param`,
`cube`) plus `field` and `object`.
**Landing site: `packages/services/service-analytics` only.**
- `src/structured-json-dimension-door.ts` (new):
`assertNoStructuredJsonDimension`. The class is
`@objectstack/spec/data`'s `STRUCTURED_JSON_TYPES`, called, never
re-listed. It is the same predicate the engine's `groupBy` door
(`packages/objectql/src/group-by-structured-json-door.ts`) reads, so
there is one "is this field structured JSON" test for both doors. That
file is read, not edited.
- `src/analytics-service.ts`: one private method,
`assertDimensionsGroupScalarColumns`, called in `ensureCube` right after
the dimension source-field gate on each of its three paths (inferred
cube, augmented cube, declared cube). It supplies the two answers only
the service has: the dimension `sql` a member resolves to
(`declaredMemberEntry`, the strategies' own `'dimension'` lookup, and
the member itself when the cube declares none), and the column's
declared type (`sourceFieldMeta`).
- The column is read the way `NativeSQLStrategy` compiles it. A bare
identifier is a column of the cube's object. A dotted identifier path
(`account.hq`) is its last segment, on the object the cube's DECLARED
join for that path names, which is the alias the dataset compiler
registers and the strategy joins. A path with no declared join is a
synthetic traversal and is not judged.
## Before, measured on `origin/main` `793fb839`
Through the real `dispatcher-plugin` route over the service
`AnalyticsServicePlugin` composes on a real `ObjectQL` engine, with both
of its auto-bridges live (`executeRawSql` to `engine.execute`,
`executeAggregate` to `engine.aggregate`). Three rows: `title` x, x, y,
and a different `meta` document per row. Drivers: `SqlDriver` on SQLite
(better-sqlite3), and on a private PostgreSQL 16.13 started for this
run.
| request | SQLite | PostgreSQL 16 |
|:--|:--|:--|
| `/analytics/query`, `dimensions: ['title']` (text, the control) | 200,
`x` 2 · `y` 1 | same |
| `/analytics/query`, cube dimension `meta` (json) | 200, one group per
serialized document (3 groups, `count` 1 each) | 500 `DATABASE_ERROR`
(42883, "could not identify an equality operator for type json") |
| `/analytics/query`, dataset dimension `meta_doc` (over `meta`) | 200,
3 groups | 500 `DATABASE_ERROR` |
| `/analytics/sql`, cube dimension `meta` | 200, the statement `SELECT
meta AS "meta", COUNT(*) AS "count" FROM ... GROUP BY meta` | same |
| `/analytics/dataset/query`, inline dataset dimension `meta_doc` | 200,
3 groups | 500 `DATABASE_ERROR` |
Counted at the engine for the json dimension: raw SQL 1,
`engine.aggregate` 0, on both dialects, so `NativeSQLStrategy` answered
and the engine's `groupBy` door never saw the query.
A dataset dimension over a JOINED object's json field (`include:
['account']`, `field: 'account.hq'`) answered the same two ways: SQLite
200 with one group per document (2 groups), PostgreSQL 500 (42883). This
was measured through the service on the first fix commit `b1befe2a6`,
which judged only bare columns, and through `/analytics/dataset/query`
under the ablation below. The second fix commit closes it.
## After, on this branch (`075a46340`)
Every refused row above answers `400 INVALID_FIELD` naming the member
(`meta`, `meta_doc`, `acct_hq`), with zero raw-SQL statements and zero
engine aggregates for the object. The `title`, `title_dim` and
`acct_name` controls answer `x` 2 · `y` 1 (and `A` 2 · `B` 1) from the
native strategy, unchanged.
## Mechanism assumptions (zone 2): which held
- **B1: held.** It was reproduced on `793fb839` as the red pins. See the
table above and the raw-SQL / aggregate counts.
- **B2: held.** A dataset dimension compiles to a cube dimension whose
key is the dataset dimension's name (`dataset-compiler.ts`:
`dimensions[d.name] = { sql: d.field }`). `DatasetExecutor` passes
`selection.dimensions` through as the cube query's `dimensions`, so the
member reaching `ensureCube` IS the name the selection wrote. The
compiler checks only the relationship path (`assertDeclared`).
- **B3: held.** The one predicate class is `STRUCTURED_JSON_TYPES`, read
at the engine door. The engine door's exported function takes `groupBy`
entries and names `groupBy[i]`, and `@objectstack/objectql` is only a
devDependency of this package, so the constant is what is called. No
edit to `packages/objectql` or `packages/spec`.
- **B4: held, measured.** `cube-registry.ts` stores cubes and knows no
field type. `dataset-compiler.ts` knows a dataset dimension's name and
the declared type, but it compiles the whole dataset, whether or not a
dimension is selected (a refusal there would refuse every selection of
the dataset), and it does not see cube queries. `analytics-service.ts`'s
`ensureCube` is the first step every door passes through that knows both
the member as written and the column's declared type, ahead of strategy
selection. The GUARD and the per-face unit cases show one answer on both
strategies. Before this, the ObjectQL face reached the engine door but
was refused under `groupBy[0]`.
- **B5: no CI harness runs this route live.** The `Temporal Conformance
(live PG + MySQL)` job sets `OS_TEST_POSTGRES_URL` for three steps only:
`driver-sql`, `metadata-protocol`'s `live-postgres` files, and
`runtime`'s cascade-delete matrix. It runs `service-analytics` without a
URL, and no step runs `@objectstack/rest`'s or `@objectstack/runtime`'s
analytics pins. The PostgreSQL cells therefore sit beside each HTTP pin
as a named skip without the URL, like
`packages/rest/src/data-group-by-json-door.test.ts`. They are
red-capable and un-run in CI. The local PostgreSQL 16.13 runs are quoted
below.
- **B6: measured `no (narrowing)`.** No new key reaches a published
payload: the error envelope's members are the ones `invalidMemberError`
and the dimension source-field gate already attach. The accept set
narrows: SQLite answered 200, and it now refuses. Changeset `minor`,
BREAKING banner, `Clause-②: no (narrowing)`. The ADR-0087 category
measured is `not-required (no-migration-prescription)`: the package
publishes, no ADR-0087 id covers a grouping target, and nothing
authorable, exported or stored moves.
- **B7: reproduced.** See the acceptance notes. Not fixed here.
**Where the `/api/v1/analytics/query` pin lives, measured.** That route
is served by `@objectstack/runtime`'s `dispatcher-plugin` (through
`domains/analytics.ts`), not by `@objectstack/rest`. The REST package
serves `/analytics/dataset/query`, and `runtime` depends on `rest`, so a
REST-package test cannot reach the runtime route. The cube-face pin is
therefore a new file beside the repo's other `/analytics/query` HTTP
pins (`packages/runtime/src/analytics-*.test.ts`). The dataset-door pin
is a new file in `packages/rest/src/`. Both are new pin files only.
## Tests
All at `075a46340`, the final commit.
- New
`packages/services/service-analytics/src/__tests__/dimension-structured-json-door.test.ts`:
**13 passed**. It covers every structured-JSON type on both strategy
faces (native and ObjectQL), with the envelope (`code`, `status`,
`member`, `param`, `field`, `object`) and zero raw SQL and zero
aggregates. It also covers a cube key over another column, the
cube-qualified spelling, a bucketed time dimension on both faces, a
declared default granularity, a dataset dimension, a dataset dimension
over an included relationship (judged on the joined object), an ad-hoc
inferred cube, and the dry run. Controls: a text dimension is served, an
unknown field keeps the existence gate's answer first, a synthetic
dotted traversal is not judged, and a host without `sourceFieldMeta`
stands down. GUARD: over every `FieldType`, the refused types are
exactly `STRUCTURED_JSON_TYPES`.
- New `packages/runtime/src/analytics-json-dimension-door.test.ts`
(`POST /api/v1/analytics/query` and `/sql`): **8 passed**, SQLite 4 and
live PostgreSQL 16.13 4.
- New `packages/rest/src/analytics-dataset-json-dimension-door.test.ts`
(`POST /api/v1/analytics/dataset/query`): **6 passed**, SQLite 3 and
live PostgreSQL 16.13 3.
- Red first: at `46ee85eda` (the pins on the base code) the unit file
was 8 failed and 3 passed, the runtime file 6 failed and 2 passed, and
the REST file 2 failed and 2 passed. The failures were the refusals; the
controls were green.
- `pnpm --filter @objectstack/service-analytics exec vitest run`: **145
files / 3325 passed**.
- `typecheck`, exit 0: `@objectstack/service-analytics` (`tsc
--listFilesOnly` includes the new door and the new test);
`@objectstack/rest` (the new test is in its test program, and
`check:test-typecheck` holds at 0 files / 0 errors);
`@objectstack/runtime` (`check:test-typecheck` holds at 27 files / 190
errors / 68 signatures, unchanged).
- Downstream consumers are declared to CI. No export, published type or
`exports` entry of `@objectstack/service-analytics` changes, so only
behaviour moves. A census of `examples/` at `793fb839` found no producer
grouping by a structured-JSON field: 5 files declare a cube or dataset,
18 distinct dimension sources, none of them structured JSON.
**Reverse verification (ablation), from the committed fix at
`075a46340`.** It ran through `scripts/ablation-replace.mjs` in WRAP
mode, trap-restored, under the verify lock. The door's own verdict line
gained an always-true `continue` guard keyed on the marker
`__ablated_20807__`. On disk: anchor 1 to 0, blob `47ce2b2acc11` to
`a286ca8a75ae`. `service-analytics` was rebuilt, and
`ablation-dist-preflight` found the marker in 2 built files
(`dist/index.js`, `dist/index.cjs`).
- Predicted direction: red. Observed: red.
- Unit: **9 failed / 4 passed**. Every refusal case and the GUARD
failed; the four controls stayed green.
- `/analytics/query` pin: **6 failed / 2 passed**. On SQLite the cube
and dataset dimensions answered 200 with 3 groups and the dry run served
the statement. On PostgreSQL they answered 500 `DATABASE_ERROR`. The
controls stayed green.
- `/analytics/dataset/query` pin: **4 failed / 2 passed**. `meta_doc`
and `acct_hq` answered SQLite 200 and PostgreSQL 500. The controls
stayed green.
- Restore leg: blob equals HEAD (`47ce2b2acc11`), `git diff HEAD` is
empty, and the whole-tree `git status --porcelain` is empty. After a
rebuild, `--absent` found the marker absent from all 6 built files and
the tree clean. The re-run was green: 13, 8 and 6 passed.
- The same ablation on the first fix commit `b1befe2a6`, before the
joined-column extension, read 8/3, 6/2 and 2/2, the same direction.
## Gates
`node scripts/pm/dispatch-gates.mjs --commands` (no paths) at
`075a46340` derived **62** commands over the 6 changed paths. That is
the same list the PM derived at `793fb839`. The four roster families the
order names were run too: `node scripts/check-changeset-fixed.mjs`,
`pnpm check:authz-resolver`, `pnpm check:error-code-casing` and `pnpm
check:filter-alias-parity`. `--ran` reconciles: **62 derived, 62 run, 0
NOT-MEASURED, 0 UNRUN**. All 66 commands exit 0.
`check:dual-build-cjs-loads` and `check:type-check-debt` first exited 3
(`PREREQUISITE NOT MET`) and were re-run to exit 0 after a full `turbo
run build --filter='./packages/*' --filter='./packages/*/*'`. Among
them:
- `check:adr-0087-registration --base origin/main`:
`[BREAKING+bang+clause-②-narrowing] not-required
(no-migration-prescription)` accepted.
- `check:changeset-no-major`, `check:empty-changeset`,
`check:doc-authoring`, `check:nul-bytes`, `check:issue-citations`.
- `check:cross-package-test-inputs`, `check:test-source-alias`,
`check:driver-memory-census`, `check:rest-log-spy-declared`,
`check:engine-double-contract`, `check:query-options-erasure`,
`check:type-check-coverage`, `check:type-check-debt` (re-measure: none
above its record).
Note: `dispatch-gates` flagged the tree as 6 commits behind
`origin/main` `2d5fe76f4`. None of those commits touches this diff's
paths, `service-analytics`, the analytics routes, the engine door or
`STRUCTURED_JSON_TYPES`, so `main` was not merged (the order merges only
on a surface hit). The merge ref CI builds covers the joint tree.
Lint, narrowed and proven: `pnpm exec eslint --no-inline-config --format
json` over the 5 changed `.ts` files at `075a46340` found **5 files, 0
errors, 0 warnings**. Three facts make this narrowing a measurement:
- The checked population comes from eslint's own config: `isPathIgnored`
answers `false` for all 5.
- The file count comes from the JSON output: 5 results.
- Untouched files cannot change verdict: `parserOptions.project` and
`projectService` are `null` for every file, so type-aware linting is not
enabled.
## Changeset
`.changeset/20807-analytics-json-dimension-refused.md`:
`@objectstack/service-analytics` `minor`, BREAKING banner, `Clause-②: no
(narrowing)`, and exactly one ADR-0087 marker, `not-required
(no-migration-prescription)`, in the form the engine door's changeset
uses. It states the refused shape, names the refusal's code, and says
who is affected and what is unchanged. No export or published type
changes.
## Acceptance notes
- **Reported to the PM, not filed here:**
- **The PostgreSQL count is a string on the native path (class a, B7).**
Through `POST /api/v1/analytics/query` on PostgreSQL 16, the `text`
control's rows came back `{"title":"x","count":"2"}` while `fields` said
`{"name":"count","type":"number"}`. SQLite returned `2`. The registered
dataset's `row_count` behaved the same. This is the result-typing class
triage directed out of this card.
- **No authoring leg for this refusal (class c).** `os validate` (the
built CLI at `075a46340`) passes a stack whose dataset declares a
dimension over a `json` field: exit 0, "Validation passed". The runtime
now refuses that dimension at query time. The same stack with a measure
`avg` over that field is refused by
`measure-aggregate-field-type-refused`, so the command does judge
dataset members against declared types.
- **Boundaries of this door, not measured as defects:**
- A `timeDimensions` entry with no `granularity` over a json field. It
bounds a range and groups nothing, so it is a filter's question, not
this door's.
- A dotted dimension the cube declares no join for (a synthetic
traversal).
- A host that wires no `sourceFieldMeta`.
- The draft preview. `queryDataset` with `previewDrafts` over a pending
seed evaluates in memory (`evaluateAnalyticsQueryOverRows`) and does not
pass `ensureCube`.
- `MemoryAnalyticsService` (`driver-memory`'s cube face) is #20859's
position and is not touched.
- **Order: existence first, then type.** A member naming a column the
object does not have keeps the existing `INVALID_FIELD` ("does not
have") answer, because the dimension source-field gate runs first.
- #20810 is not addressed here: it lowers filters, and this card refuses
dimensions.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent 26437ae commit 00a92e1
6 files changed
Lines changed: 1030 additions & 0 deletions
File tree
- .changeset
- packages
- rest/src
- runtime/src
- services/service-analytics/src
- __tests__
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
Lines changed: 260 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
0 commit comments