Repository navigation
feat(objectql): serve the nested-relation filter in where — lowered at the engine seam, the related object read as the caller, a loud cap, drivers untouched (#20802) - #20872
Conversation
Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…als name the nested route Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…e REST bound Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…esets, ADR anchor Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…stage Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…lation-filter-lowering
…'s words Claude-Session: https://claude.ai/code/session_01DEvba2nBuD4tWzfq8r8NFY Co-authored-by: Claude <noreply@anthropic.com>
…lation-filter-lowering
📓 Docs Drift CheckThis PR changes 3 package(s): 25 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: ⛔ 6 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails. What this run could not see
Coarse fallback — 139 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 6dfbe15fcd231284b26b0f0ca86f981072ef980c && git checkout 6dfbe15fcd231284b26b0f0ca86f981072ef980c
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 72f8c3820154c69cdf9faa6f887dc544e4c4b823 56da9b6d50340f2cbc1674236f8957cab1e5c2ff && git checkout -B drift-repro 72f8c3820154c69cdf9faa6f887dc544e4c4b823 && git merge --no-ff 56da9b6d50340f2cbc1674236f8957cab1e5c2ff
node scripts/docs-audit/affected-docs.mjs --json 72f8c3820154c69cdf9faa6f887dc544e4c4b823
|
Contract reviewServed-tier: At-tier review of PR #20872, the engine half of card #20802 (ruling 5907789183, letter A), rendered at 2026-09-30T14:19Z. Read only: the PR body, its 20-file list and the net diff against Check-runs on this head, as read at the stamp above: 32 runs. 17 ① Derived judgmentsEach accept-set and public-surface change the diff implies, named right or wrong against the ruling's execution parameters.
② Semver level
③ Boundary flagsEvery dev flag and every
Implemented-by: VERDICT: PASS Generated by Claude Code |
…nested-relation form (objectstack-ai#20906) Fixes objectstack-ai#20876 Clause-②: no ## What changed `content/docs/protocol/objectql/query-syntax.mdx`, section "Filtering Across Relationships" only. The old callout said relation traversal in `where` is "not supported" and blamed `SqlDriver.applyFilters()` for compiling a nested object and emitting a dotted key to Knex. Both were false: the headline since PR objectstack-ai#20872 (`ca5408c62`), the mechanism sentences since the engine refuses the nested form before any driver. The section now states the served form, per triage 5914865117: - `{ relation: { field: value } }` in `where`, one level, forward only, evaluated as the caller; a multi-valued relation matches any member; refused past 1000 ids; an unreadable related field answers `403`. - Verbs and REST doors that serve it; second level and reverse direction refused; dotted path still `INVALID_FIELD` / 400; an aggregation's `filter` and `having` refuse the form. - The two-step example is kept, now framed as the route past the cap and for the reverse direction (a reverse snippet is added). - No driver-mechanism prose. ## Code anchors (all at `origin/main` 33b6e8b) - Lowering seam: `packages/objectql/src/engine.ts:11268` (`resolveRelateThenLowerWhere`, calls `lowerRelationConditions` at :11276); the related read is the engine's own `find` with the caller's context, `:11322`. - `$in` vs `$contains`: `packages/objectql/src/relation-filter-lowering.ts:213-218` (`lowerRelationSite`): `$in` for single-valued, an `$or` of one `$contains` per id for `multiple: true`. - Cap: `RELATION_FILTER_ID_CAP = 1000` at `relation-filter-lowering.ts:84`; read asks for cap+1 (`engine.ts:11325`), refuses at `:11330` via `relationFilterCapError` (`relation-filter-lowering.ts:297`) as `INVALID_FILTER` / 400 (`filter-comparand-shape.ts:78-85`). - 403: the related read goes through the security layer's `assertReadableQueryFields` (`packages/plugins/plugin-security/src/predicate-guard.ts:106`, called at `security-plugin.ts:3652`), `PERMISSION_DENIED` / 403. Pinned: `engine-nested-relation-lowering.test.ts:227`, `packages/rest/src/data-nested-relation-permission.test.ts`. - One level: `admitRelationCondition`, `relation-filter-lowering.ts:162`; dotted key inside the condition `:189`, relation key inside `:197` (`second-level`), both `INVALID_FILTER` / 400. Pinned: `engine-nested-relation-lowering.test.ts:284`, `packages/rest/src/data-nested-object-door.test.ts` (REFUSED table). - Reverse direction: not served by the lowering (`relation-filter-lowering.ts:16-17`); over REST the parent has no such field, so `assertFilterFieldsExist` answers `INVALID_FIELD` / 400 (`packages/metadata-protocol/src/protocol.ts:10057`, called at `:11510`). The direct `engine.find` has no field-name door (`query-syntax.mdx` section 9, "Unknown Fields Are Tolerated"), so the page states the reverse refusal for the REST doors only. - Dotted path: `classifyDottedFilterHead` (`packages/spec/src/data/filter-dotted-head.ts:119`); engine door `assertFilterIsMaterializable` (`filter-comparand-shape.ts:204`, code set at `:270`); REST ingress `protocol.ts:10114`. - Verbs: `find` `engine.ts:11662`, `findOne` `:11935`, `update` `:13558`, `delete` `:16227`, `count` `:16760`, `aggregate` `:17138`; pinned for all six plus `judgeFilter` at `engine-nested-relation-lowering.test.ts:190`. REST: `GET /data/:object` and `POST /data/:object/query` (`packages/rest/src/rest-server.ts:8635`, `:8818`) both call `findData` (`protocol.ts:11183`); the POST query door is the one the REST test drives. - `aggregations[i].filter` and `having` refuse the form: `no-operator-object-door.ts:307`; pinned `engine-nested-relation-lowering.test.ts:329`, REST `data-nested-object-door.test.ts:256`. ## Shared wording with objectstack-ai#20888 The skill half is PR objectstack-ai#20902 (open, not merged). Two passages are quoted verbatim from its `skills/objectstack-query/rules/filters.md`: the served-form sentence ("A condition on a related record's fields beneath a relation field ... any member when `multiple: true`).") and the "Limits — one level: ..." sentence. The code sentence is identical; if objectstack-ai#20902 changes those sentences before landing, this page must be re-quoted. ## Census of hand-written `content/docs/**` Searched: "Relation traversal", "not supported" near where/relation, `applyFilters`, dotted `'account.` examples, "two queries" / "$in its ids" / "nested form". - Fixed: `protocol/objectql/query-syntax.mdx` (the section above), the only hit. - Already correct, no change: `kernel/contracts/data-engine.mdx:187-215` states the served form, the 1000 cap, one level, `403`, aggregation `filter`/`having` and the dotted refusal (landed with objectstack-ai#20872). - Not hits: `permissions/*.mdx` dotted `'account.annual_revenue'` keys are field-permission keys, not filters; `protocol/objectql/types.mdx:1035` `'metadata.color'` is the deliberate structured/JSON carve-out; `query-syntax.mdx` :1135/:1167 and `data-modeling/queries.mdx:442,519`, `schema-design.mdx:111`, `api/data-api.mdx:177` concern the search axis and `expand`/`fields` paths, not `where`. - Generated reference pages (`content/docs/references/**`) not touched. Also fixed (contract review, second commit `a2a66881e2`): the `query-syntax.mdx` line 4 frontmatter `description` listed "joins", a tombstoned key (`packages/spec/src/data/query.zod.ts:562`; the page says so at its Joins section); it now reads "expand", which the page's section 4 (Relationships (Expand)) covers. ## Gates `node scripts/pm/dispatch-gates.mjs --commands` derived 40 commands at head `a2a66881e2` (re-run after the second commit; the first pass was at `bd718b653a`); 39 ran green (exit 0) after `pnpm install` and a lint-closure build. `--ran` reconciliation: 39 of 40 run, 1 UNRUN: `pnpm --filter @objectstack/spec run check:skill-examples` exited 3 (prerequisite: the client-SDK surface has no built output). NOT MEASURED: it type-checks marked TypeScript examples in skills and docs, and this diff adds no marked block. CI owns it. No changeset (docs only, nothing published). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VDtqoecgES7ScQYGbFVDRv --------- Co-authored-by: Claude <noreply@anthropic.com>
…ter in where, its limits, and the two-step route past them (objectstack-ai#20902) Fixes objectstack-ai#20888 Clause-②: no ## What this changes Since PR objectstack-ai#20872 (`ca5408c62`, the engine half of ruling 5907789183 on objectstack-ai#20802) the engine serves `{ relation: { field: value } }` in `where`, lowered at its filter seam (`packages/objectql/src/relation-filter-lowering.ts`, `engine.ts` `lowerRelationConditions`): the related object is read through the engine's own `find` with `fields: ['id']` and `limit: RELATION_FILTER_ID_CAP + 1` in the caller's execution context, and the ids become `{ relation: { $in: ids } }` on a single-valued relation or an `$or` of one `$contains` per id on a `multiple: true` one. The published skill `skills/objectstack-query`, rewritten by PR objectstack-ai#20811 (`1d202453`) the same day and before that landing, still told AI authors the form is refused (`INVALID_FILTER` / 400) and that the two-step `$in` route is the only way. This PR states the served form, its limits and the two-step route past them, at every site PR objectstack-ai#20811 rewrote — and nothing about analytics, which objectstack-ai#20887 carries. Every limit is stated from the code and its pins, read at `origin/main` `00a92e18d`: | Limit | Where it lives | Code / status | |:--|:--|:--| | One level: every key a field the related object declares; a relation condition or a dotted key inside is refused before any read | `admitRelationCondition` (`second-level`, `dotted-key`, `undeclared-key`, `empty`, `unregistered-target`), pinned in `engine-nested-relation-lowering.test.ts` | `INVALID_FILTER` / 400, "one level only" | | Forward only: the condition sits beneath a relation field of the queried object; a parent by its children is not served | `relation-filter-lowering.ts` header ("The reverse form (a parent filtered by its children) is not served here at all") | the two-step route stays | | `where` only: an aggregation's own `filter` and `having` refuse the form | `no-operator-object-door.ts` `relationWords` ("which the engine serves in 'where' and not in an aggregation's 'filter'" / "not in 'having'") | `INVALID_FILTER` / 400 | | At most 1000 related ids; past that refused, never truncated | `RELATION_FILTER_ID_CAP = 1000`, `relationFilterCapError` ("a cut-off id list would silently drop matching rows") | `INVALID_FILTER` / 400 | | As the caller: the related object's row scope and field permissions apply; a field the caller cannot read is refused, never an empty result | `packages/rest/src/data-nested-relation-permission.test.ts` (the security layer's filter-oracle guard) | `PERMISSION_DENIED` / 403 | Landing sites, all in `skills/objectstack-query`, before (`00a92e18d`) and after (`b73023a34`): | Site | Before | After | |:--|:--|:--| | `SKILL.md:76`, Removed-key row `query.joins` | "`expand` (display), or filter the related object and `$in` its ids" | "`expand` (display), or `{ relation: { field: value } }` in `where` (filter)" | | `SKILL.md:189-:191`, "Filtering by a related record" | the refusal (`INVALID_FILTER` / 400) and the two-step route | served in `where`, read as the caller; the five limits by name and the two-step route past them, by pointer to the rule (now `:189-:192`) | | `SKILL.md:331`, the `search` paragraph's last sentence | "To *filter* by a related record's column, `$in` ids from its own query" | "`where: { relation: { column: value } }`" (now `:332`) | | `SKILL.md:340`, Cross-Object row "Filter rows by their lookup target's column" | query the target, then `{ lookup: { $in: ids } }` | the served form in `where` up to 1000 related ids; past the cap the two-step route, `$contains` per id when `multiple` (now `:341`) | | `rules/filters.md:98-:115`, `## Relation Filters` | the refusal, one two-step example, the multi-valued spelling, the reverse direction | the served form with one ✅ example, one limits paragraph, the two-step route past the cap, the multi-valued spelling and the reverse direction unchanged | Re-read against the code and left as they are: `SKILL.md:88` (the rules index, "filtering by a related record"); `SKILL.md:326-:327` and `:342` ("`search` never traverses": `searchFields` names are judged exactly at the ingress, `protocol.ts` "Names are judged EXACTLY (no dotted-head tolerance)", so a dotted name is still refused and the mirror-field route still holds); `SKILL.md:341` "Filter parent by child conditions" (the reverse form, not served, stays two-step). The same-day rows `:26` and `:38-:44` (PR objectstack-ai#20776) are untouched. **Census.** `grep -rn -i -E 'nested.relation|two steps|two-step|\$in its ids|never traverses|relation.*INVALID_FILTER|filter the related object' skills/` at `00a92e18d` finds the sites above plus `evals/filters-pagination-search.json:37` (a `search` case: the mirror field, still true) and, outside this skill, `objectstack-ui/rules/list-views.md:235` and `objectstack-data/SKILL.md:120` (searching by a related record's title — still true). No other sentence in `skills/**` states the refusal or the two-step route as the only way. The evals carry no relation-filter case, so nothing there asserts the refusal; no block in the skill carries an `os:check` marker, so `check:skill-examples` does not apply. ## Sentences for objectstack-ai#20876 to quote `rules/filters.md` `## Relation Filters`, verbatim at `b73023a34`: A condition on a related record's fields beneath a relation field (`lookup`, `master_detail`, `user`, `tree`) is served in `where`: the engine reads the related object with it **as the caller**, then matches the field against the ids it returns (`$in`; any member when `multiple: true`). ```typescript // ✅ Orders whose customer is in the US where: { customer: { country: 'US' } } ``` Limits — one level: every key a field the related object declares, no relation or dotted key inside; forward only: never a parent by its children; `where` only: an aggregation's `filter` and `having` refuse it, `INVALID_FILTER` / 400; at most 1000 related ids, refused past that, `INVALID_FILTER` / 400, never truncated; as the caller: the related object's row scope and field permissions apply, so a field the caller cannot read is refused, `PERMISSION_DENIED` / 403, never an empty result. Past the cap, run the two steps yourself — filter the related object, then `$in` its ids: ```typescript const us = await engine.find('customer', { where: { country: 'US' }, fields: ['id'] }); where: { customer: { $in: us.map((c) => c.id) } } ``` On a `multiple: true` lookup match each id with `$contains` (an `$or` of those for several); the SQL driver refuses `$in` there. A parent by its children's fields is the same two steps reversed: query the child with the condition and `fields: [the lookup]`, then `{ id: { $in: … } }` on the parent. These sentences agree with the first landed wording of the fact — `content/docs/kernel/contracts/data-engine.mdx` (PR objectstack-ai#20872), the `FilterCondition` docblock form 4, and the changesets `20802-nested-relation-filter-served.md` / `20802-dotted-relation-route.md` / `20802-nested-relation-prose.md` — each read against the code; none contradicts it, and the skill contradicts none of them. The `where` dotted path (`{ 'account.industry': 'tech' }`) stays refused `INVALID_FIELD` / 400 and its words now name the nested form; the skill teaches no dotted `where` path, so no sentence changes for it. ## Paying for it inside `rules/filters.md` The file sat at 2149 / 2149 tokens (headroom 0). The section rewrite (881 → 1463 bytes) is paid for by deleting the `## Logical Operators` section (579 bytes, 41 lines), whose two rules already have a home in the same package; the file lands at 2148 / 2149, ceiling untouched. | Deleted from `rules/filters.md` | Home | |:--|:--| | `### AND (implicit)` — sibling keys are AND-combined; explicit `$and` | `SKILL.md` Logical Operators (`:173-:181`, the `$and` example) and this file's first Common Mistake ("sibling keys always are" AND) | | `### OR` — `$or`, and `$in` as the one-field equivalent | `SKILL.md` Logical Operators (`:170-:171`, the `$or` example) and `$in` (`:123`, `:128`); this file's Operator Reference `$in` row | The Common Mistake's pointer "(see **Logical Operators** above)" now reads "(SKILL.md, **Logical Operators**)". The five `role: …` literals in the deleted examples moved `check:role-word`'s count for the file 7 → 2; the gate prescribes the ratchet-down ("run `--update` and commit the baseline"), and commit `b73023a34` is that `--update` output: one row of `scripts/role-word-baseline.json`. That file is outside the claim's declared surface; declared here and in the report. ## `skills/**` readings (lines and tokens; tokens are the ratchet's `ceil(utf8 bytes / 4)`) | | Before (`00a92e18d`) | After (`b73023a34`) | Δ | |:--|--:|--:|--:| | `SKILL.md` | 399 lines · 4055 tokens (ceiling 5552) | 400 lines · 4109 tokens | +1 line · +54 tokens | | `rules/filters.md` | 214 lines · 2149 tokens (ceiling 2149) | 185 lines · 2148 tokens | −29 lines · −1 token | | Whole package `skills/objectstack-query` (6 files) | 1145 lines · 10527 tokens | 1117 lines · 10580 tokens | −28 lines · +53 tokens | | Whole catalog `skills/**` (every file) | 13375 lines | 13347 lines | −28 lines | Line budget (PM-set, net +4 at most): −28. No untouched line was re-wrapped; no ceiling row moved. **Changeset.** `skills/**` ships in no package's `files[]`: of the 82 tracked `package.json` files, 69 declare `files[]` and 0 entries name `skills` (measured at `b73023a34`); the catalog ships from `main` via `npx skills add`. Nothing published moves, so `skip-changeset` is the seat's to apply; this PR writes no label. ## Verification Gates at head `b73023a34`, each exit code captured by redirect and the verdict line quoted from its log: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` (no paths; "change set derived from git — 3 path(s) vs merge base 00a92e1") derives 31 commands, six families more than the dispatch's list because the baseline row adds `scripts/**`; reconciled with `--ran`: "31 derived famil(ies) accounted for — 31 run, 0 NOT-MEASURED (a DERIVED zero — all 31 recorded an exit code and none of them is 3)". Every one exit 0: `check-skills-token-ratchet` ("54 authored bundle file(s) within their ceilings") and its `--self-test` (65 cases), `check:role-word` after the ratchet-down ("Ledger: 44 baselined file(s) still carrying it (117 occurrence(s))"), `check:corpus-claim-drift`, `check:skill-identifier-liveness` ("Leg 1: 457 citation(s) over 53 published file(s)"), `check:doc-authoring`, `check:nul-bytes`, `check:skill-compatibility`, `check:skill-frame-sync`, `check:agent-test-spelling`, `check:cross-package-test-inputs`, `check:driver-memory-census`, `check:gitlink-declared`, `check:pm-governed-merges` (the `--self-test`, as the script spells it), `check:refd-timer-probe`, `check:watch-hint-literal`, `check-ci-filter-parity` (+ `--self-test`), `check-closing-keyword-parity` (+ `--self-test`), `check-comment-mask-corpus` ("7664 files, 0 disagree"), `check-doc-route-spelling --advisory` (+ `--self-test`), `check-scripts-symbol-anchors` (+ `--self-test`), `check:bash32-floor`, `check:cli-command-ids`, `check:entry-guard`, `check:parse-guard`, `check:pnpm-filter-targets`, `@objectstack/lint` `check:doc-formula-expressions` ("22 record-scoped formula example(s) across 458 files / 1380 TS blocks judged clean", after building `@objectstack/lint...` under `os-verify-lock.sh`, `VERDICT command-exit 0`), spec `check:skill-docs` ("Skill docs in sync") and spec `check:skill-refs` ("9 generated files in sync"). The two ratchet families were re-run after the final commit at `b73023a34` (both exit 0, verdicts as quoted). Pins of the served form, run first-hand under `os-verify-lock.sh` after building each package's dependency closure (`VERDICT command-exit 0` on both): `pnpm --filter @objectstack/objectql exec vitest run --maxWorkers=2 src/engine-nested-relation-lowering.test.ts` → "Test Files 1 passed (1), Tests 13 passed (13)" (the served rows on every relation type, the any-member `$or` of `$contains`, the cap refusal with the two-step route, the one-level / dotted / undeclared refusals, the `aggregations[i].filter` and `having` refusals naming `where`, the caller's context on the related read); `pnpm --filter @objectstack/rest exec vitest run --maxWorkers=2 src/data-nested-relation-permission.test.ts src/data-nested-object-door.test.ts` → "Test Files 2 passed (2), Tests 9 passed | 12 skipped (21)" — the 12 are the PostgreSQL and MySQL cells of the door test, named skips (`OS_TEST_POSTGRES_URL` / `OS_TEST_MYSQL_URL` unset here); the memory and SQLite cells and the 403 pin ran. No package is touched, so no package build or test suite is owed beyond that. ## Acceptance notes - `SKILL.md`'s `compatibility: Requires @objectstack/spec 17.x` is left as is: the catalog states `main`'s behaviour (as PR objectstack-ai#20811 did) and `main` is 17.5.0 with the served form; the published 17.5.0 engine still refuses it, and which release carries `ca5408c62` is the ruling's sequencing, not this card's. Noted, not filed. - The skill says nothing about analytics (the cube read and the read scope): objectstack-ai#20887 is open, and the sentences above name `where` on the engine and the data door only. - `content/docs/protocol/objectql/query-syntax.mdx` is read-only here; objectstack-ai#20876 carries it and quotes the section above. - The report's `api_writes` lists every relay write of this run. ## 维护者速读(草稿) **改了什么。** 发布的查询技能包 `skills/objectstack-query` 今天早些时候(PR objectstack-ai#20811)被改成「引擎拒绝在关联字段下直接写条件」;同一天晚些时候引擎侧 PR objectstack-ai#20872 落地,引擎已经支持这种写法(`where: { customer: { country: 'US' } }`)。本 PR 把同样的几处句子改成引擎现在的真实行为:支持,以及四条边界(只到一层、只能正向、只在 `where` 里、关联 id 最多 1000 条超出即拒绝),并且以调用者身份读关联对象(读不到的字段报 403,不会静默给空结果);超过上限或反向(用子记录条件筛父记录)仍走原来的两步 `$in`。规则文件里删掉了与 SKILL.md 重复的一节逻辑运算符示例,用来支付这段改写;整个包净减 28 行,每个文件的 token 上限都没动。 **为什么改。** 技能包是客户项目里 AI 的教材。裁决 5907789183(「20802 同意」)明确写了技能包在同一轮更新;不改,AI 作者会照着旧句子手写两步查询,正是裁决要去掉的模式。措辞与已落地的文档页(`data-engine.mdx`)、三份 changeset 和 `FilterCondition` 的注释逐句核对过,一致;objectstack-ai#20876 改 `query-syntax.mdx` 时引用本 PR 的句子,两处不会分叉。 **风险与代价(含回滚)。** 只改两份 markdown 与一行门禁基线(`role-word` 计数 7 → 2,门禁自己要求的下调),不改任何代码或发布包。风险在两点:一是措辞若与后续分析(objectstack-ai#20887)落地后的行为有出入,分析那一半另改;二是已发布的 17.5.0 引擎仍拒绝这种写法,技能包描述的是 `main`。回滚即 revert 本 PR 的两个提交。 **席位意见。** **你要做的。** 复核五条边界的措辞与「逻辑运算符」一节的删除归宿;批准后由席位落地(Tier H)。 --- _Generated by [Claude Code](https://claude.ai/code/session_01KTZmMfzVzjNvyaLyQ8mHvg)_ --------- Co-authored-by: objectstack-fleet[bot] <332303061+objectstack-fleet[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
… answer on every analytics face — the related object read as the caller, capped (objectstack-ai#20887) (objectstack-ai#20916) Fixes objectstack-ai#20887 Clause-②: yes (narrowing) The analytics half of ruling 5907789183, whose parent card is objectstack-ai#20802 (its engine half landed as objectstack-ai#20872, `ca5408c62`). The nested-relation filter `{ relation: { field: value } }` now gets ONE answer on every analytics face, and it is the engine's: the related object read as the caller (its row scope and field permissions), capped at `RELATION_FILTER_ID_CAP`, a multi-valued relation matching on any member. The analytics layer holds no copy of that rule. The native-SQL strategy declines a query carrying the form, and the engine-aggregate strategy hands the form to the engine as written. ## Per face The engine's answer for the same filter, computed in the same test over the same rows, is the reference for every cell. Fixture: a ledger with `owner` (lookup) and `owners` (multiple lookup) to an owner object; the member cannot read `owner.secret`, and its row scope hides owners in region `HIDDEN`. Past the cap means 1,001 matching owners. Measured with the real `SecurityPlugin`, `ObjectQL` and `SqlDriver` (SQLite). | face | strategy | engine rows equal | caller permissions | cap | |---|---|---|---|---| | cube read, `POST /api/v1/analytics/query` (`AnalyticsService.query`) | NativeSQL composition (declines, so the engine answers) | yes: single, multi, related row scope, `$not`, `$or` (was 500 `DATABASE_ERROR`: the join named a table `owner` that does not exist) | 403 `PERMISSION_DENIED`, as the engine (was 500) | 400 `INVALID_FILTER`, as the engine (was 500) | | cube read | ObjectQL | yes (was 400 `INVALID_FIELD`, "cannot evaluate a cross-object filter") | 403 (was 400) | 400 (was 400 cross-object) | | dataset door, `POST /api/v1/analytics/dataset/query`, the dataset `include`s `owner` | NativeSQL composition | yes (was: single-valued rows via the JOIN; multi-valued 400 `DATASET_INVALID`; under `$not` the member got `b` where the engine answers `b, d`) | 403 (was **200 with rows a, c**: filtered by a field the caller cannot read) | 400 (was 200 with no rows) | | dataset door | ObjectQL | yes (was 400) | 403 (was 400) | 400 (was 400) | | a measure's own `filter` carrying the form | both | refused 400 `INVALID_FILTER`, as the engine refuses it at an aggregation's `filter` (was: native counted it through the JOIN; ObjectQL 400 `INVALID_FIELD`) | n/a | n/a | | SQL echo, `POST /api/v1/analytics/sql` | both | refused 400 `INVALID_FILTER`, naming the served route (was: native printed the JOIN; ObjectQL 400) | n/a | n/a | | a read scope carrying the form (a host `getReadScope`) | NativeSQL | refused 500 `READ_SCOPE_COMPILE_FAILED`, policy withheld, words now naming the route (outcome unchanged) | n/a | n/a | | a read scope carrying the form | ObjectQL | served, the engine's rows (unchanged: the scope reaches the engine as written) | as the caller | the engine's | ## Mechanism assumptions, measured - **B1 held.** The engine's answer for the fixture, as the member: `{ owner: { region: 'NA' } }` is d1, d3; the multi-valued form is d1, d3; `{ owner: { secret: 's1' } }` is 403 `PERMISSION_DENIED` naming `secret` (a system caller gets d1, d3); region `HIDDEN` gives no rows (a system caller gets d4); past the cap is 400 `INVALID_FILTER` for both spellings; `$not` gives d2, d4; the `$or` gives d1, d2, d3; `{ owner: {} }` and a second level are 400. `RELATION_FILTER_ID_CAP` is exported (`packages/objectql/src/index.ts:148`), and nothing here imports it: the analytics layer never counts ids, the engine does. - **B2: no on every axis**, per the table. The native path joined the related table itself: the related row scope rode in as a `WHERE` conjunct, the field permissions did not, nothing bounded the match, and a multi-valued relation or an undeclared join failed. The ObjectQL path refused the form outright. - **B3: call the engine.** `@objectstack/objectql` exports only the cap. The lowering (`admitRelationCondition`, `lowerRelationSite`) is module-internal, and this package has `@objectstack/objectql` as a dev dependency only. The route that needs no export: the engine-aggregate strategy hands the condition to `engine.aggregate` through `executeAggregate`, with the caller's context. No export was needed, and there is no second permission rule. - **B4: the read scope keeps its refusal, and the words name the served route.** `compileScopedFilterToSql` is a synchronous string builder. It holds the caller's `ExecutionContext` for placeholders only, and no data engine, so its compile cannot run the inner read as the caller. Routing a read scope carrying the form to the engine instead was built and measured, then withdrawn: on a native-only host it traded the declared `READ_SCOPE_COMPILE_FAILED` (policy withheld) for a generic no-strategy fault (`packages/rest/src/analytics-read-scope-refusal-envelope.test.ts` went red). No in-repo producer emits the form in a scope: the RLS compiler refuses a relation traversal when it compiles the policy. On the ObjectQL path the scope reaches the engine as before, and the engine serves it as the caller. - **B5: `yes (narrowing)`.** Widening: the cube read (both strategies), the ObjectQL dataset door, a multi-valued relation and a dataset without the declared join on the native path, and a dataset's own `filter` on the ObjectQL path all now serve the form (they answered 500 or 400). Narrowing: on the native path the dataset door now refuses a condition on a related field the caller cannot read (was rows), a match past the cap (was an empty 200), and a measure filter carrying the form (was a count). The SQL echo refuses the form. And a query combining the form with something only the native strategy serves (a cross-object measure, a multi-hop dimension) is refused by the engine-aggregate path. `@objectstack/service-analytics` ships `minor` with the BREAKING banner and an ADR-0087 `not-required (no-migration-prescription)` disposition; `check-adr-0087-registration` and `check-changeset-no-major` pass. - **B6: no page to update.** No hand-written `content/docs/**` page states how the analytics read or the read scope treats the nested form. `data-engine.mdx`, and `query-syntax.mdx` (objectstack-ai#20906, which landed during this work), describe the engine only. ## What changed - `strategies/filter-normalizer.ts`: a nested-relation condition becomes a `relation` node carrying the condition as written. It is no longer flattened to the dotted member. `shieldNestedRelations` holds it out of the shared lowering, because under `$not` the lowering guarded the relation column, and this package's engine hand-off spells that guard `$ne: null`, which `driver-sql` refuses over a multi-valued JSON column. Measured: the multi-valued `$not` pin went red before the shield, and the engine guards what it lowers the condition to itself. `findNestedRelationCondition` is the routing detector. - `strategies/native-sql-strategy.ts`: `canHandle` declines when the `where`, the dataset's own `filter` or a requested measure's `filter` carries the form. This is the mechanism of the cross-field decline (maintainer ruling 2026-08-12, Q1 = B). Its compiler refuses a `relation` node bare, as routing drift. - `strategies/objectql-strategy.ts`: the condition goes to the engine as its own conjunct, under the key the author wrote. The display-SQL echo declines it. - `read-scope-sql.ts`: the nested-relation form's refusal has its own words, naming the route. An empty or mixed value object keeps the old words. - `analytics-service.ts`: the no-strategy error names the nested-relation decline. - The mixed-wrapper refusal no longer says a nested member "compiles to the dotted member". ## Pins, red first (`568727629`) - `packages/rest/src/analytics-nested-relation-filter.test.ts`: both compositions, the cube read and the dataset door through its route, against the engine's answer. It was red 10 of 10 on the base, and is 10 of 10 green now. - `packages/services/service-analytics/src/__tests__/nested-relation-engine-handoff.test.ts`: the native decline per producer, the ObjectQL hand-off as written, the compile backstop and the read-scope words. It was 7 red with 2 controls green on the base, and is 9 of 9 green now. ## Ablations, predicted before running, at `5bb764181` Each ablation mutated the committed file through `scripts/ablation-replace.mjs` (anchor hit once, blob moved), rebuilt `@objectstack/service-analytics`, and passed `ablation-dist-preflight` (the marker present in 2 built files). It then ran both pin files, plus `where-door-shared-lowering-seam.test.ts` in the unit run. The restore leg proved the blob equal to HEAD and `git diff HEAD` empty, rebuilt, and found the marker absent from all 6 built files. Every observed count equals its prediction. | ablation | face it guards | unit (27) | route pins (10) | |---|---|---|---| | A1 the native decline removed | cube read and dataset door, native | 4 red | 4 red (native: rows, refusals, measure filter, echo) | | A2 the hand-off drops the condition | both strategies' rows, permission, cap | 2 red | 6 red | | A3 the aggregate call forwards no caller context | caller permissions | 0 | 4 red: the member then saw `d` (a hidden owner's row), and the unreadable field answered rows | | A4 the read-scope route words | read scope, native | 1 red | 1 red | | A6 the lowering shield removed | multi-valued `$not` | 1 red | 2 red | | A7 the echo refusal removed | SQL echo | 0 | 2 red | A first round at `dca1af7cb` matched its own predictions too, including A5, the read-scope decline arm, which B4's correction removed from the code. ## Pins re-judged These pins recorded the flattening this change removes, so each was re-judged: - respelled to the dotted cube member where the pin was about the traversal: `filter-normalizer-not-null-safe`, `icontains-text-comparand-refusal`; - re-expected as a `relation` node where the pin was about acceptance: `where-equality-slot-list-refusal`, `where-face-arms-refusal`, `where-type-face-refusal`, `filter-normalizer-mixed-wrapper`'s pure-shape block; - replaced where the pin held the removed branch: mixed-wrapper row 6 and its `guardFieldEntry` recursion row (the engine refuses that inner wrapper, `INVALID_FILTER` / 400, measured), `where-door-shared-lowering-seam`, `infer-cube-relation-traversal`, `infer-cube-where-spelling-parity`, and `where-source-field-gate`, which now judges the relation field `owner` as a column of the queried object; - re-worded to the read-scope refusal's new words: `read-scope-sql`, `read-scope-not-null-safe`, `read-scope-undefined-comparand`, and `read-scope-refusal-envelope`, which gains row 17 because the nested-relation form now has a throw site of its own. ## Verification - `@objectstack/service-analytics`: `test` 146 files, 3334 passed; `typecheck` exit 0, with 146 of 146 test files in the tsc program (`--listFiles`). Both at `4d383dac0`, after merging `main`. - `@objectstack/rest`: the full `local` project, 239 files, 4662 passed and 106 skipped, at `5bb764181`. The merge brought no rest or analytics change. At `4d383dac0`, the new pin, the read-scope envelope pin and the engine half's permission pin: 3 files, 21 passed. `typecheck` passes, including the test layer (`check:test-typecheck` OK). - Consumer sweep, narrowed to the files that load this package: `@objectstack/runtime` `analytics-*` plus `cross-field-refusal-operand-withhold`, 5 files, 38 passed and 4 skipped; `@objectstack/client` `analytics-automation-json-erasure`, 7 passed. - Gates at `4d383dac0`: `dispatch-gates --commands` derived 62. All 62 were run, plus the 4 roster families (`check-changeset-fixed`, `check:authz-resolver`, `check:error-code-casing`, `check:filter-alias-parity`), all exit 0. `dispatch-gates --ran`: 62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN. `check:dual-build-cjs-loads` and `check:type-check-debt` first exited 3 (prerequisite not met) and were re-run green after `turbo run build` over `./packages/*`. - Lint, narrowed and proved at `4d383dac0`. The population is the 21 changed `.ts` files, none ignored by eslint's own config (`isPathIgnored` false for all 21). `eslint --no-inline-config --format json` over them gives 21 files, 0 errors, 0 warnings. `parserOptions.project` and `projectService` are unset for all 21, so no type-aware lint runs and no untouched file's verdict can move. - NOT MEASURED: a live PostgreSQL cell (the dialect axis of the lowering is the engine's, pinned by objectstack-ai#20872's `data-nested-object-door.test.ts`; this file's axis is the analytics faces), the dogfood and integration lanes, and the whole-workspace typecheck. All are left to CI. ## Acceptance notes - The ObjectQL path's cross-object refusal still says "Run this query on a native-SQL driver". A query the form routed away from the native strategy can meet those words on a SQL deployment. - The engine's cap refusal names the position the engine received. The strategy ANDs the condition in, so the words read `where.$and[0].owner` where the author wrote `where.owner`. - `packages/types/src/error-leak.test.ts` keeps a hand-written stand-in of the read-scope refusal shapes. Its nested-relation line is the old wording. It is a heuristic fixture, not a pin of this module, and it stays green. - `MemoryAnalyticsService` (driver-memory's cube face, objectstack-ai#20859's position) is not touched, and its answer for the form is not measured here. The draft preview refuses the form as an operator it cannot evaluate, unchanged. ## Patch rounds (the seat's append from the dev's reports `5917211340`, `5917807320` and `5918233769`; the dev writes a body only once) ### Patch round 1 `Test Core (3/6)` went red on `4d383dac0`, in `packages/client`'s `envelope-caller-census.test.ts`: 2 of its tests failed. Reproduced locally: the client suite fails 1 file and 2 tests at `4d383dac0`, and passes 50 files and 641 tests at the merge base `9509ea106`. **Root cause.** The census walks the whole workspace for call sites of `analytics.query(` and requires a hand-ledger row for each one. This PR's new pin, `packages/rest/src/analytics-nested-relation-filter.test.ts`, calls the real `AnalyticsService`'s `analytics.query` five times. Those are producer reads, the census's `NOT_SDK` class, and the ledger has no row for them. The failing assertions are the §3 key comparison (the one extra key is that file, `analytics.query`, `service`, 5) and the §2 producer-receiver count (1 expected, 6 found). **What it is not.** It is not a product defect. It is not a client pin of the nested-relation form or of the read-scope wording either. **The fix, pending the seat.** It lives in `packages/client/src/envelope-caller-census.test.ts`, which is outside this card's claim surface. It adds one `NOT_SDK` ledger row with a count of 5, and moves the two producer-read counts from 1 to 6. Measured on a scratch copy of that file: 20 of 20 tests passed. The copy was restored byte-identical, and nothing was committed. ### Patch round 2 The seat authorised the census remedy, round 1's option A, for one file: `packages/client/src/envelope-caller-census.test.ts`. - **main moved.** `975b2481c` (objectstack-ai#20808) touched `packages/services/service-analytics`, so `origin/main` was merged into the branch as `1fdaff7e5` (no rebase). The merge was clean, with no regeneration pending. - **The census change is its own commit, `7eb2ecf20`.** - It adds one `NOT_SDK` ledger row for `packages/rest/src/analytics-nested-relation-filter.test.ts` (`analytics.query`, `service`, count 5). - The producer-receiver count goes from 1 to 6, and the set of two files is asserted. - `verdictTotal('NOT_SDK')` goes from 1 to 6, and its test title changes with it. - Nothing else in that file changed. - **Measured at `7eb2ecf20`.** Each exit code was captured before any pipe. - `pnpm --filter @objectstack/client test`: exit 0, 50 files and 641 tests passed. The census file run alone passed 20 of 20. - `pnpm --filter @objectstack/client typecheck`: exit 0. `tsc --noEmit` passed, and `check:test-typecheck` answered OK. - `@objectstack/service-analytics` test: exit 0, 146 files and 3335 tests passed. Its typecheck: exit 0. - The three rest files (`analytics-nested-relation-filter`, `analytics-read-scope-refusal-envelope`, `data-nested-relation-permission`): exit 0, 3 files and 21 tests passed. - ESLint over the 22 changed `.ts` files: 0 errors and 0 warnings. The config ignores none of them and lints none type-aware, so this diff cannot move a verdict on an untouched file. - Gates, re-derived: 63 derived and 63 run, 0 not measured, plus the 4 roster families. All exit 0 except one. - **The one red is `check:cross-package-test-inputs`.** - Cause: the new ledger row spells the rest pin's path as a literal, and `@objectstack/client`'s declared cross-package input globs do not cover it. The gate is green at `1fdaff7e5`, the commit before. - The gate's own remedy: declare that one file in `scripts/cross-package-test-inputs.mjs`, and mirror it in `turbo.json`'s `@objectstack/client#test` inputs. - Measured on the working tree: that gate and `check-ci-filter-parity` both exit 0. The two files were then restored byte-identical. - Both files lie outside the authorised surface, so the remedy waits for the seat. ### Patch round 3 The seat authorised the gate's own remedy for `check:cross-package-test-inputs`, in two files. - **main.** No commit since `975b2481c` touched this card's surface, the census or either of the two files, so there was no merge. The commits checked were `def279a39`, `4d0b9cd54` and `d78a0bda0`. - **The declaration is its own commit, `2881f478c`.** - `scripts/cross-package-test-inputs.mjs`: in `@objectstack/client`'s entry, one per-file glob, `packages/rest/src/analytics-nested-relation-filter.test.ts`, with a 3-line comment. - `turbo.json`: `$TURBO_ROOT$/packages/rest/src/analytics-nested-relation-filter.test.ts` in `@objectstack/client#test`'s inputs. The line before it gains the comma JSON requires. - Nothing else changed in either file. - **Measured at `2881f478c`.** - Gates, re-derived: 81 derived and 81 run, 0 not measured. Also run: the 11 roster families whose roster lies under a path this diff touches, `check-ci-filter-parity --self-test` and `check:select-shard-packages`. All 93 commands exit 0. - `check:cross-package-test-inputs` (with `--self-test`) is green: "OK: 29 package(s) read outside themselves, all declared". - `check-ci-filter-parity` is green: "all 188 declared cross-package glob(s) (135 unique) are covered". - `check:turbo-task-graph` is green. - `pnpm --filter @objectstack/client test`, as the control: exit 0, 50 files and 641 tests passed, the same as at `7eb2ecf20`. The declaration moved no verdict. - Layer A at work: `--union-into`, given a diff of the rest pin alone, now pulls `@objectstack/client` into the run (8 packages). At `7eb2ecf20` it did not (7 packages). No other package changed. - Turbo hashes, from `--dry=json` before and after, over build, test, test:repo and typecheck (303 tasks): - The global hash is unchanged, and no build or typecheck hash moved. - 7 test hashes moved. `@objectstack/client#test` moved through `turbo.json`: its task definition changed, and the rest pin is a new input. - The other 6 moved only because the content of `scripts/cross-package-test-inputs.mjs`, an input they declare, changed. They are `cli#test`, `plugin-auth#test`, `vitest-filter-preflight#test`, `objectql#test:repo`, `runtime#test:repo` and `spec#test:repo`. - ESLint over the 23 changed `.ts` and `.mjs` files: 0 errors and 0 warnings. The config ignores none of them and lints none type-aware. ### Patch round 4 (the seat's append from the dev's report `5922062971`) `main` was merged (no rebase) to take in four landings in `service-analytics`: - PR objectstack-ai#20931: the field-read gate at the door. - PR objectstack-ai#20955: the queryable-field gate. - PR objectstack-ai#20954: `plugin-security`'s comparand guard. - PR objectstack-ai#20962: relationship-path objects in the admitted and scoped set. **The merge, `6b6bffb3e`.** It is clean at the text level, in `analytics-service.ts` and in `native-sql-strategy.ts`. Every line either side added is present in the merged files, checked line by line. **What the landed gate and object set do with the `relation` node.** This was measured on the merged tree, in the shipped composition (the real `SecurityPlugin` over `ObjectQL` on SQLite), under both strategies. - **The gate judges the relation field.** `collectFilterLeaves` yields the nested form's relation field as its member: `{ owner: { region: 'NA' } }` gives `owner`, with operator `relation`. It does the same under `$not` and inside `$or`. So the field gate judges the relation field on the base object. - A caller who may not read `owner` is refused by the gate: 403 `PERMISSION_DENIED`, in the engine's own words, with no engine call made. - **The related object does not enter `queryObjects`.** The security service is asked only about the base object. - **The engine guards the related object instead.** The nested form is served on the ObjectQL path, where the engine reads the related object as the caller. Each of these is refused with the same code, status and words as `engine.find`, and never answered: - a related field the caller cannot read: 403; - a masked related field (objectstack-ai#20935): 403; - a related object the caller cannot read at all: 403. - **Before this branch, the answer was a refusal.** On `main` alone, even a readable nested condition was refused 403, "reading "owner" is not permitted", because the flattened `owner.region` named the relation field as an object to admit. With this branch, the answer is the engine's. **Measured at `6b6bffb3e`.** Every run was under the shared lock, with each exit code captured before any pipe. - `@objectstack/service-analytics`: tests exit 0 (149 files, 3434 tests), and typecheck exits 0. - The five rest route pins pass 61 of 61: - `analytics-nested-relation-filter`: 10 - `analytics-read-scope-refusal-envelope`: 8 - `data-nested-relation-permission`: 3 - `analytics-field-permission-gate`: 12 - `analytics-relationship-path-admission`: 28 - The client census passes 20 of 20. - Gates, re-derived and run as one locked sequential script: 81 derived, 81 run, 0 not measured. Also run: the 11 roster families and 2 extras. All 93 commands exit 0. - ESLint over the 23 changed `.ts` and `.mjs` files: 0 errors and 0 warnings. **Acceptance notes.** - **A dotted path on an inferred cube is still refused.** `{ 'owner.region': 'NA' }` is refused 403, "reading "owner" is not permitted". The hop object is taken from the alias, because an inferred cube declares no join. This is the same on `origin/main`, and it is outside this card; it went to the seat as a finding. - **A host read scope does not reach the related object.** The related object is not in `queryObjects`, so a host-supplied `getReadScope` is not asked about it. In the shipped composition that provider is the security service's `getReadFilter`, the same row scope the engine applies when it reads the related object as the caller. That case is pinned in `analytics-nested-relation-filter` ("the related row scope"). --- _Generated by [Claude Code](https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
…is membership, on every face (objectstack-ai#20984) Fixes objectstack-ai#20874 Clause-②: yes (widening) ## What changes `driver-memory` now answers `$contains` and `$notContains` on a **declared JSON-stored field** by whole-element membership, the reading `driver-sql` compiles on all three dialects. A field counts as JSON-stored when it is `multiple: true`, a `multiselect` / `checkboxes` / `tags` field, or a `STRUCTURED_JSON_TYPES` member such as `json`. A scalar text column keeps the case-exact substring test. The change covers every face of the package: - the query path's `$`-spelling (`normalizeFieldOperators`) and its AST spelling (`convertConditionToMongo`): `find`, `count`, and every verb that goes through `convertToMongoQuery`; - the analytics (cube) face's mingo `$match`; - the analytics face's SQL echo (`generateSql`), which now renders SQLite's `json_each` membership construct for such a column, so the echoed statement still returns the rows the chart was drawn from. One rule backs all of them: `InMemoryDriver.filterContainsTest(object, field, value)`. On a JSON-stored field it returns `{ $elemMatch: { $in: members, $not: { $type: 'array' } } }`, and on any other field `{ $regex: filterSubstringPattern(value) }`. `$notContains` is `$not` over that test, on both faces. The members come from the comparand's text, read the way `driver-sql`'s `jsonMembershipCandidates` reads it: `'1'` names the string or the number 1, `'1.50'` names 1.5, and `'true'` / `'false'` / `'null'` name the string or the JSON literal. The `$contains` docblock in `packages/spec/src/data/filter.zod.ts` lost its stale "driver-memory DOES NOT ANSWER IT YET" bullet. Its dead tracker link went with it (that card answers 404). The replacement text states the measured status, and states that the other text operators over a stored array are not ruled by that section. ## Measured on `origin/main` `f6ccca4a` before the change, and after (HEAD `10656601`) Fixture: `driver-sql`'s objectstack-ai#17590 fixture plus a multi-valued lookup `owners` (`['u1','u2']`, `['u10']`, `['u3','u1']`, `[]`). | `where` | memory before | SQLite (`driver-sql`) | memory after | |:--|:--|:--|:--| | `{ owners: { $contains: 'u1' } }` | 1, **2**, 3 | 1, 3 | 1, 3 | | `{ owners: { $notContains: 'u1' } }` | 4 (**2 dropped**) | 2, 4 | 2, 4 | | `{ tags_: { $contains: 'red' } }` | 1, **2** | 1 | 1 | | `{ nums: { $contains: '1' } }` (members are numbers) | **none** | 1 | 1 | | `{ label: { $contains: 'red' } }` (scalar control) | 1, 2 | 1, 2 | 1, 2 | | the cube face, the same filters | the query path's rows | n/a | the query path's rows | `engine.find` on a real `InMemoryDriver` with the nested-relation filter `{ owners: { region: 'NA' } }` (owner `u1` NA, `u10` EU; this is PR objectstack-ai#20872's lowering): `d1, d3, d5` before and `d1, d3` after. Its `$not` gave `d2, d4` before and `d2, d4, d5` after, which matches the rest suite's SQLite rows. That was a one-off run (a throwaway probe file, not committed): this package may not import the engine and the engine's packages may not import this driver (`check:driver-memory-census`). What is pinned instead is the driver input the engine sends, an `$or` of one `$contains` per related id, which objectql's `engine-nested-relation-lowering.test.ts` asserts. ## Mechanism hypotheses (order Zone 2) - **H1 confirmed.** The `$contains` arm lowered to an escaped `$regex`, and mingo applies a `$regex` to each element of an array value. Reproduced on the fixture above before any edit. - **H2: the fork is by the DECLARED field, not the row's runtime shape.** Memory has the declaration (`valueShapes`, recorded by `syncSchema` since the `$empty` work), and SQL forks on its JSON-column registry, which is also filled from the declaration. The population is the spec's JSON-stored classes: `STRUCTURED_JSON_TYPES` plus `isMultiValueField`, the two halves `driver-sql`'s registry is built from. The two readings DO give different rows on one fixture, so the choice goes to the PM in the report's `open_questions`: a declared `json` field holding the scalar string `'u1'`. SQLite answers no member, because its constructs are array-only. A per-row-shape fork would answer it by substring. The declared fork matches SQLite, and the contract text says the question is "selected by the COLUMN", declared metadata. Undeclared fields (an object never passed through `syncSchema`) keep the substring reading, as `SqlDriver.isJsonColumn` answers `false` for a table it was never told about. - **H3 confirmed and pinned.** Number members answered nothing; `'1'`, `'2'`, `'10'`, `'0'` and `'1.50'` now give SQLite's rows. - **H4 confirmed: `$notContains` diverged.** It is the mirror arm of the same defect, changed under os-dev rule 3's bounded in-place exemption. All four conditions hold: 1. Same defect class. 2. Mechanical, with its shape pinned by SQL's `col IS NULL OR NOT (…)`: `$not` over the test admits null and missing rows, pinned on both fixtures. 3. `memory-driver.ts` is held by no other claim; the sibling `$exists` card is kept off it by its own claim. 4. Same tests, no new gate. **The claim's surface should be amended to include it.** - **H5 confirmed.** The cube face borrowed `filterSubstringPattern` and wrapped it in its own `$regex`, so it had the same defect. It now takes the driver's whole test. The echo reads its member set off that same test, so the chart and its echo cannot name different sets. - **H6 confirmed.** See the `engine.find` run above. ## Compile-surface conclusions | # | face | conclusion | |---|---|---| | 1 | `driver-sql` `applyFilterCondition` (`driver-sqlite-wasm`, `driver-turso` local inherit it) | **already compliant**: `applyJsonMembership` emits membership on JSON columns. Evidence: the objectstack-ai#17590 suite now carries a multi-valued lookup column and the `u1`/`u10` case, green on SQLite here. Its live PostgreSQL/MySQL cells are unprovisioned locally; the required live job runs the whole driver-sql suite. The two inheriting drivers were not separately run. | | 2 | turso `RemoteTransport.buildWhereSQL` | **out of scope** (another package). By reading, its `$contains` arm emits `pushLike` (a GLOB substring) on every column, JSON columns included, so remotely `u1` would match `["u10"]` while the local transport answers membership. Not measured. Reported as a finding. | | 3 | service-analytics `read-scope-sql` `compileScopedFilterToSql` | **out of scope** (another package). **Measured**: `{ owners: { $contains: 'u1' } }` on a declared `lookup` + `multiple: true` field compiles to `instr("t"."owners", ?) > 0` on SQLite and `LIKE '%u1%'` on PostgreSQL/MySQL, a substring over the stored JSON text. This is an RLS read-scope face, so it is reported as a security-relevant finding. | | 4 | service-analytics `filter-normalizer` `lowerAnalyticsWhere` | **out of scope**. By reading, it lowers `$contains` to the cube `contains` operator, which the native SQL strategy renders as `LIKE`, a substring. Reported. | | 5 | `formula` `matchesFilterCondition` | **out of scope**. By reading, the arm is `typeof actual === 'string' && actual.includes(v)`, so a stored array never matches (fail-closed). Reported. | | half | objectql `having-filter` | **out of scope by the claim** (serial behind another card). Untouched. | | unfrozen | `driver-memory` query path and cube face (`memory-analytics.ts`), behind `filter-refusal.ts` | **changed** (this PR). `filter-refusal.ts` and the `$exists` arm are untouched (sibling card). | | unfrozen | `driver-mongodb` `translateFieldOperators` | **out of scope** (another package). By reading, it lowers `$contains` to a native `$regex`, which MongoDB applies per array element: the same defect. Not measured. Reported. | ## Why the shared pin is a mirrored literal table, not `FILTER_TEXT_CASES` (order Zone 3, not taken) `FILTER_TEXT_CASES` has no array column, so adding one changes the fixture of all five enrolled drivers. `driver-mongodb` imports every row of it and still carries the per-element defect, so its suite would go red. The table's own rule 2 says rows join a driver's suite in the PR that ends that driver's gap. Doing it here would also widen the spec touch beyond the one declared docblock. A new sibling case-set would add DEBT rows to a ledger that only goes down. So the new memory file mirrors `driver-sql`'s objectstack-ai#17590 fixture row for row and asserts the same literal row sets. That is how the objectstack-ai#17590 file already holds its three dialect cells to one answer. ## Files - `packages/drivers/driver-memory/src/memory-driver.ts`: the population (`isJsonStoredField`), the members, the one test (`filterContainsTest`), both query-path spellings. - `packages/drivers/driver-memory/src/memory-analytics.ts`: the `$match` rows take the driver's test; the echo renders membership (`sqliteMembershipPredicate`). - `packages/drivers/driver-memory/src/memory-20874-contains-membership.test.ts` (new): two fixtures. The first is the objectstack-ai#17590 fixture plus `owners`. The second holds the stored shapes SQLite decides: a scalar, an object, a nested array, `[null]`, `[[null]]`, `[true, 1.5]`, `['']` and a NULL row, each answer measured on `driver-sql`/SQLite first. Every case runs on `find` in both spellings, on `count`, and on the cube's `$match`. The cube's echo is EXECUTED on sql.js over the same rows. The file also covers the nested-relation driver input and the fork. - `packages/drivers/driver-sql/src/sql-driver-17590-json-column-membership.test.ts`: an `owners` multi-valued lookup column and the `u1`/`u10` case (the claim's SQLite pin). - `packages/spec/src/data/filter.zod.ts`: the one declared docblock. - `.changeset/20874-memory-contains-membership.md`: **`minor`**, not the `patch` the order suggested. `filterContainsTest` is a new public method on the exported `InMemoryDriver`. It ships in `dist/index.d.ts` (the existing `filterSubstringPattern` appears there as the positive control), and the level ruling quoted in `pr-automation.yml` (WHICH LEVEL) grades an additive widening of a published package's public surface at least `minor`. `Clause-②: yes (widening)`: the new public method widens the published surface; no filter key or operator is added. - **Sweep, beyond the claim's listed surface.** The order's pin sweep asks for every same-semantics statement to be flipped in one round. Each item below is an edit to a statement this change makes false: - `packages/rest/src/data-nested-object-door.test.ts` (a header comment that cited the gap and the removed docblock clause); - `.changeset/20802-nested-relation-filter-served.md`, one sentence of a **pending** release note that said the in-memory driver still matches per element. ## Confirmation requested: a pending release note is corrected This PR changes `.changeset/20802-nested-relation-filter-served.md`, a pending release note this PR did not add. This is the DELIBERATE CORRECTION class `check-empty-changeset.mjs` names, so `Check Changeset` stays **red** on purpose. It is not a required context. `skip-changeset` must not be applied. The note's last sentence said: > On the in-memory driver, a multi-valued relation's `$contains` still matches a stored id by substring per element, so there an id that is a substring of another stored id (`u1` inside `u10`) also matches; SQLite and PostgreSQL match the element. This PR makes that false, so it now reads: > SQLite, PostgreSQL and the in-memory driver match the element of a multi-valued relation, so an id that is a substring of another stored id (`u1` inside `u10`) does not match it. Please confirm the correction on this PR. If the release that consumes that note ships before this PR lands, the edit should be dropped on rebase. ## Tests (HEAD `10656601`) - `pnpm --filter @objectstack/driver-memory typecheck` passes, and `vitest run` gives **68 files / 1553 tests passed**. Both typecheck programs include the new test file (`--listFiles`). - The new file alone gives **113 passed**. - `pnpm --filter @objectstack/driver-sql typecheck` passes (its program includes the edited test). `sql-driver-17590-json-column-membership.test.ts` gives **22 passed, 2 skipped**; the skips are the live PostgreSQL/MySQL cells, unprovisioned here, NOT MEASURED locally. - `pnpm --filter @objectstack/spec typecheck` passes. `spec build` and then `check:generated` report "All 15 generated artifacts are up to date". **Reverse verification.** The code change was committed first. Then `memory-driver.ts` and `memory-analytics.ts` were checked out at BASE under an EXIT/INT/TERM restore trap. The landed mutation was verified on disk before the run: `filterContainsTest` count 0, two `escapeRegex(val)` arms back, and both files byte-equal to BASE. The predicted direction was that the cases where per-element substring and membership disagree go red and the agreeing ones stay green. Measured: **69 failed / 44 passed**. The failures were every disagreeing case, on all four faces, plus the executed echo, the relation-lowering input and the fork pin. The `label` controls and the `redwood`, `ab`, `u10` and `'0'` cases stayed green. The restore was proven by blob hashes equal to HEAD (`d376dadb…` / `9de9198b…`), an empty `git diff HEAD` and clean porcelain. **Guard ablation** (`scripts/ablation-replace.mjs`, anchor 1 to 0). Dropping `$not: { $type: 'array' }` turned exactly the nested-array cases red: 12, the json `'u1'`, its complement and `'null'`, on all four faces. The executed echo stayed green, since SQLite is the oracle there. The file was restored to the HEAD blob with an empty `git diff HEAD`. **Lint (narrowed, a measurement).** 1. Population: `eslint.config.mjs` lints `**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}` minus `NEVER_LINTED`. The six changed TS files are all in it; the two changesets are not. 2. `eslint --no-inline-config --format json` over those six files reports 6 files, 0 errors, 0 warnings at `10656601`. 3. The config never enables type-aware linting (no `parserOptions.project`, no typed rules; it says so itself), so this diff cannot move any untouched file's verdict. **Driver conformance ledger**: `node scripts/check-driver-conformance.mjs` reads "50 covered cell(s), 0 in the DEBT ledger, 0 exempt" both at `f6ccca4a` (before) and at `10656601` (after). **Gates**: `dispatch-gates --commands` was re-derived at `10656601`. That gives 84 commands, six more than the dispatch list: `engine-double-contract`, `objectql-double-limit`, `query-options-erasure`, `type-check-coverage`, `type-check-debt` and `where-matcher`. The run also covered the eight roster gates flagged as sharing a directory with these paths. All exit 0 except: - `check-empty-changeset`: exit 1, the deliberate correction above. - `check:dual-build-cjs-loads` and `check:type-check-debt`: exit 3, `PREREQUISITE NOT MET` (they need the whole workspace built). **NOT MEASURED**; CI's Build/Lint jobs own those. - `check:lean-entry-closure`: exit 3 at first. It passed after building objectql. ## Acceptance notes (observed, not filed by this PR) - The other text operators over a stored array are not ruled by the membership section and still differ: `{ owners: { $startsWith: 'u1' } }` gives `d1, d3, d5` on memory (per element) and no rows on SQLite (GLOB over the serialized text). The spec docblock now says so. - objectql `engine.ts`'s delete-probe docblock (the `$contains` pushdown) still says every backend answers a substring SUPERSET. That is stale for `driver-sql` since objectstack-ai#17590 and for memory after this PR. The code stays correct because it narrows exactly through `storedReferenceIncludes`. Comment only, no carrier. - A field `driver-memory` holds no declaration for keeps the substring reading, and mingo still applies that `$regex` per element of an array value. This is the deliberate counterpart of `driver-sql`'s `isJsonColumn` returning false for an unknown table. --- _Generated by [Claude Code](https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
Part of #20802
Clause-②: yes (widening)
The engine half of ruling 5907789183 (letter A). The analytics cube read and the analytics read-scope face are the second half, after #5930 step 3 (#20810);
skills/objectstack-query(#20782) belongs to the skills seat, who is told after this lands. So this PR does not complete the card.What this does
The nested-relation form
{ relation: { field: value } }is served inwhere. It is lowered at the engine's filter seam that #5930 step 2 built (cfa931535). The drivers receive$in/$containsand are not changed (ADR-0053 D-D1 item 5, D4 (b)).whereadmission now has three steps: resolve the placeholders, then lower each nested-relation condition, then run the sharedlowerFilterCondition.ObjectQL.resolveRelateThenLowerWhereholds that order, so a verb cannot resolve without it.resolveWhereTokensandwithResolvedWherebecame async to carry it.lowerRelationConditionsreads the related object with the engine's ownfind:fields: ['id'],limit: RELATION_FILTER_ID_CAP + 1, and the caller's execution context. The ids it returns become{ relation: { $in: ids } }on a single-valued relation. On a multi-valued one they become an$orof one$containsper id, which matches on any member. That is the spec's own any-of spelling, and the SQL family refuses$inover the JSON column. No related record matching gives$in: []or$or: []: FALSE, never an absent predicate.walkConditioninnumber-comparand-declared-type-door.ts.admitRelationCondition), instead of refusing it.mapRelationConditions) twice: once to collect the conditions, and once, after the reads, to replace them in the same order.RELATION_FILTER_ID_CAP= 1000, one named constant, exported from@objectstack/objectql. Past it the filter is refused withINVALID_FILTER/ 400. The refusal names the cap, the related object and the two-step route. The filter is never run over a cut-off list.The accept set that widens. It is the engine's
where(find,findOne,count,aggregate,update,delete) plusjudgeFilter, and the REST query doors that reachfindData(POST /api/v1/data/:object/queryand thefilter/$filterspellings). Per relation kind:lookup,master_detail,user,tree, single-valued: refused, now served ($in).multiple: true: refused, now served (any member).Nothing served today narrows.
where.What stays refused. Each is refused in the engine's words, before any read:
jsonfield's object comparand, and the provisionedid(unchanged);{};{ 'owner.region': 'NA' }, stillINVALID_FIELD([finding] The FILTER axis has no DOTTED-path verdict —where: { project_id.name: 'x' }rides its head segment past both doors, where SORT refuses the same spelling (#4256) #8371);filterand inhaving. The engine evaluates both itself, and its evaluator has no member test for a stored list. Their words now saywhereserves it.Text.
FilterCondition's docblock item 4 now states the served semantics.QueryFilterexample shows the form again.data-engine.mdxexample that PR fix(objectql)!: a no-operator object beneath a relation, structured-JSON or undeclared id column is refused INVALID_FILTER / 400 on every driver (#20745) #20781 removed comes back, with the cut, the cap and the two-step route.The dotted-path words, made true again (a bounded in-place fix, named here).
filter-comparand-shape.ts) and the query-parameter door's (metadata-protocolprotocol.ts, outside the claim's declared file surface).{ "owner": { "region": VALUE } }), in the same words, and keep the shared denormalise remedy.Measured
On this branch at
56da9b6d50throughPOST /api/v1/data/:object/query. Owneru1is region NA ond1andd3, andd4has no owner. Before, onorigin/mainafter PR #20781, every relation row answeredINVALID_FILTER/ 400 on every driver.where{ owner: { region: 'NA' } }(lookup),boss(master_detail)d1,d3d1,d3d1,d3{ owners: { region: 'NA' } }(multiple lookup)d1,d3d1,d3d1,d3(see the note){ parent: { title: 'a' } }(tree)d2,d3d2,d3d2,d3{ $not: { owner: { region: 'NA' } } }d2,d4d2,d4d2,d4{ $or: [{ owner: { region: 'EU' } }, { title: 'a' }] }d1,d2d1,d2d1,d2{ owner: { region: 'APAC' } }(no match)INVALID_FILTER, the cap words$containsover a stored array by substring per element. That is the gapFILTER_OPERATORS'$containsdocblock records for that driver. So with idsu1andu10, a multi-valued condition meaningu1also matches the row holding['u10']: measured memoryd1,d3,d5, against SQLd1,d3. Single-valued relations are exact everywhere.check:driver-memory-censusrefuses a new test consumer of that driver without a ruling.record.owner.region == 'NA'is refused at compile: "cross-object/nested field path … is not pushdown-able". The policy is dropped to the deny sentinel, so it answers zero rows. The RLS compile seam is untouched, and no async read was added there.Mechanism hypotheses: which held
lowerFilterConditionis pure and synchronous, and the relation step needs the engine. So the step lives in objectql, between token resolution and the shared lowering, and each engine filter position reaches it at most once.whereonfind,findOne,count,aggregate,updateanddelete(the multi and by-predicate paths alike), and the judge.aggregations[i].filterandhaving.REFERENCE_VALUE_TYPESkind, not onlylookup/master_detail.userandtreepoint at a related object the same way, and the [finding] a no-operator object under a lookup, master_detail or json field answers per driver: the declared nested-relation filter returns no rows on memory and a 400 on SQL, and a json object comparand deep-equals on memory and is refused on SQL #20745 seat answer ruled that one class gets one answer.userneedssys_userregistered; where it is not, it is refused loudly ("no object 'sys_user' is registered here").403 PERMISSION_DENIED, the security layer's filter-oracle guardassertReadableQueryFields.INVALID_FILTER; see the open question in the report.$indoes not mean "any member" everywhere;$orof$containsdoes on SQL.$inover a multi-valued lookup is refused on SQL (JSON column) and is any-member on memory.$containsis membership on SQLite and PostgreSQL, and substring-per-element on memory (the note above).$orof$containsis the spec's declared any-of spelling, so it is the lowered form.expand's batch loader bounds nothing: it deliberately forwards no limit. The cap is a new named constant.$and/$orcompose as written.$notover a relation condition takes the shared lowering's NULL-safe negation, so a row with no relation satisfies it. The driver input is pinned equal to the hand-written two-step route's, and$not+ no match gives every row. An empty inner result is FALSE, never "no filter".$noris not in the vocabulary.Tests (all on
56da9b6d50)@objectstack/objectqltest: 345 files / 6786 passed. typecheck exit 0,check:test-typecheckOK.@objectstack/resttest, withOS_TEST_POSTGRES_URLset to a local PostgreSQL 16.13: 237 files / 4706 passed / 35 skipped (MySQL cells and suites with no URL). typecheck exit 0.@objectstack/metadata-protocoltest: 191 files passed, 3 skipped / 2801 passed, 19 skipped. typecheck exit 0.@objectstack/spectest: 578 files / 17066 passed / 1 todo. typecheck exit 0.check:generated: all 15 artifacts up to date (no regeneration needed; docblock only).@objectstack/plugin-security149 files / 3227 passed, 23 skipped.driver-memory65 / 1470 passed.driver-sql201 files passed, 11 skipped / 3254 passed, 188 skipped. The other...@objectstack/objectqlconsumers are declared to CI.packages/objectql/src/engine-nested-relation-lowering.test.ts(13 tests, recording driver). It covers:$and/$or/$not/ sugar, pinned equal to the two-step route's driver input;filter/havingrefusals;packages/rest/src/data-nested-object-door.test.ts(rewritten): [finding] a no-operator object under a lookup, master_detail or json field answers per driver: the declared nested-relation filter returns no rows on memory and a 400 on SQL, and a json object comparand deep-equals on memory and is refused on SQL #20745's table turned into rows on SQLite and live PostgreSQL, plus the two-step equivalence, the kept refusals with the route inside the 500-character REST bound, the cap pin (1001 related records refused, 1000 served) and the controls.packages/rest/src/data-nested-relation-permission.test.ts: the permission pin, with the realSecurityPluginon a real engine and SQLite, through the REST door. An unreadable related field is refused 403, never emptied, while a system read can filter by it. A hidden related record matches nothing.engine-nested-object-door.test.tskeeps only what still refuses.query-expression-conformance.test.ts: its nested-form control now pins the served rows, and a new pin checks that both doors' dotted refusals name the same route.protocol-explicit-filter-field-gate.test.ts: its GUARD still proves the name gate never descends.Ablations, each from the committed fix. Each ran through
scripts/ablation-replace.mjsin WRAP mode, trap-restored. After each mutation objectql was rebuilt, andablation-dist-preflightfound the marker in 4 built files.if (sites.length === 0) return where;became an unconditionalreturn where(marker__ablated_20802_lowering__). Blob9237c3dfc995→f9994599356f. Predicted red, observed red:...(execCtx ? { context: execCtx } : {}),becameisSystem: true(marker__ablated_20802_caller__). Blob →b0c1748c5b70. Predicted red, observed red:d4.execCtxunused, and its DTS step failed on TS6133; the JS carried the marker. It was re-run type-clean, and those numbers are the ones above.)9237c3dfc995, andgit diff HEADis empty. After a rebuild, the--absentpreflight found both markers absent from all 14 built files, and the whole tree was clean. The pins were green again: 245 passed; 15 passed / 6 skipped.Gates
node scripts/pm/dispatch-gates.mjs --commandsat56da9b6d50(fresh, not stale) derived 120 commands, and all 120 were run.--ranwith each exit code recorded: 120 derived, 120 run, 0 NOT-MEASURED, 0 UNRUN, all exit 0.Three gates refused first with
PREREQUISITE NOT MET(exit 3), and none of the three is counted as a failure:check:skill-examplescheck:dual-build-cjs-loadscheck:type-check-debtThey were re-run green after
turbo run build --filter='./packages/*' --filter='./packages/*/*'.Lint, narrowed and proven:
eslint --no-inline-config --format jsonover the 15 changed.tsfiles gave 15 files, 0 errors, 0 warnings.isPathIgnoredis false for all 15.eslint.config.mjs's**/*.{ts,…}block.parserOptions.project/projectServiceare unset for every file. Type-aware linting is off, so this diff cannot move a verdict on an untouched file.Changesets
.changeset/20802-nested-relation-filter-served.md:@objectstack/objectqlminor,Clause-②: yes (widening). It says it supersedes the relation-field paragraph of the pending20745-nested-object-doorentry, and it states the in-memory$containssubstring caveat..changeset/20802-nested-relation-prose.md:@objectstack/specpatch(shipped JSDoc)..changeset/20802-dotted-relation-route.md:@objectstack/metadata-protocolpatch(refusal words).scripts/adr-anchors/packages__objectql__src__relation-filter-lowering.ts.json→ ADR-0053.Acceptance notes
where/ preview door, the read scope) and the memory cube face's door, with the F5 / F11 output vocabulary #20810 first). Until then the analytics face still flattens the nested form to cube members, and the read scope still refuses it.PERMISSION_DENIED/ 403, the one existing check, reused. It is not the ruling's parentheticalINVALID_FILTER, and it is raised to the PM as an open question.@objectstack/lint's list-view dotted-path hint still says "Filter on a column of … itself". That is an instruction rather than a claim this change made false. Carrier: none; not changed here.$contains. It is reported, not fixed here (no driver file).Generated by Claude Code