diff --git a/.changeset/18670-project-banned-keys.md b/.changeset/18670-project-banned-keys.md new file mode 100644 index 0000000000..aa0f6239aa --- /dev/null +++ b/.changeset/18670-project-banned-keys.md @@ -0,0 +1,26 @@ +--- +"@objectstack/spec": minor +--- + +**BREAKING (published artifact narrows)** — `packages/spec/json-schema/**` now states the banned-key rule the tracing sampling filter enforces, so a validator reading the published files stops answering PASS on `{ "dialect": "cel" }` at `TraceSamplingConfig.composite[].condition` — the card's own worked instance of a published file saying yes to metadata the runtime refuses (#18670 item 2, the fourth of the ruling's named arms). + +Clause-②: yes (narrowing) + +One named pattern joins the closed list, and only one: + +- **`banned-keys` — "no document may carry any of these keys"**, emitted as `propertyNames` with a `not` over the banned names. `TraceSamplingConfig.composite[].condition` is a structured filter of match criteria that refuses an object carrying `dialect`, because such an object is an expression attempt and this slot's expression arm was retired in 17.5.0. The published file now says so. + +**The rows retired, by name.** `packages/spec/dropped-refinements.baseline.json` goes from 202 entries / 553 sites to **200 entries / 551 sites**: + +| row | before | after | +|:---|:---|:---| +| `system/TraceSamplingConfig` | `sites: ["composite.element.condition"]` | **deleted** — the schema drops nothing now | +| `system/TracingConfig` | `sites: ["sampling.composite.element.condition"]` | **deleted** — the same node, reached through the parent | + +2 sites closed, **0 sites added anywhere**, and the ledger diff is deletions only. Generator census after: 551 dropped across 200 published schemas, **357 projected** — 224 `non-blank-string`, 129 `required-one-of`, 2 `dependent-required`, **2 `banned-keys`** — 9 undecidable. + +**⛔ Not a behaviour change, and no document the runtime accepts becomes refused.** The arm is EXACT rather than approximate: a JSON object's properties are exactly its own enumerable string-keyed ones and `propertyNames` judges exactly those names, so "none of the banned names is an own property" and "no property name is one of the banned names" are one sentence read from two ends. It is presence and never value — a banned key present with a `null` value is present on both sides. The accept set at the slot is **unchanged in both directions**: every document the runtime takes (`{}`, `{ "service": "api" }`, any filter carrying no `dialect` key) the file still takes, and every document the runtime refuses the file now refuses too — a `dialect`-bearing object of any shape, the CEL envelope included, since that arm is retired and nothing here revives it. Across the published tree, **1528 of the 1530 per-schema files are byte-identical**; the two that move gain the ban and lose the matching `x-dropped-refinements` row, and nothing else in either file changes. + +**The list stays CLOSED.** `packages/spec/src/shared/refinement-projection.ts` declares the vocabulary and builds each predicate from its own declaration — the key list is read once and used by both the published keyword and the enforced rule — so the two cannot name different keys. The predicate judges OWN properties and never `key in value`: `in` walks the prototype chain, so a ban on a name `Object.prototype` carries would refuse `{}` itself while `propertyNames` accepts it, and that is a disagreement about a JSON document rather than an edge outside the domain. A ban over an OPEN set of names — every key starting with `$`, which is what `data/filter.zod.ts`'s normalized field condition refuses — is deliberately not this arm: its keys are a finite list, and a list that merely sampled an open set would be wider than the rule, so those sites stay unprojected — and because the detector reads them `undecidable` rather than `dropped`, they carry NO annotation and hold NO ledger row: published yet unratcheted. + + diff --git a/packages/spec/dropped-refinements.baseline.json b/packages/spec/dropped-refinements.baseline.json index 1749dbdee5..4b07ab619f 100644 --- a/packages/spec/dropped-refinements.baseline.json +++ b/packages/spec/dropped-refinements.baseline.json @@ -2,10 +2,10 @@ "description": "Shrink-only ledger of every PUBLISHED JSON Schema that is STILL WIDER than the Zod type it was generated from, because a rule written as `.refine()` reaches the runtime and not the file (#18670). `z.toJSONSchema()` has no arm for a `custom` check: a plain record, the same record with a `.refine()`, and the same record with an ABORTING `.refine()` all project byte-identically (measured on zod 4.4.3, the version packages/spec resolves). So a document one of these files ACCEPTS can still be refused at parse time, and an author -- or an AI -- validating against packages/spec/json-schema/** finds out a release later. Each `sites` path is a position under that schema at which a refinement is dropped; the same paths are written onto the artifact itself as `x-dropped-refinements`. Item 2 closed the first patterns: a refinement DECLARED through the closed list in src/shared/refinement-projection.ts is emitted into the published file, reads `projected` rather than `dropped`, and its row LEAVES this ledger in the same PR -- which is why the ledger shrinks and never grows on a repair. Every refinement outside that closed list stays here, and adding an arm to the list is a public-contract decision, not a refactor. Hand-edited on purpose and with no `gen:` script: a generator would let a new gap be admitted by running a command instead of by a decision, which is the silence this ledger exists to end. Adding, removing or moving a site fails packages/spec/scripts/build-schemas.ts until the line moves with it, and the failure prints the corrected entry in full. ⛔ Do not delete or weaken a refinement to shorten this file -- the runtime rule is correct; it is the projection that is silent, and the remedy is to teach the closed list a NAMED pattern, never to drop the rule.", "measured": { "zod": "4.4.3", - "publishedSchemasWithDroppedRefinements": 202, - "droppedRefinementSites": 553, - "refinementSitesThatDidProject": 367, - "refinementSitesWithNoJsonFormToCompare": 3 + "publishedSchemasWithDroppedRefinements": 200, + "droppedRefinementSites": 551, + "refinementSitesThatDidProject": 357, + "refinementSitesWithNoJsonFormToCompare": 9 }, "entries": { "ai/BlueprintField": { @@ -1044,16 +1044,6 @@ "rateLimit" ] }, - "system/TraceSamplingConfig": { - "sites": [ - "composite.element.condition" - ] - }, - "system/TracingConfig": { - "sites": [ - "sampling.composite.element.condition" - ] - }, "ui/Action": { "sites": [ "in", diff --git a/packages/spec/scripts/lib/refinement-projection.ts b/packages/spec/scripts/lib/refinement-projection.ts index eb994ebb89..0a21fb6f9b 100644 --- a/packages/spec/scripts/lib/refinement-projection.ts +++ b/packages/spec/scripts/lib/refinement-projection.ts @@ -164,6 +164,36 @@ function emitDependentRequired( jsonSchema.allOf = [...allOf, { dependentRequired: emitted }]; } +/** + * `propertyNames` with a `not` over the banned names — the keyword JSON Schema + * has for a rule about NAMES, so nothing is encoded and nothing approximated. + * + * A node that already carries `propertyNames` is conjoined through `allOf` + * rather than overwritten, for the reason `emitRequiredOneOf` is: a record + * emits `propertyNames: { type: 'string' }` of its own, and replacing it would + * trade the rule this arm adds for the key-type rule the node already stated. + * An identical rule already present is left alone rather than duplicated, so + * the arm is idempotent the way `emitNonBlankString` is. + * + * An EMPTY key list emits nothing, and the reason is stronger than "it would + * ban nothing": `enum` is specified as a non-empty array, so `{ not: { enum: + * [] } }` is an INVALID SCHEMA rather than a vacuous one — ajv refuses it with + * "enum must have non-empty array", which would take the whole published file + * down with it instead of leaving a keyword nobody reads. + */ +function emitBannedKeys(jsonSchema: JsonObject, keys: readonly string[]): void { + if (keys.length === 0) return; + const rule = { not: { enum: [...keys] } }; + if (!('propertyNames' in jsonSchema)) { + jsonSchema.propertyNames = rule; + return; + } + if (JSON.stringify(jsonSchema.propertyNames) === JSON.stringify(rule)) return; + const allOf = Array.isArray(jsonSchema.allOf) ? (jsonSchema.allOf as unknown[]) : []; + if (allOf.some((clause) => JSON.stringify(clause) === JSON.stringify({ propertyNames: rule }))) return; + jsonSchema.allOf = [...allOf, { propertyNames: rule }]; +} + /** Write one declared arm's keywords onto one emitted node. */ export function emitProjectableRefinement(jsonSchema: JsonObject, declared: ProjectableRefinement): void { switch (declared.pattern) { @@ -176,6 +206,9 @@ export function emitProjectableRefinement(jsonSchema: JsonObject, declared: Proj case 'dependent-required': emitDependentRequired(jsonSchema, declared.dependencies); return; + case 'banned-keys': + emitBannedKeys(jsonSchema, declared.keys); + return; } } diff --git a/packages/spec/scripts/refinement-projection.test.ts b/packages/spec/scripts/refinement-projection.test.ts index 4cc1399ff9..1c829f054f 100644 --- a/packages/spec/scripts/refinement-projection.test.ts +++ b/packages/spec/scripts/refinement-projection.test.ts @@ -20,6 +20,9 @@ * set and `\S` is its complement, so the pin is the whole argument; JSON * Schema specifies `pattern` as an ECMA-262 regex, which is the same engine * this assertion runs on. + * - `banned-keys` — over the whole key-presence lattice, with a + * present-but-`null` value, and on a name `Object.prototype` carries, which + * is the one JSON shape where "own property" and `in` come apart. * * ## Why an equality pin and not a comment * @@ -44,11 +47,13 @@ import { NON_BLANK_PATTERN, NON_BLANK_STRING, PROJECTABLE_REFINEMENT_PATTERNS, + bannedKeys, dependentRequired, projectableRefinementOf, requiredOneOf, } from '../src/shared/refinement-projection'; import { SSLConfigSchema } from '../src/data/driver-sql.zod'; +import { TraceSamplingConfigSchema } from '../src/system/tracing.zod'; import { emitProjectableRefinement, projectPublishedJsonSchema, @@ -85,6 +90,30 @@ const requiredOneOfSatisfied = (node: Record, doc: Record branch.required.every((key) => Object.prototype.hasOwnProperty.call(doc, key))); }; +/** + * The node's banned-key rule — `propertyNames.not.enum`, wherever the emitter put + * it — evaluated the way a validator would, and refusing to report anything when + * the node carries no such rule, so a projection that stopped emitting fails + * rather than passing vacuously. + * + * Both placements are read because the emitter chooses between them by what the + * node already carries: a record already states `propertyNames: { type: + * 'string' }`, so its ban is conjoined through `allOf`; a bare object has no + * `propertyNames` and takes the rule directly. + */ +const bannedKeysSatisfied = (node: Record, doc: Record): boolean => { + const clauses = [node, ...((node.allOf as Record[] | undefined) ?? [])]; + const banned = clauses + .map((clause) => (clause.propertyNames as { not?: { enum?: string[] } } | undefined)?.not?.enum) + .filter((list): list is string[] => Array.isArray(list)); + if (banned.length === 0) { + throw new Error('the node carries no propertyNames.not.enum — nothing to evaluate'); + } + return banned.every((list) => + Object.keys(doc).every((name) => !list.includes(name)), + ); +}; + /** * ECMA-262 WhiteSpace ∪ LineTerminator, by code point so no control byte is * ever written into this file (`scripts/check-nul-bytes.mjs` is the authority @@ -98,7 +127,7 @@ const BLANK_CODE_POINTS = [ ]; describe('the list of projectable patterns is CLOSED', () => { - it('names exactly the two arms this change landed', () => { + it('names exactly the arms this list has landed, and nothing else', () => { // ⛔ Growing this is a public-contract decision: every arm narrows a // published artifact. A new arm updates this line in the same PR, which is // what makes it a reviewed diff rather than a quiet widening of the @@ -107,6 +136,7 @@ describe('the list of projectable patterns is CLOSED', () => { 'required-one-of', 'non-blank-string', 'dependent-required', + 'banned-keys', ]); }); @@ -462,6 +492,175 @@ describe('dependent-required: one dependency map, read twice', () => { }); }); +describe('banned-keys: one key list, read twice', () => { + it('declares the keys it was given', () => { + const rule = bannedKeys(['dialect']); + expect(projectableRefinementOf(rule)).toEqual({ pattern: 'banned-keys', keys: ['dialect'] }); + }); + + it('emits `propertyNames` with a `not` over the names, when the node states none', () => { + const node: Record = { type: 'object' }; + emitProjectableRefinement(node, { pattern: 'banned-keys', keys: ['a', 'b'] }); + expect(node).toEqual({ type: 'object', propertyNames: { not: { enum: ['a', 'b'] } } }); + }); + + it('conjoins through `allOf` rather than replacing the `propertyNames` a record already states', () => { + // A record emits `propertyNames: { type: 'string' }` of its own. Replacing + // it would trade the key-TYPE rule the node already stated for the key-NAME + // rule this arm adds, which is a narrowing paid for with a widening. + const node: Record = { type: 'object', propertyNames: { type: 'string' } }; + emitProjectableRefinement(node, { pattern: 'banned-keys', keys: ['dialect'] }); + expect(node.propertyNames).toEqual({ type: 'string' }); + expect(node.allOf).toEqual([{ propertyNames: { not: { enum: ['dialect'] } } }]); + }); + + it('⛔ never writes a TOP-LEVEL `anyOf` or disturbs the node’s own shape', () => { + const node: Record = { type: 'object', properties: { a: { type: 'string' } } }; + emitProjectableRefinement(node, { pattern: 'banned-keys', keys: ['b'] }); + expect(node.anyOf).toBeUndefined(); + expect(node.properties).toEqual({ a: { type: 'string' } }); + }); + + it('is idempotent — the same arm twice states one rule, not two', () => { + const node: Record = { type: 'object', propertyNames: { type: 'string' } }; + emitProjectableRefinement(node, { pattern: 'banned-keys', keys: ['dialect'] }); + emitProjectableRefinement(node, { pattern: 'banned-keys', keys: ['dialect'] }); + expect(node.allOf).toEqual([{ propertyNames: { not: { enum: ['dialect'] } } }]); + }); + + it('drops an empty key list \u2014 `enum: []` is an INVALID schema, not a vacuous one', () => { + // ⛔ Not "it would ban nothing": `enum` is specified as a non-empty array, + // so `{ not: { enum: [] } }` fails validator schema-compilation outright + // (ajv: "enum must have non-empty array") and would take the whole + // published file down rather than sit there unread. + const node: Record = { type: 'object' }; + emitProjectableRefinement(node, { pattern: 'banned-keys', keys: [] }); + expect(node).toEqual({ type: 'object' }); + }); + + it('the predicate and the keywords agree over the whole presence lattice', () => { + const rule = bannedKeys(['x', 'y']); + const node = publish(z.record(z.string(), z.unknown()).refine(rule)); + const keys = ['x', 'y', 'z'] as const; + for (let mask = 0; mask < 8; mask += 1) { + const doc: Record = {}; + keys.forEach((key, i) => { + if (mask & (1 << i)) doc[key] = 'v'; + }); + const asJson = JSON.parse(JSON.stringify(doc)) as Record; + expect( + rule(asJson), + `runtime vs keywords disagree for ${JSON.stringify(asJson)}`, + ).toBe(bannedKeysSatisfied(node, asJson)); + } + }); + + it('a banned key present with a `null` value is PRESENT on both sides', () => { + const rule = bannedKeys(['dialect']); + const node = publish(z.record(z.string(), z.unknown()).refine(rule)); + const doc = JSON.parse('{"dialect":null}') as Record; + expect(rule(doc)).toBe(false); + expect(bannedKeysSatisfied(node, doc)).toBe(false); + }); + + it('⛔ judges OWN properties — a name `Object.prototype` carries is not "present" in an empty document', () => { + // The measurement behind the predicate reading `hasOwnProperty` and never + // `key in value`: `in` walks the prototype chain, so a ban spelled with it + // would refuse `{}` itself while `propertyNames` accepts it. That is a + // disagreement about a JSON DOCUMENT, not an edge outside the domain. + const rule = bannedKeys(['toString']); + const node = publish(z.record(z.string(), z.unknown()).refine(rule)); + const empty = JSON.parse('{}') as Record; + expect('toString' in empty).toBe(true); + expect(rule(empty)).toBe(true); + expect(bannedKeysSatisfied(node, empty)).toBe(true); + }); + + it('LIT CONTROL — the same name written INTO the document is refused by both', () => { + const rule = bannedKeys(['toString']); + const node = publish(z.record(z.string(), z.unknown()).refine(rule)); + const doc = JSON.parse('{"toString":"x"}') as Record; + expect(rule(doc)).toBe(false); + expect(bannedKeysSatisfied(node, doc)).toBe(false); + }); +}); + +describe("the LIVE seam: the card's own worked instance stops saying yes", () => { + /** + * The published `TraceSamplingConfig.composite[].condition` node. + * + * ⭐ It is the node ITSELF, not a union arm. #18118 retired this slot's CEL + * expression arm (PR #19084), so the union collapsed to the structured-filter + * record it always had beside it — which is why the ban lands directly on + * `condition` and is still conjoined through `allOf`: a record states its own + * `propertyNames: { type: 'string' }`, and that key-TYPE rule is not the one + * this arm adds. + */ + const conditionNode = (): Record => { + const node = publish(TraceSamplingConfigSchema); + const composite = (node.properties as Record>).composite; + const item = composite.items as Record>>; + return item.properties.condition as unknown as Record; + }; + + /** A `TraceSamplingConfig` that parses, with only `condition` varying. */ + const parses = (condition: unknown): boolean => + TraceSamplingConfigSchema.safeParse({ + type: 'composite', + composite: [{ strategy: 'always_on', condition }], + }).success; + + it('states the ban, and keeps the record shape it always stated', () => { + const node = conditionNode(); + expect(node.type).toBe('object'); + expect(node.propertyNames).toEqual({ type: 'string' }); + expect(node.allOf).toEqual([{ propertyNames: { not: { enum: ['dialect'] } } }]); + }); + + it("the card's own specimen — `{ dialect: 'cel' }` — is refused by BOTH sides now", () => { + const doc = JSON.parse('{"dialect":"cel"}') as Record; + expect(parses(doc)).toBe(false); + expect(bannedKeysSatisfied(conditionNode(), doc)).toBe(false); + }); + + it('⛔ the runtime and the emitted keywords agree on every document in the corpus', () => { + const node = conditionNode(); + const corpus: Array> = [ + {}, + { amount: { $gt: 1 } }, + { service: 'api', attributes: { 'http.route': '/v1/orders' } }, + { dialect: 'cel' }, + { dialect: null }, + // Since #18118 retired the expression arm, a healthy CEL envelope is + // refused at this slot too — so the two sides agree here as well, where + // before the retirement the union's other arm accepted it. + { dialect: 'cel', source: 'record.amount > 10' }, + ]; + for (const doc of corpus) { + const asJson = JSON.parse(JSON.stringify(doc)) as Record; + // Equality, not implication: this arm is exact, so a one-sided pin would + // pass a projection that had stopped narrowing at all. + expect( + bannedKeysSatisfied(node, asJson), + `disagreement on ${JSON.stringify(asJson)}`, + ).toBe(parses(asJson)); + } + }); + + it('LIT CONTROL — a structured filter with no `dialect` is accepted by both', () => { + const doc = JSON.parse('{"amount":{"$gt":10}}') as Record; + expect(parses(doc)).toBe(true); + expect(bannedKeysSatisfied(conditionNode(), doc)).toBe(true); + }); + + it('its ledger row is gone because the site now reads `projected`, naming the arm', () => { + const census = collectDroppedRefinements('system/TraceSamplingConfig', TraceSamplingConfigSchema); + const site = census.projected.find((s) => s.path === 'composite.element.condition'); + expect(site?.declaredPatterns).toEqual(['banned-keys']); + expect(census.dropped).toEqual([]); + }); +}); + describe('the verdict is adjudicated per NODE over every check on it', () => { const nonBlank = (): z.ZodString => z.string().refine(NON_BLANK_STRING, 'non-blank'); diff --git a/packages/spec/src/shared/refinement-projection.ts b/packages/spec/src/shared/refinement-projection.ts index 003c265e0b..ff40d660a0 100644 --- a/packages/spec/src/shared/refinement-projection.ts +++ b/packages/spec/src/shared/refinement-projection.ts @@ -105,13 +105,40 @@ export type ProjectableRefinement = readonly pattern: 'dependent-required'; /** Key ⇒ the keys its presence requires. Read once here, and by the predicate. */ readonly dependencies: Readonly>; - }; + } + /** + * "no document may carry any of these keys" — published as `propertyNames` + * with a `not` over the banned names, the spelling JSON Schema has for a rule + * about NAMES rather than about values. + * + * Exact in the JSON domain: a JSON object's properties are exactly its own + * enumerable string-keyed ones, and `propertyNames` judges exactly those + * names, so "none of the banned names is an own property" and "no property + * name is one of the banned names" are one sentence read from two ends. It is + * PRESENCE and never value — a banned key present with a `null` value is + * present on both sides, the same equality {@link requiredOneOf} rests on. + * + * ⛔ The predicate reads OWN properties and never `key in value`. `in` walks + * the prototype chain, so a ban on a name `Object.prototype` carries — + * `toString`, `constructor`, `valueOf` — would refuse every object including + * `{}`, while `propertyNames` accepts it: `'toString' in JSON.parse('{}')` is + * `true`. That is a disagreement about a JSON DOCUMENT, not an edge outside + * the domain, and an arm that could only approximate its rule does not belong + * in this list. + * + * ⛔ A ban over an open set of names — every key starting with `$`, say — is + * NOT this arm: its keys are a finite list, and a list that merely sampled an + * open set would be wider than the rule. Such a rule stays dropped and + * annotated until the list learns a pattern-shaped arm of its own. + */ + | { readonly pattern: 'banned-keys'; readonly keys: readonly string[] }; /** Every arm's `pattern` tag, for a reader that needs the list itself. */ export const PROJECTABLE_REFINEMENT_PATTERNS = [ 'required-one-of', 'non-blank-string', 'dependent-required', + 'banned-keys', ] as const; /** @@ -224,3 +251,33 @@ export function dependentRequired( }; return declare(rule, declared); } + +/** + * "none of these keys is present", as a `.refine()` predicate that also + * declares itself. + * + * The key list is read once into the declaration and the predicate reads it + * from there, so the published `propertyNames` and the enforced rule cannot + * name different keys — the same construction {@link requiredOneOf} and + * {@link dependentRequired} use, and the reason this arm needs no drift pin + * either. + * + * Spell the slot's own banned keys at the call site: + * + * ```ts + * z.record(z.string(), z.unknown()).refine(bannedKeys(['dialect']), { + * message: 'A structured filter must not carry `dialect`', + * abort: true, + * }) + * ``` + */ +export function bannedKeys( + keys: readonly [K, ...K[]], +): (value: object) => boolean { + const declared: ProjectableRefinement = { pattern: 'banned-keys', keys: Object.freeze([...keys]) }; + const rule = (value: object): boolean => + !(declared as { keys: readonly string[] }).keys.some((key) => + Object.prototype.hasOwnProperty.call(value, key), + ); + return declare(rule, declared); +} diff --git a/packages/spec/src/system/tracing.zod.ts b/packages/spec/src/system/tracing.zod.ts index 43c78d0829..2718370c03 100644 --- a/packages/spec/src/system/tracing.zod.ts +++ b/packages/spec/src/system/tracing.zod.ts @@ -1,6 +1,7 @@ // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. import { z } from 'zod'; +import { bannedKeys } from '../shared/refinement-projection'; /** * Tracing Protocol - Distributed Tracing & Observability @@ -422,7 +423,7 @@ export const TraceSamplingConfigSchema = lazySchema(() => z.object({ condition: z.record(z.string(), z.unknown(), { error: (issue) => samplingConditionExpressionRefusal(issue.input), }) - .refine((value) => !('dialect' in value), { + .refine(bannedKeys(['dialect']), { message: SAMPLING_CONDITION_EXPRESSION_RETIRED, abort: true, })