diff --git a/README.md b/README.md index 613753b..2f978a1 100644 --- a/README.md +++ b/README.md @@ -48,11 +48,11 @@ graph TD style schema fill:#f9a825,stroke:#333,stroke-width:3px ``` -`ContentDocument` (the semantic pivot) is a discriminated union of five kinds: `wordprocessing` (docx/odt sections of paragraphs/runs/tables/images), `presentation` (pptx/odp slides of shapes), `spreadsheet` (xlsx/ods sheets of cells, columns, rows, print settings), `drawing` (odg pages of shapes plus vector primitives — rect/ellipse/line/path), and `formula` (an equation carrying its own MathML node tree plus StarMath source when the producing format had one, extended with the two-layer math model: an optional verbatim-LaTeX `presentation` authoritative for rendering, an optional semantic `content: MathExpression` tree authoritative for computation, and provenance — neither layer stored derived from the other, so editing one never silently mutates the other). `ContentEmbeddedObjectSchema` lets any of the five embed another whole `ContentDocument`. Every paragraph/run/image/table/shape/vector/spreadsheet-cell leaf also carries its own canonical `headingLevel`-or-position fields directly: a `ContentParagraph`'s optional `headingLevel` (1 = the outermost heading, independent of the round-trip-only `styleId`), and every such leaf's optional `frames: LayoutFrame[]` — that node's own rendered page position(s) (`pageIndex` plus PDF user-space `xPt`/`yPt`/`widthPt`/`heightPt`), fused directly onto the content tree once a layout pass has run. `DocumentPackage` is the single hierarchical artefact — structure, layout, and content fused in one tree (see [The package tree](#the-package-tree)): the root carries `kind`, `metadata`, the optional document-level `symbolTable` and rendered `pages`, the optional package-level `styles`/`definitions`/`layers`/`attachments`/`destinations` tables (see [Definitions tables and styles](#definitions-tables-and-styles)), and `children` — one group per top-level container with the content tree grouped inside it, where a group's node may also be one of the six fidelity construct descriptors (see [Fidelity constructs](#fidelity-constructs)); the schema does not keep populated `frames` fields and `pages` in sync or detect staleness, and does not check that a tree's `style` refs name table entries (both are producer responsibilities, exactly as the frames/pages pairing always was). Every one of the five kinds also accepts an optional document-level `symbolTable` — the math curation layer mapping each written symbol glyph (within a scope) to its id, quantity kind, preferred unit, and definition source, alongside the unit registry (SI dimension-exponent vectors, exact rational conversions, per-unit-system normalisation contexts) that the `qty` nodes of lowered formulas resolve against. +`ContentDocument` (the semantic pivot) is a discriminated union of five kinds: `wordprocessing` (docx/odt sections of paragraphs/runs/tables/images), `presentation` (pptx/odp slides of shapes), `spreadsheet` (xlsx/ods sheets of cells, columns, rows, print settings), `drawing` (odg pages of shapes plus vector primitives — rect/ellipse/line/path), and `formula` (an equation carrying its own MathML node tree plus StarMath source when the producing format had one, extended with the two-layer math model: an optional verbatim-LaTeX `presentation` authoritative for rendering, an optional semantic `content: MathExpression` tree authoritative for computation, and provenance — neither layer stored derived from the other, so editing one never silently mutates the other). `ContentEmbeddedObjectSchema` lets any of the five embed another whole `ContentDocument`. Every paragraph/run/image/table/shape/vector/spreadsheet-cell leaf also carries its own canonical `headingLevel`-or-position fields directly: a `ContentParagraph`'s optional `headingLevel` (1 = the outermost heading, independent of the round-trip-only `styleId`), and every such leaf's optional `frames: LayoutFrame[]` — that node's own rendered page position(s) (`pageIndex` plus PDF user-space `xPt`/`yPt`/`widthPt`/`heightPt`), fused directly onto the content tree once a layout pass has run. `DocumentPackage` is the single hierarchical artefact — structure, layout, and content fused in one tree (see [The package tree](#the-package-tree)): the root carries `kind`, `metadata`, the optional document-level `symbolTable` and rendered `pages`, the optional package-level `styles`/`definitions`/`layers`/`attachments`/`destinations` tables (see [Definitions tables and styles](#definitions-tables-and-styles)), and `children` — one group per top-level container with the content tree grouped inside it, where a group's node may also be one of the six fidelity construct descriptors, which the flat form carries instead as a matched `constructStart`/`constructEnd` block pair (see [Fidelity constructs](#fidelity-constructs)); the schema does not keep populated `frames` fields and `pages` in sync or detect staleness, and does not check that a tree's `style` refs name table entries (both are producer responsibilities, exactly as the frames/pages pairing always was). Every one of the five kinds also accepts an optional document-level `symbolTable` — the math curation layer mapping each written symbol glyph (within a scope) to its id, quantity kind, preferred unit, and definition source, alongside the unit registry (SI dimension-exponent vectors, exact rational conversions, per-unit-system normalisation contexts) that the `qty` nodes of lowered formulas resolve against. The `LayoutDocument` family (pages of positioned `LayoutItem`s — `text`/`image`/`rect`/`line`/`ellipse`/`path`/`link` in PDF user-space coordinates) no longer lives here: 4.0.0 demoted it to a pdf-codec-private model ([pdf-codec#65](https://github.com/ExaDev/pdf-codec/issues/65)), where the only codec that ever read or wrote it owns it outright. `documentFromJson` recognises old layout-document `$schema` URIs and throws a tombstone pointing at pdf-codec rather than failing as if the value were unrelated. Dependents stay on document-schema.js 3.x via semver until their own majors, so the demotion is not a cascade-breaker. -The package contains only [Zod](https://zod.dev) schemas, their inferred types, trivial schema-attached helpers (hex-colour conversion, recursive structural type guards, the style-resolution helpers of `src/definitions.ts`), and one small structural interface (`ContentCodec`, see [Codecs](#codecs)). No XML, ZIP, PDF, or binary handling; the sole dependency is `zod`. +The package contains only [Zod](https://zod.dev) schemas, their inferred types, trivial schema-attached helpers (hex-colour conversion, recursive structural type guards, the style-resolution helpers of `src/definitions.ts`, the construct-marker balance check of `src/content.ts`), and one small structural interface (`ContentCodec`, see [Codecs](#codecs)). No XML, ZIP, PDF, or binary handling; the sole dependency is `zod`. Two format-agnostic helpers live here because they operate on the content model itself: cell-addressing utilities in `src/a1.ts` (0-based row/column indices, row-first order matching `ContentSheetCell`'s `{row, column}`) and the `FontFace` interface in `src/font-port.ts` (`{family, bold, italic}`). @@ -91,7 +91,7 @@ const laidOut = DocumentPackageSchema.parse({ ...pkg, pages: [{ widthPt: 612, he `DocumentPackage` ([#20](https://github.com/ExaDev/document-schema.js/issues/20)) is the promoted single hierarchical artefact — one tree where 3.x carried `{ formatVersion, content, pages }` with the content flat. The tree's vocabulary is defined in `src/package-node.ts` and was proven first as [document-outline.js](https://github.com/ExaDev/document-outline.js)'s phase-1 `decompose`/`flatten` implementation ([document-outline.js#2](https://github.com/ExaDev/document-outline.js/issues/2)); this package's schemas are that shape's schema-home port, matching it node for node: - **Groups** are `{ node, children }` where `node` embeds either an anchor paragraph (heading groups and list-item groups carry the full `ContentParagraph` — runs, formatting, frames — never a projected text label) or a container descriptor: `{ kind: 'section', pageSize, margins }`, `{ kind: 'slide', size, notes }`, `{ kind: 'sheet', name, cells, columns, rows, printSettings }`, `{ kind: 'drawPage', size }`, each tagged with a `kind` the flat container type does not carry, or a shape group's untagged frame descriptor, or — since 4.1.0 — a **construct descriptor** (see [Fidelity constructs](#fidelity-constructs)). -- **Bare leaves** carry their own `kind` and never `children`. Discrimination is structural on `node`+`children`, not on the presence of a `kind`. +- **Bare leaves** carry their own `kind` and never `children`. Discrimination is structural on `node`+`children`, not on the presence of a `kind`. Every `ContentBlock` kind is a legal leaf except the two construct boundary markers, which are the flat form's encoding of something the tree carries as a group (see [Constructs in the flat form](#constructs-in-the-flat-form)). - **Section groups are mandatory** — one per `ContentSection` — because a section carries pre-layout page geometry (`pageSize`/`margins`) that a rendered `pages` array cannot hold. - **Grouping never crosses container boundaries**: a shape is its own group with its inner blocks grouped inside it (never a slide's paragraphs flattened across its shapes — that is a TOC projection, not a decomposition); a sheet's grid rides on the sheet node with images and embedded documents as children; embedded documents stay intact as one leaf. - **Style refs ride on group wrappers only** — a group may carry `style: string` naming a `styles` table entry; `ContentDocument` nodes carry no ref field, so the flat codec-exchange form is always fully materialised. @@ -108,7 +108,7 @@ The codecs do not change: they keep producing flat `ContentDocument`s (their nat Every format carries constructs the flat content model drops — docx SDTs and field codes, ODF fields and tracked changes, PDF form widgets and link annotations, markdown footnotes and link titles. [#22](https://github.com/ExaDev/document-schema.js/issues/22) replaces those per-repo caveats with one **harmonised semantic vocabulary**: no format-specific node kinds, ever. Four codec inventories ([ooxml.js#65](https://github.com/ExaDev/ooxml.js/issues/65), [odf.js#59](https://github.com/ExaDev/odf.js/issues/59), [markdown-codec#63](https://github.com/ExaDev/markdown-codec/issues/63), [pdf-codec#66](https://github.com/ExaDev/pdf-codec/issues/66)) audited what each format actually carries, and 4.1.0 lands their answer additively ([#24](https://github.com/ExaDev/document-schema.js/issues/24)) — new descriptor kinds and package tables, no change to any existing node shape, so a 4.0.0 tree parses unchanged. -A construct is a group like any other: `{ node: , children }`, where the children are the extent the construct spans. The tree was designed construct-capable from day one for exactly this, which is why the kinds could land in a minor rather than a second structural break. +In the tree, a construct is a group like any other: `{ node: , children }`, where the children are the extent the construct spans. The tree was designed construct-capable from day one for exactly this, which is why the kinds could land in a minor rather than a second structural break. In the flat form — the shape every codec actually reads and writes — the same construct is a matched pair of boundary markers bracketing its extent (see [Constructs in the flat form](#constructs-in-the-flat-form) below). ```ts // A tracked insertion inside a docx content control, and a footnote marker whose body lives in the definitions table. @@ -150,10 +150,39 @@ The six kinds (`src/construct.ts`): Four things bound the vocabulary, and each is a decision rather than an omission: - **Extents are block-scoped.** A construct group wraps the block flow of a section, heading group, shape, or list item. It cannot wrap a sub-sequence of one paragraph's runs, because a run-level extent is not expressible without changing `ContentParagraph`, and 4.1.0 changes no content shape. Run-level constructs keep their existing homes: **an external hyperlink stays on `ContentRun.hyperlink`** — `link` groups are for block-scoped and annotated extents a flat run field cannot express, never a replacement for it — and the inline field, bookmark, and tracked-change cases wait on a run-level mechanism rather than being forced into a wrapper that would split their paragraph. -- **Two group variants, one per block flow.** `SectionConstructGroup` sits in a section's or heading group's flow and admits heading children; `ShapeConstructGroup` sits in a shape's or list item's and does not — exactly the `SectionChild`/`ShapeChild` split that already existed. A construct nests in and around every other group, so a `provenance` wrapper inside a `contentControl` inside a `division` is a legal (and real) docx shape. Constructs are **not** legal as direct children of a slide, sheet, drawing page, or the package root: those hold containers and leaves, not block flow. +- **Two group variants, one per block flow.** `SectionConstructGroup` sits in a section's or heading group's flow and admits heading children; `ShapeConstructGroup` sits in a shape's or list item's and does not — exactly the `SectionChild`/`ShapeChild` split that already existed. A construct nests in and around every other group, so a `provenance` wrapper inside a `contentControl` inside a `division` is a legal (and real) docx shape. Constructs are **not** legal as direct children of a slide, sheet, drawing page, or the package root: those hold containers and leaves, not block flow. The flat form's marker pair follows the same rule by construction: it is a `ContentBlock`, so it can only appear where block flow already runs. - **`division` is first-class, not degraded.** [#24](https://github.com/ExaDev/document-schema.js/issues/24) posed ODF `text:section` as a choice between a new generic kind and degrading to `contentControl` with the specifics in residue. It is first-class, on the odf inventory's own recommendation: `ContentSection` cannot host it (that is page geometry, one `pageSize`/`margins` pair, and it does not nest, while a division nests arbitrarily and usually changes no page geometry at all), and burying a structural container in the form-control vocabulary would make `contentControl` mean two unrelated things. It clears #22's no-format-specific-kinds bar on a real analogue — tagged PDF's `/Sect` and `/Div` are the same construct — not on ODF's say-so. It is spelled `division` rather than `section` because `{ kind: 'section' }` is already the page-geometry container descriptor. - **Residue is not here.** #22's channel 2 — a per-node and package-level `source: { format, xml }` facility for what has no cross-format meaning — spans the whole content model and remains open on #22. Descriptors are closed objects with no escape hatch, deliberately: a descriptor-only residue field would mint exactly the parallel shape the general facility exists to avoid. +### Constructs in the flat form + +The tree has a wrapper node to hang an extent off; the flat `ContentDocument` does not — a section's, shape's, or table cell's content is one block list and nothing else. So the flat encoding of a construct is a **matched pair of boundary markers** bracketing the blocks it spans, added to `ContentBlock` as two new kinds (`src/content.ts`): + +```ts +import type { ContentBlock } from 'document-schema.js'; + +// The flat form of the same construct region the tree example above carries as nested groups: a tracked +// insertion and a footnote marker, both inside one content control. +const blocks: ContentBlock[] = [ + { kind: 'constructStart', descriptor: { kind: 'contentControl', controlType: 'richText', tag: 'ClientBlock', lock: 'container' } }, + { kind: 'constructStart', descriptor: { kind: 'provenance', change: 'insertion', author: 'A. Reviewer', dateIso: '2026-08-18T09:00:00Z' } }, + { kind: 'paragraph', runs: [{ text: 'Inserted sentence.' }] }, + { kind: 'constructEnd' }, + { kind: 'constructStart', descriptor: { kind: 'anchor', anchorType: 'footnote', name: '1', definition: 'n1' } }, + { kind: 'constructEnd' }, + { kind: 'constructEnd' }, +]; +``` + +This is what makes the descriptor vocabulary reachable at all from the shape a codec produces: every codec reads and writes `ContentDocument`, so a construct facility wired only onto the tree is a facility no codec can emit into. `decompose` promotes each matched pair into the construct group the tree already has, and `flatten` emits the pair back. + +- **Pairing is ordinary bracket matching.** A `constructEnd` closes the nearest preceding still-open `constructStart` in the **same block list**, and the blocks between them are the extent. That is the entire mechanism — there is deliberately **no id, name, or other pairing key** on either marker. An id would have to be minted by whichever producer emitted the pair and then reproduced byte-for-byte by `flatten` to satisfy law 1 above, and a construct group carries a descriptor and its children and nothing else, so the id would be a value with no home on the tree side and no deterministic way back. A bare bracket has nothing to reproduce and nothing to get wrong, and bracket matching already generalises to arbitrary nesting depth and to different construct kinds nested inside each other. +- **Matching never straddles a block list.** A pair opened in a section's blocks closes in that same array; a pair opened inside a table cell closes inside that cell. That cell is also where a construct inside a table is expressible in *either* encoding, because decomposition treats a table as one leaf and never descends into it — so a cell's block list stays flat in a tree too, markers and all. +- **An unbalanced list is invalid input, not a shape to repair.** A close with nothing open, or a start still open when the list ends, is malformed: `decompose` throws rather than inventing a boundary. `findConstructMarkerImbalance(blocks)` is the one shared definition of that check — it returns `{ kind: 'unmatchedEnd' | 'unclosedStart', index }` for the first fault or `undefined` when the list balances, and it exists here because a codec emitting pairs, another reading them, and `decompose` promoting them must all agree on exactly one answer, and no schema can express balance (it is a property of a list's sequence, not of any block in it). It is deliberately non-recursive: each block list is its own bracket scope, so a caller walking nested lists calls it once per list. Balance is checked here, not by the schema — `ContentDocumentSchema.parse` still accepts a section whose blocks carry an unmatched marker (pinned by a dedicated test), since a Zod-only refinement expressing balance would validate against a rule the published `content-document.schema.json` fragment cannot express and so would silently diverge from it. +- **Balance is necessary but not sufficient — an extent must not cross a heading-group or list-group scope boundary.** Headings and list items have no delimiter of their own in the flat form; a heading paragraph's scope runs until the next paragraph whose `headingLevel` is shallower than or equal to its own, and a list item's scope runs until nesting shallows back out — exactly the nesting `decompose` infers when it builds `HeadingGroupNode`/`ListGroupNode`. A pair whose extent contains a paragraph that closes a scope open before the pair started gives `decompose` no single correct tree: nesting the construct group inside the closing scope strands that paragraph with no legal parent, and closing the scope at the `constructStart` and hoisting the construct group out silently moves everything after that paragraph out of a scope it belonged to. `findConstructMarkerImbalance` cannot see this — balance is a property of the marker pair alone, not of what sits between the two markers — so this package does not check it either; `decompose` is the sole enforcement point, since only it walks the heading/list nesting needed to detect a crossing, and it must reject a crossing extent exactly as it already rejects an unbalanced one, rather than silently picking between the two divergent trees. A producer (`ooxml.js`, `odf.js`, `markdown-codec`, `pdf-codec`) must never open a marker pair inside a heading or list scope that some other block inside the extent goes on to close. +- **A marker carries nothing but its kind and (on the open half) its descriptor.** No `frames` or `sourcePath` — a boundary renders nothing, occupies no space, and has no position — and no `style` ref, since refs are tree-only and a construct group's own ref never resolves onto the construct anyway; it only extends the chain passed to its children, which already carry their own resolved properties. +- **The tree refuses markers at leaf positions.** `PackageBlockLeaf` (`src/package-node.ts`) is `ContentBlock` minus the two marker kinds, and every block-flow child position uses it. A construct is a *group* in the tree; admitting the marker pair there as well would put one fact in two encodings inside one tree, and `decompose(flatten(x)) === x` could never hold for it. Table cells are not an exception to this — a cell's blocks are flat in both encodings, so nothing there crosses the boundary. + Two harmonisations the inventories asked for are also deliberately absent. There is **no `fieldType` enum**: `instruction` is required and verbatim, so nothing is lost without one, and #24 asks for exactly one new vocabulary — `link`'s internal targets — rather than one per kind, with the four inventories' own corpus gate still standing before a field-type member set could be frozen honestly. And a **sheet-scoped named range** (xlsx defined names and tables, ODF `table:named-expressions`) is a definitions-table entry, not an `anchor`: a sheet group's children are its images and embedded documents, never a block flow, so there is no extent for an anchor to wrap — which is the odf inventory's own verdict for the identical construct. ## Definitions tables and styles @@ -212,7 +241,7 @@ node_modules/document-schema.js/schemas/document-package.schema.json node_modules/document-schema.js/schemas/content-document.schema.json ``` -Each file's `$id` is a jsdelivr URL pinned to the exact npm version — immutable and live on publish, and (see [Versioning by `$schema`](#versioning-by-schema)) the version of anything stamped with it. Both files carry the same hand-authored `$defs` block (the same object emitted twice in one generator run, so the copies cannot drift), covering the recursive paragraph/table/embedded-object, MathML, and package-tree node models that Zod's converter cannot express directly; `content-json-schema-defs.ts` holds those fragments, and a regression test compares each fragment that has a real Zod counterpart against a live `z.toJSONSchema()` of that schema so a field changed without updating its fragment fails a test. The one deliberate cross-file `$ref` is the embedded-object cycle back to a whole `ContentDocument`. Fragments downstream of a `z.custom()` node (`ContentBlock`, `ContentTable`/`Cell`/`Row`, `ContentEmbeddedObject(Block)`, the nine package-tree group wrappers, `MathMlNode`/`Element`/`Attribute`, `ContentFormula`, `MathExpression` and its recursive variants) still need hand re-verification — the construct descriptors themselves do not, since each is a plain `z.strictObject` reaching no opaque node and is held to the live comparison against `src/content.ts`/`src/package-node.ts`/`src/mathml.ts`/`src/math.ts` — see below. +Each file's `$id` is a jsdelivr URL pinned to the exact npm version — immutable and live on publish, and (see [Versioning by `$schema`](#versioning-by-schema)) the version of anything stamped with it. Both files carry the same hand-authored `$defs` block (the same object emitted twice in one generator run, so the copies cannot drift), covering the recursive paragraph/table/embedded-object, MathML, and package-tree node models that Zod's converter cannot express directly; `content-json-schema-defs.ts` holds those fragments, and a regression test compares each fragment that has a real Zod counterpart against a live `z.toJSONSchema()` of that schema so a field changed without updating its fragment fails a test. The one deliberate cross-file `$ref` is the embedded-object cycle back to a whole `ContentDocument`. Fragments downstream of a `z.custom()` node (`ContentBlock` and the tree's marker-free `PackageBlockLeaf`, `ContentTable`/`Cell`/`Row`, `ContentEmbeddedObject(Block)`, the nine package-tree group wrappers, `MathMlNode`/`Element`/`Attribute`, `ContentFormula`, `MathExpression` and its recursive variants) still need hand re-verification — the construct descriptors and the two boundary markers do not, since each is a plain `z.object`/`z.strictObject` reaching no opaque node and is held to the live comparison against `src/content.ts`/`src/package-node.ts`/`src/mathml.ts`/`src/math.ts` — see below. ### `z.custom()` vs `z.lazy()` for recursive schemas diff --git a/src/content-json-schema-defs.test.ts b/src/content-json-schema-defs.test.ts index 82b1fee..8f8f988 100644 --- a/src/content-json-schema-defs.test.ts +++ b/src/content-json-schema-defs.test.ts @@ -4,6 +4,8 @@ import { ColorSchema } from './color'; import { ContentBorderSchema, ContentCellBordersSchema, + ContentConstructEndSchema, + ContentConstructStartSchema, ContentImageBlockSchema, ContentListMembershipSchema, ContentPageBreakSchema, @@ -65,7 +67,7 @@ import { } from './package-node'; import { AlignmentSchema } from './style'; -// This is the regression test scripts/generate-json-schemas.mjs's own top comment calls for: the only structural defence that generator has against silently drifting away from src/content.ts/src/color.ts/src/geometry.ts/src/style.ts/src/math.ts/src/package-node.ts/src/definitions.ts, since CONTENT_DEFS (content-json-schema-defs.ts) is transcribed by hand rather than generated. Not every entry in CONTENT_DEFS can be checked this way -- ContentBlock/ContentTable/ContentTableRow/ContentTableCell/ContentEmbeddedObject(Block)/MathMlNode/MathMlElement/MathMlAttribute all sit downstream of one of the genuinely un-representable z.custom() nodes (ContentBlockSchema, ContentEmbeddedObjectSchema, MathMlNodeSchema), the nine package-tree group wrappers sit downstream of the tree's own per-kind group schemas (src/package-node.ts, z.custom over recursive guards, reached only through the hand fragments' own children pointers), and ContentFormula/MathExpression/MathApp/MathSum/MathProd/MathMatrix sit downstream of the fourth opaque node (MathExpressionSchema, reached through ContentFormulaSchema.content for the first and through the grammar's own recursion for the rest) -- see that module's own top comment -- so a bare z.toJSONSchema() call over their real schema counterpart either throws or degrades to `{}` for the recursive/custom part, which is exactly the problem CONTENT_DEFS exists to work around in the first place. What CAN be checked -- because a real, non-recursive, non-custom exported Zod schema exists for it -- is every leaf and near-leaf fragment: Color, Box, LayoutFrame, Alignment, ContentStrokeStyle, ContentBorder, ContentCellBorders, ContentListMembership, ContentRun, ContentParagraph, ContentImageBlock, ContentPageBreak, PageSize, Margins, SectionDescriptor, SlideDescriptor, SheetDescriptor, DrawPageDescriptor, ShapeDescriptor, HeadingParagraph, ListParagraph, the whole construct descriptor vocabulary (ContentControlDescriptor, FieldDescriptor, AnchorDescriptor, LinkTarget, LinkDescriptor, ProvenanceDescriptor, DivisionSource, DivisionDescriptor, and the ConstructDescriptor union over them -- each a plain z.strictObject or a union of them, reaching no opaque node), ContentSheetCell, ContentCellValue, ContentSheetCellComment, ContentSheetColumn, ContentSheetRow, ContentSheetPrintSettings, ContentSheetPrintRange, ContentSheetRepeatRange, ContentSheetImage, ContentStroke, ContentPathPoint, ContentPathSegment, ContentSubpath, ContentVector, StyleParagraphProperties, StyleRunProperties, StyleEntry, DefinitionEntry, ExactRational, DimensionVector, MathPresentation, MathProvenance, MathUncertainty, MathNum, MathQty, MathSym, MathUnparsed, MathSymbolEntry, MathUnit, MathNormalisationContext, SymbolTable. None of these reaches ContentBlockSchema, ContentEmbeddedObjectSchema, MathMlNodeSchema, MathExpressionSchema, or a tree group schema from anywhere in its own field tree, so each can be generated live and compared directly. +// This is the regression test scripts/generate-json-schemas.mjs's own top comment calls for: the only structural defence that generator has against silently drifting away from src/content.ts/src/color.ts/src/geometry.ts/src/style.ts/src/math.ts/src/package-node.ts/src/definitions.ts, since CONTENT_DEFS (content-json-schema-defs.ts) is transcribed by hand rather than generated. Not every entry in CONTENT_DEFS can be checked this way -- ContentBlock/ContentTable/ContentTableRow/ContentTableCell/ContentEmbeddedObject(Block)/MathMlNode/MathMlElement/MathMlAttribute all sit downstream of one of the genuinely un-representable z.custom() nodes (ContentBlockSchema, ContentEmbeddedObjectSchema, MathMlNodeSchema), the nine package-tree group wrappers sit downstream of the tree's own per-kind group schemas (src/package-node.ts, z.custom over recursive guards, reached only through the hand fragments' own children pointers), and ContentFormula/MathExpression/MathApp/MathSum/MathProd/MathMatrix sit downstream of the fourth opaque node (MathExpressionSchema, reached through ContentFormulaSchema.content for the first and through the grammar's own recursion for the rest) -- see that module's own top comment -- so a bare z.toJSONSchema() call over their real schema counterpart either throws or degrades to `{}` for the recursive/custom part, which is exactly the problem CONTENT_DEFS exists to work around in the first place. What CAN be checked -- because a real, non-recursive, non-custom exported Zod schema exists for it -- is every leaf and near-leaf fragment: Color, Box, LayoutFrame, Alignment, ContentStrokeStyle, ContentBorder, ContentCellBorders, ContentListMembership, ContentRun, ContentParagraph, ContentImageBlock, ContentPageBreak, PageSize, Margins, SectionDescriptor, SlideDescriptor, SheetDescriptor, DrawPageDescriptor, ShapeDescriptor, HeadingParagraph, ListParagraph, the whole construct descriptor vocabulary (ContentControlDescriptor, FieldDescriptor, AnchorDescriptor, LinkTarget, LinkDescriptor, ProvenanceDescriptor, DivisionSource, DivisionDescriptor, and the ConstructDescriptor union over them -- each a plain z.strictObject or a union of them, reaching no opaque node), the flat form's two construct boundary markers (ContentConstructStart, whose only non-literal field is that same ConstructDescriptor union, and ContentConstructEnd, whose kind literal is its whole payload), ContentSheetCell, ContentCellValue, ContentSheetCellComment, ContentSheetColumn, ContentSheetRow, ContentSheetPrintSettings, ContentSheetPrintRange, ContentSheetRepeatRange, ContentSheetImage, ContentStroke, ContentPathPoint, ContentPathSegment, ContentSubpath, ContentVector, StyleParagraphProperties, StyleRunProperties, StyleEntry, DefinitionEntry, ExactRational, DimensionVector, MathPresentation, MathProvenance, MathUncertainty, MathNum, MathQty, MathSym, MathUnparsed, MathSymbolEntry, MathUnit, MathNormalisationContext, SymbolTable. None of these reaches ContentBlockSchema, ContentEmbeddedObjectSchema, MathMlNodeSchema, MathExpressionSchema, or a tree group schema from anywhere in its own field tree, so each can be generated live and compared directly. // // Comparison strategy: a bare `z.toJSONSchema(SomeSchema)` call, run in isolation, would INLINE every nested schema it encounters (ColorSchema inside ContentRunSchema, AlignmentSchema inside ContentParagraphSchema, etc.) rather than emit the `{ $ref: '#/$defs/X' }` pointers CONTENT_DEFS itself uses -- because those nested schemas aren't registered anywhere. To reproduce the exact cross-reference shape CONTENT_DEFS hand-authors, this test registers the identical set of real schemas under the identical id strings CONTENT_DEFS uses as its own $defs keys, with a `uri` callback matching the `#/$defs/` convention CONTENT_DEFS was written against -- confirmed empirically (see this file's own construction) to make Zod's registry-based multi-schema generation emit exactly that $ref shape for every registered schema referenced from within another. Each per-schema result still carries its own top-level `$schema`/`$id` (since z.toJSONSchema(registry, ...) treats every registered schema as its own standalone root), which CONTENT_DEFS's own nested fragments never have -- those two keys are stripped before comparison, since they're an artefact of testing each fragment as a registry root rather than a real structural difference. @@ -100,6 +102,8 @@ const REGISTERED_SCHEMAS = { DivisionSource: DivisionSourceSchema, DivisionDescriptor: DivisionDescriptorSchema, ConstructDescriptor: ConstructDescriptorSchema, + ContentConstructStart: ContentConstructStartSchema, + ContentConstructEnd: ContentConstructEndSchema, ContentSheetCell: ContentSheetCellSchema, ContentCellValue: ContentCellValueSchema, ContentSheetCellComment: ContentSheetCellCommentSchema, diff --git a/src/content-json-schema-defs.ts b/src/content-json-schema-defs.ts index 7087f4f..45b6dec 100644 --- a/src/content-json-schema-defs.ts +++ b/src/content-json-schema-defs.ts @@ -5,11 +5,11 @@ import { schemaUriFor } from './schema-io'; // The hand-authored JSON Schema $defs fragments spliced into content-document.schema.json's `override()` callback (scripts/generate-json-schemas.mjs), lifted out into their own src module rather than staying inline in that script. The reason is single-sourcing, not tidiness: this exact object needs to be reachable from two places that cannot share an import graph -- // // 1. scripts/generate-json-schemas.mjs itself, which only ever runs against the freshly-built ../dist/ (it imports every other schema it needs the same way), so it imports CONTENT_DEFS from '../dist/content-json-schema-defs.js', the file tsdown emits for this module (entry: 'src/**/*.ts', one dist file per src file -- see tsdown.config.ts). -// 2. content-json-schema-defs.test.ts (src/, run directly by vitest's "unit" project against source, never against dist), which imports this exact same CONTENT_DEFS value straight from here and asserts it stays byte-for-byte in step with a live z.toJSONSchema() call over each fragment's real exported Zod schema counterpart (ContentParagraphSchema, ContentRunSchema, ContentListMembershipSchema, ContentImageBlockSchema, ContentPageBreakSchema, ColorSchema, BoxSchema, LayoutFrameSchema, PageSizeSchema, MarginsSchema, AlignmentSchema, ContentStrokeStyleSchema, ContentBorderSchema, ContentCellBordersSchema, the package tree's non-recursive descriptors and anchors from src/package-node.ts: SectionDescriptorSchema, SlideDescriptorSchema, SheetDescriptorSchema, DrawPageDescriptorSchema, ShapeDescriptorSchema, HeadingParagraphSchema, ListParagraphSchema, the sheet grid and vector leaves from src/content.ts: ContentSheetCellSchema, ContentCellValueSchema, ContentSheetCellCommentSchema, ContentSheetColumnSchema, ContentSheetRowSchema, ContentSheetPrintSettingsSchema, ContentSheetPrintRangeSchema, ContentSheetRepeatRangeSchema, ContentSheetImageSchema, ContentStrokeSchema, ContentPathPointSchema, ContentPathSegmentSchema, ContentSubpathSchema, ContentVectorSchema, and the definitions facility from src/definitions.ts: StyleParagraphPropertiesSchema, StyleRunPropertiesSchema, StyleEntrySchema, DefinitionEntrySchema, the whole construct descriptor vocabulary from src/construct.ts: ContentControlDescriptorSchema, FieldDescriptorSchema, AnchorDescriptorSchema, LinkTargetSchema, LinkDescriptorSchema, ProvenanceDescriptorSchema, DivisionSourceSchema, DivisionDescriptorSchema, ConstructDescriptorSchema, plus the non-recursive math leaves from src/math.ts: ExactRationalSchema, DimensionVectorSchema, MathPresentationSchema, MathProvenanceSchema, MathUncertaintySchema, MathNumSchema, MathQtySchema, MathSymSchema, MathUnparsedSchema, MathSymbolEntrySchema, MathUnitSchema, MathNormalisationContextSchema, SymbolTableSchema) -- see that test file's own top comment for why this is the only structural defence this generator has against silently drifting away from the schemas it's meant to describe. +// 2. content-json-schema-defs.test.ts (src/, run directly by vitest's "unit" project against source, never against dist), which imports this exact same CONTENT_DEFS value straight from here and asserts it stays byte-for-byte in step with a live z.toJSONSchema() call over each fragment's real exported Zod schema counterpart (ContentParagraphSchema, ContentRunSchema, ContentListMembershipSchema, ContentImageBlockSchema, ContentPageBreakSchema, ColorSchema, BoxSchema, LayoutFrameSchema, PageSizeSchema, MarginsSchema, AlignmentSchema, ContentStrokeStyleSchema, ContentBorderSchema, ContentCellBordersSchema, the package tree's non-recursive descriptors and anchors from src/package-node.ts: SectionDescriptorSchema, SlideDescriptorSchema, SheetDescriptorSchema, DrawPageDescriptorSchema, ShapeDescriptorSchema, HeadingParagraphSchema, ListParagraphSchema, the sheet grid and vector leaves from src/content.ts: ContentSheetCellSchema, ContentCellValueSchema, ContentSheetCellCommentSchema, ContentSheetColumnSchema, ContentSheetRowSchema, ContentSheetPrintSettingsSchema, ContentSheetPrintRangeSchema, ContentSheetRepeatRangeSchema, ContentSheetImageSchema, ContentStrokeSchema, ContentPathPointSchema, ContentPathSegmentSchema, ContentSubpathSchema, ContentVectorSchema, and the definitions facility from src/definitions.ts: StyleParagraphPropertiesSchema, StyleRunPropertiesSchema, StyleEntrySchema, DefinitionEntrySchema, the whole construct descriptor vocabulary from src/construct.ts: ContentControlDescriptorSchema, FieldDescriptorSchema, AnchorDescriptorSchema, LinkTargetSchema, LinkDescriptorSchema, ProvenanceDescriptorSchema, DivisionSourceSchema, DivisionDescriptorSchema, ConstructDescriptorSchema, the flat form's two construct boundary markers from src/content.ts: ContentConstructStartSchema, ContentConstructEndSchema, plus the non-recursive math leaves from src/math.ts: ExactRationalSchema, DimensionVectorSchema, MathPresentationSchema, MathProvenanceSchema, MathUncertaintySchema, MathNumSchema, MathQtySchema, MathSymSchema, MathUnparsedSchema, MathSymbolEntrySchema, MathUnitSchema, MathNormalisationContextSchema, SymbolTableSchema) -- see that test file's own top comment for why this is the only structural defence this generator has against silently drifting away from the schemas it's meant to describe. // // If CONTENT_DEFS stayed inline in the .mjs script, only path 1 above would work: the script imports Zod schemas exclusively from '../dist/index.js' (a build artefact that may not exist, and per eslint.config.ts/tsconfig.json is deliberately excluded from both linting and typechecking, matching test/smoke.test.mjs's own precedent) -- a test that has to import through that path would only ever run after a build, which `pnpm test` (the "unit" vitest project, run standalone in CI's own "test" job, with no build step beforehand) never guarantees. Living here instead, this is an ordinary, fully typechecked and linted src module like any other -- CONTENT_DEFS just happens to be consumed by a script as well as by the package's own test suite. // -// The fragments below still cover exactly what scripts/generate-json-schemas.mjs's own top-of-file comment already explains: ContentBlockSchema, ContentEmbeddedObjectSchema, MathMlNodeSchema, and MathExpressionSchema are z.custom() predicates z.toJSONSchema() cannot introspect at all (recursion the pinned Zod version's z.lazy() can't express -- see src/content.ts's isContentBlock/isContentEmbeddedObject, src/mathml.ts's isMathMlNode, and src/math.ts's isMathExpression), so every schema reachable only through one of those four is transcribed by hand here, field-for-field, from the real Zod object definitions. The package tree added its own opaque set in the 4.0.0 major: DocumentPackageSchema's children reach the tree's per-kind group schemas (src/package-node.ts, all z.custom over recursive guards), so the whole PackageNode vocabulary -- container descriptors, anchor paragraphs, the nine group wrappers (the seven of 4.0.0 plus 4.1.0's two construct groups), and the sheet-image/vector leaves -- is transcribed here too, and the generator splices CONTENT_DEFS into document-package.schema.json as well as content-document.schema.json so both files resolve their local #/$defs pointers without depending on each other's file layout (the one deliberate cross-file ref stays $defs.ContentEmbeddedObject(Block)'s document pointer, CONTENT_DOCUMENT_URI). Three further schemas are transcribed despite being real z.objects themselves: ContentFormulaSchema (its mathml/content fields reach the opaque MathMlNodeSchema/MathExpressionSchema nodes, exactly like ContentTableCellSchema's blocks), SymbolTableSchema (transcribed so each ContentDocument arm's symbolTable field is one named $ref rather than five inlined copies of the whole unit-registry subtree), and now StyleEntrySchema/DefinitionEntrySchema (same five-copies reason for the package arms' styles/definitions fields) -- the generator's override() replaces each with a $ref to its fragment here. Anything transcribed here that DOES have a real, non-custom, exported Zod schema counterpart is exactly what content-json-schema-defs.test.ts holds to a live z.toJSONSchema() comparison -- which is every construct descriptor fragment, since a descriptor is a plain z.strictObject reaching no opaque node; re-verify the rest (ContentTableCell/ContentTableRow/ContentTable, ContentEmbeddedObject(Block), the nine group wrappers, MathMlElement/MathMlNode, MathApp/MathSum/MathProd/MathMatrix/MathExpression, ContentFormula) against src/content.ts/src/package-node.ts/src/mathml.ts/src/math.ts by hand whenever those files' field shapes change, exactly as before. +// The fragments below still cover exactly what scripts/generate-json-schemas.mjs's own top-of-file comment already explains: ContentBlockSchema, ContentEmbeddedObjectSchema, MathMlNodeSchema, and MathExpressionSchema are z.custom() predicates z.toJSONSchema() cannot introspect at all (recursion the pinned Zod version's z.lazy() can't express -- see src/content.ts's isContentBlock/isContentEmbeddedObject, src/mathml.ts's isMathMlNode, and src/math.ts's isMathExpression), so every schema reachable only through one of those four is transcribed by hand here, field-for-field, from the real Zod object definitions. The package tree added its own opaque set in the 4.0.0 major: DocumentPackageSchema's children reach the tree's per-kind group schemas (src/package-node.ts, all z.custom over recursive guards), so the whole PackageNode vocabulary -- container descriptors, anchor paragraphs, the nine group wrappers (the seven of 4.0.0 plus 4.1.0's two construct groups), and the sheet-image/vector leaves -- is transcribed here too, and the generator splices CONTENT_DEFS into document-package.schema.json as well as content-document.schema.json so both files resolve their local #/$defs pointers without depending on each other's file layout (the one deliberate cross-file ref stays $defs.ContentEmbeddedObject(Block)'s document pointer, CONTENT_DOCUMENT_URI). Three further schemas are transcribed despite being real z.objects themselves: ContentFormulaSchema (its mathml/content fields reach the opaque MathMlNodeSchema/MathExpressionSchema nodes, exactly like ContentTableCellSchema's blocks), SymbolTableSchema (transcribed so each ContentDocument arm's symbolTable field is one named $ref rather than five inlined copies of the whole unit-registry subtree), and now StyleEntrySchema/DefinitionEntrySchema (same five-copies reason for the package arms' styles/definitions fields) -- the generator's override() replaces each with a $ref to its fragment here. Anything transcribed here that DOES have a real, non-custom, exported Zod schema counterpart is exactly what content-json-schema-defs.test.ts holds to a live z.toJSONSchema() comparison -- which is every construct descriptor fragment, since a descriptor is a plain z.strictObject reaching no opaque node; re-verify the rest (ContentTableCell/ContentTableRow/ContentTable, ContentEmbeddedObject(Block), the two block unions -- ContentBlock and the tree's marker-free PackageBlockLeaf -- the nine group wrappers, MathMlElement/MathMlNode, MathApp/MathSum/MathProd/MathMatrix/MathExpression, ContentFormula) against src/content.ts/src/package-node.ts/src/mathml.ts/src/math.ts by hand whenever those files' field shapes change, exactly as before. type JsonSchema = z.core.JSONSchema.JSONSchema; @@ -231,8 +231,38 @@ export const CONTENT_DEFS: Record = { required: ['kind', 'objectKind', 'document', 'frame'], additionalProperties: false, }, - // ContentBlock itself (src/content.ts): `ContentParagraph | ContentTable | ContentImageBlock | ContentPageBreak | ContentEmbeddedObjectBlock`, in that exact declared order. + // The flat form's two construct boundary markers (src/content.ts): a matched pair bracketing the extent a construct spans, which is how a codec emits construct data into the one shape it actually produces. Both are real z.objects reaching no opaque node, so both are held to the live comparison by content-json-schema-defs.test.ts. Neither carries frames, sourcePath, or a style ref -- see the schemas' own comments for why a boundary has none of those facts to state. + ContentConstructStart: { + type: 'object', + properties: { + kind: { type: 'string', const: 'constructStart' }, + descriptor: { $ref: '#/$defs/ConstructDescriptor' }, + }, + required: ['kind', 'descriptor'], + additionalProperties: false, + }, + ContentConstructEnd: { + type: 'object', + properties: { + kind: { type: 'string', const: 'constructEnd' }, + }, + required: ['kind'], + additionalProperties: false, + }, + // ContentBlock itself (src/content.ts): `ContentParagraph | ContentTable | ContentImageBlock | ContentPageBreak | ContentEmbeddedObjectBlock | ContentConstructStart | ContentConstructEnd`, in that exact declared order. ContentBlock: { + oneOf: [ + { $ref: '#/$defs/ContentParagraph' }, + { $ref: '#/$defs/ContentTable' }, + { $ref: '#/$defs/ContentImageBlock' }, + { $ref: '#/$defs/ContentPageBreak' }, + { $ref: '#/$defs/ContentEmbeddedObjectBlock' }, + { $ref: '#/$defs/ContentConstructStart' }, + { $ref: '#/$defs/ContentConstructEnd' }, + ], + }, + // The block leaf of the package tree (src/package-node.ts's PackageBlockLeaf): every ContentBlock member except the two boundary markers, which the tree refuses because it carries a construct as a group instead. Its own fragment rather than a reuse of ContentBlock above, so the published schema forbids exactly what the runtime guards forbid -- a tree fragment pointing at ContentBlock would advertise marker leaves as legal to every non-TypeScript consumer while documentFromJson rejected them. A table cell's blocks keep pointing at ContentBlock: a table is one leaf, decomposition never descends into its cells, so a cell's list is flat in both encodings and a construct inside one is a marker pair there too. + PackageBlockLeaf: { oneOf: [ { $ref: '#/$defs/ContentParagraph' }, { $ref: '#/$defs/ContentTable' }, @@ -712,7 +742,7 @@ export const CONTENT_DEFS: Record = { required: ['objectKind', 'document', 'frame'], additionalProperties: false, }, - // The nine group wrappers, hand-verified alone (recursive through their children arrays): `{ node, style?, children }` where children's permitted members are exactly that group kind's own child types (src/package-node.ts's per-kind guards). A wordprocessing section's flow. + // The nine group wrappers, hand-verified alone (recursive through their children arrays): `{ node, style?, children }` where children's permitted members are exactly that group kind's own child types (src/package-node.ts's per-kind guards) -- which is why every block-flow wrapper points at PackageBlockLeaf rather than ContentBlock: a construct is a group at these positions, never a boundary marker. A wordprocessing section's flow. SectionGroup: { type: 'object', properties: { @@ -725,7 +755,7 @@ export const CONTENT_DEFS: Record = { { $ref: '#/$defs/HeadingGroup' }, { $ref: '#/$defs/ListGroup' }, { $ref: '#/$defs/SectionConstructGroup' }, - { $ref: '#/$defs/ContentBlock' }, + { $ref: '#/$defs/PackageBlockLeaf' }, ], }, }, @@ -745,7 +775,7 @@ export const CONTENT_DEFS: Record = { { $ref: '#/$defs/HeadingGroup' }, { $ref: '#/$defs/ListGroup' }, { $ref: '#/$defs/SectionConstructGroup' }, - { $ref: '#/$defs/ContentBlock' }, + { $ref: '#/$defs/PackageBlockLeaf' }, ], }, }, @@ -764,7 +794,7 @@ export const CONTENT_DEFS: Record = { oneOf: [ { $ref: '#/$defs/ListGroup' }, { $ref: '#/$defs/ShapeConstructGroup' }, - { $ref: '#/$defs/ContentBlock' }, + { $ref: '#/$defs/PackageBlockLeaf' }, ], }, }, @@ -794,7 +824,7 @@ export const CONTENT_DEFS: Record = { oneOf: [ { $ref: '#/$defs/ListGroup' }, { $ref: '#/$defs/ShapeConstructGroup' }, - { $ref: '#/$defs/ContentBlock' }, + { $ref: '#/$defs/PackageBlockLeaf' }, ], }, }, @@ -843,7 +873,7 @@ export const CONTENT_DEFS: Record = { { $ref: '#/$defs/HeadingGroup' }, { $ref: '#/$defs/ListGroup' }, { $ref: '#/$defs/SectionConstructGroup' }, - { $ref: '#/$defs/ContentBlock' }, + { $ref: '#/$defs/PackageBlockLeaf' }, ], }, }, @@ -862,7 +892,7 @@ export const CONTENT_DEFS: Record = { oneOf: [ { $ref: '#/$defs/ListGroup' }, { $ref: '#/$defs/ShapeConstructGroup' }, - { $ref: '#/$defs/ContentBlock' }, + { $ref: '#/$defs/PackageBlockLeaf' }, ], }, }, diff --git a/src/content.test.ts b/src/content.test.ts index d2a7d6d..47568ab 100644 --- a/src/content.test.ts +++ b/src/content.test.ts @@ -3,6 +3,8 @@ import { COLOR_BLACK } from './color'; import { type ContentBlock, ContentBlockSchema, + ContentConstructEndSchema, + ContentConstructStartSchema, type ContentDocument, ContentDocumentSchema, type ContentEmbeddedObject, @@ -15,8 +17,12 @@ import { ContentSheetRowSchema, type ContentTable, clampHeadingLevel, + findConstructMarkerImbalance, isContentBlock, + isContentConstructEnd, + isContentConstructStart, } from './content'; +import type { ConstructDescriptor } from './construct'; import { LayoutFrameSchema } from './geometry'; const paragraph: ContentBlock = { @@ -1077,3 +1083,183 @@ describe('ContentEmbeddedObjectSchema deep recursion', () => { expect(ContentDocumentSchema.safeParse(deeplyMalformed).success).toBe(false); }); }); + +// -- Construct boundary markers (the flat form's encoding of a fidelity construct) -- +// +// Every construct region below is spelled as a real one from the codec inventories the descriptor vocabulary was built from, so these read as the shapes a codec will actually emit rather than as synthetic bracket exercises. + +function constructStart(descriptor: ConstructDescriptor): ContentBlock { + return { kind: 'constructStart', descriptor }; +} + +const constructEnd: ContentBlock = { kind: 'constructEnd' }; + +// A tracked insertion inside a docx content control, with a footnote anchor beside it -- one region nested inside another, which is the case a pairing key would have existed to handle and bracket matching handles for free. +const nestedConstructBlocks: ContentBlock[] = [ + constructStart({ kind: 'contentControl', controlType: 'richText', tag: 'ClientBlock', lock: 'container' }), + { kind: 'paragraph', runs: [{ text: 'Before the tracked change.' }] }, + constructStart({ kind: 'provenance', change: 'insertion', author: 'A. Reviewer', dateIso: '2026-08-18T09:00:00Z' }), + { kind: 'paragraph', runs: [{ text: 'Inserted sentence.' }] }, + constructEnd, + constructStart({ kind: 'anchor', anchorType: 'footnote', name: '1', definition: 'n1' }), + constructEnd, + constructEnd, +]; + +describe('construct boundary markers', () => { + it('accepts an open marker carrying each of the six descriptor kinds', () => { + const descriptors: ConstructDescriptor[] = [ + { kind: 'contentControl', controlType: 'checkbox', checked: true }, + { kind: 'field', instruction: 'PAGE \\* MERGEFORMAT', cachedResult: '3' }, + { kind: 'anchor', anchorType: 'bookmark', name: 'intro' }, + { kind: 'link', target: { kind: 'internal', anchor: 'intro' }, title: 'Back to the introduction' }, + { kind: 'provenance', change: 'deletion', author: 'A. Reviewer' }, + { kind: 'division', name: 'Chapter 1', columnCount: 2, source: { href: 'chapter-1.odt' } }, + ]; + for (const descriptor of descriptors) { + const marker = constructStart(descriptor); + expect(ContentConstructStartSchema.safeParse(marker).success).toBe(true); + expect(isContentConstructStart(marker)).toBe(true); + expect(isContentBlock(marker)).toBe(true); + } + }); + + it('accepts a close marker whose kind is its whole payload', () => { + expect(ContentConstructEndSchema.safeParse(constructEnd).success).toBe(true); + expect(isContentConstructEnd(constructEnd)).toBe(true); + expect(isContentBlock(constructEnd)).toBe(true); + }); + + it('rejects an open marker with no descriptor, a malformed one, or a descriptor kind the vocabulary does not carry', () => { + expect(isContentBlock({ kind: 'constructStart' })).toBe(false); + expect(isContentBlock({ kind: 'constructStart', descriptor: { kind: 'anchor', anchorType: 'footnote' } })).toBe( + false, + ); + expect(isContentBlock({ kind: 'constructStart', descriptor: { kind: 'residue', xml: '' } })).toBe(false); + expect(isContentConstructStart({ kind: 'constructEnd' })).toBe(false); + expect(isContentConstructEnd({ kind: 'constructStart', descriptor: { kind: 'field', instruction: 'PAGE' } })).toBe( + false, + ); + }); + + it('carries no position of its own: a frames array smuggled onto a marker does not survive a parse', () => { + const parsed = ContentConstructEndSchema.parse({ + kind: 'constructEnd', + frames: [{ pageIndex: 0, xPt: 0, yPt: 0, widthPt: 10, heightPt: 10 }], + }); + expect(parsed).toStrictEqual({ kind: 'constructEnd' }); + }); + + it('nests inside a wordprocessing document and deep-equals itself after a JSON round trip', () => { + const original: ContentDocument = { + kind: 'wordprocessing', + metadata: { title: 'Constructs' }, + sections: [ + { + pageSize: { widthPt: 612, heightPt: 792 }, + margins: { topPt: 72, rightPt: 72, bottomPt: 72, leftPt: 72 }, + blocks: nestedConstructBlocks, + }, + ], + }; + const parsed = ContentDocumentSchema.parse(original); + const roundTripped: unknown = JSON.parse(JSON.stringify(parsed)); + expect(ContentDocumentSchema.parse(roundTripped)).toEqual(original); + }); + + it('brackets a region inside a table cell, the only place a construct inside a table is expressible', () => { + const cellTable: ContentBlock = { + kind: 'table', + rows: [ + { + cells: [ + { + blocks: [ + constructStart({ kind: 'field', instruction: 'DOCPROPERTY Title' }), + { kind: 'paragraph', runs: [{ text: 'Constructs' }] }, + constructEnd, + ], + }, + ], + }, + ], + columnWidthsPt: [200], + }; + expect(isContentBlock(cellTable)).toBe(true); + expect(ContentBlockSchema.safeParse(cellTable).success).toBe(true); + }); + + // Pins a deliberate schema-level gap: an unmatched constructEnd is malformed input by the bracket-matching contract above, but ContentDocumentSchema carries no refinement that rejects it. findConstructMarkerImbalance (tested below) is the one place balance is actually checked -- decompose calls it and throws on what this schema accepts. A Zod-only refinement here would validate against a rule the published content-document.schema.json fragment cannot express (JSON Schema has no way to state "these array members pair up"), so adding one would silently diverge from that published face -- the exact guard-versus-published-face misalignment the PackageBlockLeaf JSON Schema fragment exists to avoid on the tree side. This test exists so a future change reintroducing balance as a Zod refinement fails it rather than sliding in unnoticed. + it('parses a section whose blocks carry an unmatched constructEnd -- balance belongs to findConstructMarkerImbalance, not the schema', () => { + const blocks: ContentBlock[] = [ + { kind: 'paragraph', runs: [{ text: 'No open marker precedes this close.' }] }, + constructEnd, + ]; + const unbalanced: ContentDocument = { + kind: 'wordprocessing', + metadata: {}, + sections: [ + { + pageSize: { widthPt: 612, heightPt: 792 }, + margins: { topPt: 72, rightPt: 72, bottomPt: 72, leftPt: 72 }, + blocks, + }, + ], + }; + expect(ContentDocumentSchema.safeParse(unbalanced).success).toBe(true); + expect(findConstructMarkerImbalance(blocks)).toStrictEqual({ kind: 'unmatchedEnd', index: 1 }); + }); +}); + +describe('findConstructMarkerImbalance', () => { + it('finds nothing in a balanced list, however deeply the regions nest', () => { + expect(findConstructMarkerImbalance(nestedConstructBlocks)).toBeUndefined(); + }); + + it('finds nothing in a list with no markers at all, empty or otherwise', () => { + expect(findConstructMarkerImbalance([])).toBeUndefined(); + expect(findConstructMarkerImbalance([paragraph, image, pageBreak])).toBeUndefined(); + }); + + it('finds nothing across two sibling regions that open and close in turn', () => { + expect( + findConstructMarkerImbalance([ + constructStart({ kind: 'anchor', anchorType: 'bookmark', name: 'a' }), + constructEnd, + constructStart({ kind: 'anchor', anchorType: 'bookmark', name: 'b' }), + constructEnd, + ]), + ).toBeUndefined(); + }); + + it('reports the close that had nothing open, at its own index', () => { + expect(findConstructMarkerImbalance([paragraph, constructEnd])).toStrictEqual({ kind: 'unmatchedEnd', index: 1 }); + expect( + findConstructMarkerImbalance([ + constructStart({ kind: 'anchor', anchorType: 'bookmark', name: 'a' }), + constructEnd, + constructEnd, + ]), + ).toStrictEqual({ kind: 'unmatchedEnd', index: 2 }); + }); + + it('reports the outermost still-open start, not the innermost, when the list ends mid-region', () => { + expect( + findConstructMarkerImbalance([ + constructStart({ kind: 'division', name: 'Chapter 1' }), + constructStart({ kind: 'provenance', change: 'insertion' }), + { kind: 'paragraph', runs: [{ text: 'Never closed.' }] }, + constructEnd, + ]), + ).toStrictEqual({ kind: 'unclosedStart', index: 0 }); + }); + + it('reports the earlier fault when a list is unbalanced in both directions', () => { + expect( + findConstructMarkerImbalance([ + constructEnd, + constructStart({ kind: 'anchor', anchorType: 'bookmark', name: 'a' }), + ]), + ).toStrictEqual({ kind: 'unmatchedEnd', index: 0 }); + }); +}); diff --git a/src/content.ts b/src/content.ts index b823643..cb756a3 100644 --- a/src/content.ts +++ b/src/content.ts @@ -1,6 +1,7 @@ import { z } from 'zod'; import { ColorSchema } from './color'; import type { Color } from './color'; +import { ConstructDescriptorSchema } from './construct'; import { BoxSchema, LayoutFrameSchema, MarginsSchema, PageSizeSchema } from './geometry'; import type { Box, LayoutFrame } from './geometry'; import { MathExpressionSchema, MathPresentationSchema, MathProvenanceSchema, SymbolTableSchema } from './math'; @@ -77,6 +78,29 @@ export const ContentPageBreakSchema = z.object({ }); export type ContentPageBreak = z.infer; +// -- Construct boundary markers: the flat form's encoding of a fidelity construct (src/construct.ts) -- +// +// The package tree carries a construct as a group -- `{ node: , children: }` (src/package-node.ts) -- but the flat form has no wrapper to hang an extent off: a section's, shape's, or table cell's content is one block list and nothing else. So the flat encoding of a construct is a matched pair of markers bracketing the blocks it spans, and decompose promotes each pair into the group the tree already has. The pair is what makes the 4.1.0 descriptor vocabulary reachable at all from the only shape a codec ever produces: every codec (ooxml.js, odf.js, markdown-codec, pdf-codec) reads and writes ContentDocument, so a construct facility wired only onto the tree is a facility no codec can emit into. +// +// THE BRACKET-MATCHING CONTRACT, stated once and binding on every producer and consumer: markers pair exactly as balanced parentheses do -- a `constructEnd` closes the nearest preceding still-open `constructStart` in the SAME block list, and the blocks between them are that construct's extent. That is the entire pairing mechanism; there is deliberately no id, name, or other pairing key on either marker. Matching never straddles a block list: a pair opened in a section's blocks closes in that same array, and a pair opened inside a table cell closes inside that cell -- which is also the only way a construct inside a table is expressible in EITHER encoding, since decomposition treats a table as one leaf and never descends into its cells. A block list whose markers do not balance (an end with no open start, or a start still open when the list ends) is malformed input rather than a shape to repair: see findConstructMarkerImbalance below, the one shared definition of that check. +// +// BALANCE IS NECESSARY BUT NOT SUFFICIENT: a marker's extent must also not cross a heading-group or list-group scope boundary. Headings and list items carry no delimiter of their own in the flat form -- a heading paragraph's scope runs until the next paragraph in the same block list whose headingLevel is shallower than or equal to its own, and a list item's scope runs until nesting shallows back out, exactly the nesting decompose infers when it builds HeadingGroupNode/ListGroupNode (src/package-node.ts) from a flat list. A constructStart/constructEnd pair whose extent contains a paragraph that would close a heading or list scope already open when the pair started leaves decompose no single correct tree to build: nesting the construct group inside that still-closing scope strands the closing paragraph with no legal parent for it, while closing the scope at the constructStart and hoisting the construct group out silently moves everything after the closing paragraph out of a scope it belonged to -- two different, equally legal-looking trees from one balanced input, and neither is more correct than the other. findConstructMarkerImbalance below cannot see this: balance is a property of the marker pair alone, not of what sits between the two markers, and this package does not check it either -- unlike balance, detecting a scope crossing means walking the same heading/list nesting decompose already builds while constructing the tree, so decompose (documents.js) is the sole enforcement point. Exactly as it already rejects rather than repairs an unbalanced pair, decompose must reject a scope-crossing extent rather than silently choosing between the two divergent trees it could otherwise produce. A producer emitting a marker pair (ooxml.js, odf.js, markdown-codec, pdf-codec) must never open one inside a heading or list scope that some other block inside the extent then closes. +// +// WHY NO ID ON EITHER MARKER: an id would have to be minted by whichever producer emitted the pair and then reproduced byte-for-byte by flatten to satisfy the encoding pair's own first law, flatten(decompose(x)) === x (src/package.ts). A construct group carries a descriptor and its children and nothing else, so a marker id would be a value with no home on the tree side and no deterministic way back -- whereas a bare bracket has nothing to reproduce and nothing to get wrong. Bracket matching also already generalises to arbitrary nesting depth and to different construct kinds nested inside each other, which is the whole of what a pairing key would have bought. +// +// WHY NO style, sourcePath, OR frames: a marker is a boundary, not content -- it renders nothing, occupies no space, and has no position -- so `frames` and `sourcePath`, which every real block leaf carries, would name facts a boundary does not have. A `style` ref would be worse: refs are a tree-only, table-compression concept (the flat form is always fully materialised, ExaDev/document-schema.js#21), and a construct group's own style ref never resolves onto the construct itself -- a construct group is a wrapper with no anchor of its own, so its ref only extends the chain passed to its children, which are ordinary blocks and paragraphs already carrying their own fully-resolved direct properties by the time they reach a marker. Nothing on either marker beyond the descriptor the open marker names. +export const ContentConstructStartSchema = z.object({ + kind: z.literal('constructStart'), + descriptor: ConstructDescriptorSchema, // the construct this marker opens -- the identical payload a construct group carries as its `node`, so promoting a pair into a group moves this value across untouched +}); +export type ContentConstructStart = z.infer; + +// The close half of the pair: its kind is its entire payload, for the reasons stated above. Which construct it closes is a fact about the sequence it sits in, never a fact stored on the marker. +export const ContentConstructEndSchema = z.object({ + kind: z.literal('constructEnd'), +}); +export type ContentConstructEnd = z.infer; + // ContentTable is mutually recursive with ContentBlock (a cell contains blocks, which may themselves be tables) -- hand-written, mirroring ooxml.js's own XmlElement/isXmlNode pattern, since z.lazy() collapses to `unknown` for recursive children in the pinned Zod version. export interface ContentTableCell { blocks: ContentBlock[]; @@ -123,7 +147,15 @@ export interface ContentEmbeddedObjectBlock extends ContentEmbeddedObject { frames?: LayoutFrame[]; // this embedded object's own rendered position(s), once a layout pass has fused one in -- see FusedNode above } -export type ContentBlock = ContentParagraph | ContentTable | ContentImageBlock | ContentPageBreak | ContentEmbeddedObjectBlock; +// The two boundary markers join the union at the end, so a 4.1.0 block list parses identically here. That is not the whole of this addition's type-level impact, though: a consumer switching exhaustively over the union is one thing it breaks -- the same additive-plus-exhaustive-switch trade the construct descriptor kinds themselves made -- and a second, distinct break rides along with it. The package tree admits every member of this union at a leaf position EXCEPT the two markers (src/package-node.ts's PackageBlockLeaf): a construct is a group there, and one fact carried in two encodings inside one tree would break the encoding pair's laws. Enforcing that exclusion in the types narrows SectionChild/ShapeChild/ListChild/PackageLeaf from ContentBlock down to PackageBlockLeaf (src/package-node.ts), so code that previously assigned an already-typed ContentBlock/ContentBlock[] value into one of those narrower positions -- e.g. `const children: SectionChild[] = blocks` where blocks: ContentBlock[] -- stops compiling on upgrade, independent of whether it ever switches exhaustively over anything. +export type ContentBlock = + | ContentParagraph + | ContentTable + | ContentImageBlock + | ContentPageBreak + | ContentEmbeddedObjectBlock + | ContentConstructStart + | ContentConstructEnd; function isRecord(value: unknown): value is Record { return typeof value === 'object' && value !== null && !Array.isArray(value); @@ -176,6 +208,17 @@ function isContentEmbeddedObjectBlock(value: unknown): value is ContentEmbeddedO return isRecord(value) && value.kind === 'embeddedObject' && isContentEmbeddedObject(value); } +// The two marker guards, exported for the consumers that have to recognise a boundary without parsing the whole block: the package tree's leaf predicates, which reject them (src/package-node.ts), and findConstructMarkerImbalance below, which walks a block list looking for exactly these two kinds. +export function isContentConstructStart(value: unknown): value is ContentConstructStart { + return ( + isRecord(value) && value.kind === 'constructStart' && ConstructDescriptorSchema.safeParse(value.descriptor).success + ); +} + +export function isContentConstructEnd(value: unknown): value is ContentConstructEnd { + return isRecord(value) && value.kind === 'constructEnd'; +} + // Recursive structural guard. Used via z.custom so table cells (and now embedded-object documents) validate without a recursive Zod schema (which collapses to `unknown` under z.lazy in this Zod version). export function isContentBlock(value: unknown): value is ContentBlock { if (!isRecord(value)) { @@ -207,6 +250,12 @@ export function isContentBlock(value: unknown): value is ContentBlock { if (kind === 'embeddedObject') { return isContentEmbeddedObject(value); } + if (kind === 'constructStart') { + return isContentConstructStart(value); + } + if (kind === 'constructEnd') { + return true; + } return false; } @@ -218,6 +267,30 @@ export const ContentEmbeddedObjectSchema = z.custom(isCon // Standalone schema for the ContentBlock 'embeddedObject' variant, matching the sibling per-kind block schemas above (ContentParagraphSchema, ContentImageBlockSchema, ContentPageBreakSchema) even though ContentBlockSchema itself validates every kind, embeddedObject included, through the single custom guard above. export const ContentEmbeddedObjectBlockSchema = z.custom(isContentEmbeddedObjectBlock); +// Where a block list's construct markers stop balancing: an `unmatchedEnd` is a close with no construct open at that point, an `unclosedStart` is an open still standing when the list ended. `index` is the offending block's own position in the list -- the close itself for the first, and for the second the OUTERMOST still-open start, since that is where the unbalanced region begins rather than where the walk happened to notice it. +export type ConstructMarkerImbalance = + | { kind: 'unmatchedEnd'; index: number } + | { kind: 'unclosedStart'; index: number }; + +// The one shared definition of the bracket-matching contract's balance check (see the marker schemas above for the contract itself): returns the first place a block list's markers fail to match, or undefined when they balance. It lives here rather than in each consumer because at least three of them must agree exactly -- every codec that emits a pair, and documents.js's decompose, which promotes each matched pair into a construct group and so must reject a list it cannot promote instead of silently repairing one -- and because no schema can express it: balance is a property of a block list's sequence, not of any block in it, so ContentBlockSchema validating every member says nothing about whether the members pair up. +// +// Deliberately non-recursive. Each block list is its own bracket scope (a table cell's list matches independently of the list containing the table), so a caller walking nested lists calls this once per list -- which is exactly the walk decompose already performs -- rather than this helper duplicating that walk with its own idea of where the nested lists are. +export function findConstructMarkerImbalance(blocks: readonly ContentBlock[]): ConstructMarkerImbalance | undefined { + const openStartIndices: number[] = []; + for (const [index, block] of blocks.entries()) { + if (isContentConstructStart(block)) { + openStartIndices.push(index); + } else if (isContentConstructEnd(block) && openStartIndices.pop() === undefined) { + return { kind: 'unmatchedEnd', index }; + } + } + const outermostUnclosed = openStartIndices[0]; + if (outermostUnclosed !== undefined) { + return { kind: 'unclosedStart', index: outermostUnclosed }; + } + return undefined; +} + // Shared stroke/border style vocabulary -- reused by ContentStrokeSchema (drawing vector primitives, defined further down alongside them) and by ContentTableCellSchema/ContentSheetCellSchema's own per-side border fields immediately below, so a border always carries the same solid/dashed/dotted/double vocabulary regardless of which content leaf it decorates. Absent means 'solid' wherever this is optional. export const ContentStrokeStyleSchema = z.enum(['solid', 'dashed', 'dotted', 'double']); export type ContentStrokeStyle = z.infer; diff --git a/src/package-node.test.ts b/src/package-node.test.ts index a304c69..1df98b2 100644 --- a/src/package-node.test.ts +++ b/src/package-node.test.ts @@ -3,10 +3,12 @@ import type { ContentDocument, ContentEmbeddedObject, ContentFormula, ContentRun import { DrawPageGroupSchema, HeadingGroupSchema, + isPackageBlockLeaf, isPackageGroup, isPackageLeaf, isPackageNode, ListGroupSchema, + PackageBlockLeafSchema, PackageGroupSchema, PackageLeafSchema, PackageNodeSchema, @@ -429,6 +431,92 @@ describe('construct groups reject the positions and shapes they are not legal in }); }); +// The flat form's construct boundary markers (src/content.ts) are legal ContentBlocks and illegal tree leaves: a construct is a group in this encoding, and one fact carried in both encodings inside one tree is what breaks decompose(flatten(x)) === x. These are the tests of that exclusion. +describe('construct boundary markers are not tree leaves', () => { + const openMarker = { kind: 'constructStart', descriptor: { kind: 'anchor', anchorType: 'bookmark', name: 'b1' } }; + const closeMarker = { kind: 'constructEnd' }; + + it('rejects either marker at a block-leaf position, whichever flow it sits in', () => { + for (const marker of [openMarker, closeMarker]) { + expect(isPackageBlockLeaf(marker)).toBe(false); + expect(PackageBlockLeafSchema.safeParse(marker).success).toBe(false); + expect(isPackageLeaf(marker)).toBe(false); + expect(isPackageNode(marker)).toBe(false); + expect( + SectionGroupSchema.safeParse({ node: { kind: 'section', pageSize: PAGE, margins: MARGINS }, children: [marker] }) + .success, + ).toBe(false); + expect( + HeadingGroupSchema.safeParse({ + node: { kind: 'paragraph', headingLevel: 1, runs: [run('H')] }, + children: [marker], + }).success, + ).toBe(false); + expect( + ListGroupSchema.safeParse({ + node: { kind: 'paragraph', list: { level: 0 }, runs: [run('Item')] }, + children: [marker], + }).success, + ).toBe(false); + expect( + ShapeGroupSchema.safeParse({ + node: { + frame: { xPt: 0, yPt: 0, widthPt: 100, heightPt: 50 }, + insetLeftPt: 0, + insetTopPt: 0, + insetRightPt: 0, + insetBottomPt: 0, + }, + children: [marker], + }).success, + ).toBe(false); + } + }); + + it('rejects a marker inside a construct extent too -- a construct group is where the pair would have been promoted to', () => { + expect( + SectionConstructGroupSchema.safeParse({ + node: { kind: 'division', name: 'Chapter1' }, + children: [openMarker, { kind: 'paragraph', runs: [run('Body')] }, closeMarker], + }).success, + ).toBe(false); + }); + + it('still accepts every other block kind as a leaf, so the exclusion is the two markers and nothing else', () => { + for (const leaf of [ + { kind: 'paragraph', runs: [run('Body')] }, + { kind: 'pageBreak' }, + { kind: 'image', format: 'png', base64: 'aGk=', widthPt: 50, heightPt: 50 }, + ]) { + expect(isPackageBlockLeaf(leaf)).toBe(true); + expect(isPackageLeaf(leaf)).toBe(true); + } + }); + + it('accepts a marker pair inside a table cell, the one place the flat encoding survives inside a tree', () => { + const tableLeaf = { + kind: 'table', + rows: [ + { + cells: [ + { + blocks: [openMarker, { kind: 'paragraph', runs: [run('Bookmarked cell')] }, closeMarker], + }, + ], + }, + ], + columnWidthsPt: [200], + }; + expect(isPackageBlockLeaf(tableLeaf)).toBe(true); + expect( + SectionGroupSchema.safeParse({ + node: { kind: 'section', pageSize: PAGE, margins: MARGINS }, + children: [tableLeaf], + }).success, + ).toBe(true); + }); +}); + describe('the package tree rejects near-misses', () => { it('rejects a group wrapper with no children array', () => { const broken = { node: { kind: 'section', pageSize: PAGE, margins: MARGINS } }; diff --git a/src/package-node.ts b/src/package-node.ts index 139ac22..1108c38 100644 --- a/src/package-node.ts +++ b/src/package-node.ts @@ -1,6 +1,5 @@ import { z } from 'zod'; import { - ContentBlockSchema, ContentDrawPageSchema, ContentEmbeddedObjectSchema, ContentFormulaSchema, @@ -12,7 +11,12 @@ import { ContentSheetSchema, ContentSlideSchema, ContentVectorSchema, + isContentBlock, + isContentConstructEnd, + isContentConstructStart, type ContentBlock, + type ContentConstructEnd, + type ContentConstructStart, type ContentEmbeddedObject, type ContentFormula, type ContentSheetImage, @@ -20,7 +24,7 @@ import { } from './content'; import { ConstructDescriptorSchema, type ConstructDescriptor } from './construct'; -// The package tree's node vocabulary (ExaDev/document-schema.js#20's promoted DocumentPackage, as proven by document-outline.js's phase-1 reference implementation -- this module is that shape's schema-home port). Groups are `{ node, children }` where node embeds either an anchor paragraph (heading and list groups carry the full ContentParagraph, runs and formatting and frames included, never a projected text label) or a container descriptor (section / slide / sheet / drawPage, each tagged with a `kind` the flat container type does not carry). Bare leaves carry their own `kind` and never `children` -- discrimination is structural on node+children, because the earlier "anything with kind is a leaf" rule collided with `{ kind: 'slide' }` groups. Grouping never crosses container boundaries: a shape is its own group with its inner blocks grouped inside it, a sheet's grid rides on the sheet node, and an embedded document (the recursive ContentEmbeddedObject arm) stays intact as one leaf. A group may additionally carry `style` -- a string ref into the package's styles table (ExaDev/document-schema.js#21); refs exist only here, never on ContentDocument nodes, so the flat codec-exchange form is always fully materialised. Since 4.1.0 a group's node may also be a construct descriptor (src/construct.ts, ExaDev/document-schema.js#24) -- a contentControl, field, anchor, link, provenance, or division wrapping the block extent it spans -- which is what the tree's construct-capable-from-day-one design was for: the kinds landed additively, so a 4.0.0 tree carrying none of them parses identically under this release. +// The package tree's node vocabulary (ExaDev/document-schema.js#20's promoted DocumentPackage, as proven by document-outline.js's phase-1 reference implementation -- this module is that shape's schema-home port). Groups are `{ node, children }` where node embeds either an anchor paragraph (heading and list groups carry the full ContentParagraph, runs and formatting and frames included, never a projected text label) or a container descriptor (section / slide / sheet / drawPage, each tagged with a `kind` the flat container type does not carry). Bare leaves carry their own `kind` and never `children` -- discrimination is structural on node+children, because the earlier "anything with kind is a leaf" rule collided with `{ kind: 'slide' }` groups. Grouping never crosses container boundaries: a shape is its own group with its inner blocks grouped inside it, a sheet's grid rides on the sheet node, and an embedded document (the recursive ContentEmbeddedObject arm) stays intact as one leaf. A group may additionally carry `style` -- a string ref into the package's styles table (ExaDev/document-schema.js#21); refs exist only here, never on ContentDocument nodes, so the flat codec-exchange form is always fully materialised. Since 4.1.0 a group's node may also be a construct descriptor (src/construct.ts, ExaDev/document-schema.js#24) -- a contentControl, field, anchor, link, provenance, or division wrapping the block extent it spans -- which is what the tree's construct-capable-from-day-one design was for: the kinds landed additively, so a 4.0.0 tree carrying none of them parses identically under this release. A construct is a group HERE and a matched marker pair in the flat form (src/content.ts's constructStart/constructEnd), never both in one tree: the two marker kinds are the one part of the ContentBlock union this vocabulary refuses at a leaf position, see PackageBlockLeaf below. // The descriptors are built from the content schemas themselves by omit+extend rather than re-declared field by field, so a field added to a container schema in a future release rides its descriptor automatically -- the zod-first spelling of the reference implementation's `Omit & { kind: 'section' }` types. Each is strict: the omitted array (the one whose members became the group's children) is rejected, not merely absent, so a raw flat container smuggled in as a descriptor fails validation instead of parsing to a descriptor that silently dropped its content. export const SectionDescriptorSchema = ContentSectionSchema.omit({ blocks: true }) @@ -59,17 +63,24 @@ export const ListParagraphSchema = ContentParagraphSchema.extend({ }); export type ListParagraph = z.infer; -// The leaf payloads of a package tree, across all five document kinds: wordprocessing/presentation/drawing block flow yields ContentBlock leaves, spreadsheets additionally yield sheet-anchored images (ContentSheetImage) and whole embedded documents (ContentEmbeddedObject, which is not itself a ContentBlock -- it has no `kind` discriminator), drawings yield textless vector primitives (ContentVector), and a formula document yields its single ContentFormula. One union, so one guard set serves every kind. -export type PackageLeaf = ContentBlock | ContentSheetImage | ContentEmbeddedObject | ContentVector | ContentFormula; +// The ContentBlock members legal at a tree leaf position: every kind except the two construct boundary markers (src/content.ts). A construct is a GROUP in this encoding and a marker pair in the flat one -- two encodings of one fact -- so admitting a marker here would put both inside one tree and break the encoding pair's own first law outright: decompose promotes a matched pair into a construct group, so flatten(decompose(x)) could never reproduce a tree that already carried a marker leaf. Spelled as an exclusion rather than a hand-listed union so that a ContentBlock member added later is leaf-legal by default, which is right for everything that is not a boundary marker. +// +// This exclusion is also a type-level break beyond the exhaustive-switch case ContentBlock's own union addition already causes on its own (src/content.ts): SectionChild, ShapeChild, ListChild, and PackageLeaf below all resolve their block-leaf member to PackageBlockLeaf rather than ContentBlock, so a value already typed ContentBlock (or ContentBlock[]) no longer assigns into any of them -- e.g. `const children: SectionChild[] = blocks` where blocks: ContentBlock[] stops compiling on upgrade. The narrowing is required correctness, not a side effect to work around: an assignment that fails here is exactly the marker-at-a-leaf case the exclusion exists to catch at compile time, so the fix on the calling side is to prove the array holds no marker (narrow to PackageBlockLeaf[], or accept it directly from wherever it was produced), never to cast past it. +// +// Table cells are the deliberate exception, and they are not one: a table is one leaf, decomposition never descends into it, and a cell's blocks stay flat in both encodings -- so a construct inside a cell is a marker pair there in a tree exactly as it is in a flat document, and nothing about that crosses this boundary. +export type PackageBlockLeaf = Exclude; + +// The leaf payloads of a package tree, across all five document kinds: wordprocessing/presentation/drawing block flow yields block leaves, spreadsheets additionally yield sheet-anchored images (ContentSheetImage) and whole embedded documents (ContentEmbeddedObject, which is not itself a ContentBlock -- it has no `kind` discriminator), drawings yield textless vector primitives (ContentVector), and a formula document yields its single ContentFormula. One union, so one guard set serves every kind. +export type PackageLeaf = PackageBlockLeaf | ContentSheetImage | ContentEmbeddedObject | ContentVector | ContentFormula; // What a section's (or a heading group's) block flow holds: heading groups, list groups, construct groups, and bare block leaves. Headings nest under headings and lists nest inside the open heading scope or under deeper list items; a plain paragraph is a leaf. -export type SectionChild = HeadingGroupNode | ListGroupNode | SectionConstructGroupNode | ContentBlock; +export type SectionChild = HeadingGroupNode | ListGroupNode | SectionConstructGroupNode | PackageBlockLeaf; // What a shape's block flow holds: list groups, construct groups, and bare leaves only. Shapes carry no heading hierarchy of their own -- list.level is the only depth signal a slide or drawing shape's paragraphs actually carry -- so a paragraph with headingLevel but no list membership sits flat as a leaf here. -export type ShapeChild = ListGroupNode | ShapeConstructGroupNode | ContentBlock; +export type ShapeChild = ListGroupNode | ShapeConstructGroupNode | PackageBlockLeaf; // A list group's children: deeper list groups, construct groups, and block leaves. A heading never appears below a list group, because opening a heading resets the list nesting before it opens its own group. The construct group here is the shape-scoped one for exactly that reason: a list item's flow and a shape's flow admit the same children, as ListChild and ShapeChild have always spelled identically. -export type ListChild = ListGroupNode | ShapeConstructGroupNode | ContentBlock; +export type ListChild = ListGroupNode | ShapeConstructGroupNode | PackageBlockLeaf; // What a sheet's children are: its anchored images and its whole embedded documents, in that order (the two live in sibling arrays with no cross-array ordering field, and flatten's type partition reverses this fixed order). Cells are addressable data, never children -- they ride the sheet descriptor. export type SheetChild = ContentSheetImage | ContentEmbeddedObject; @@ -163,21 +174,29 @@ function isLeafChild(schema: z.ZodType, value: unknown): boolean { return schema.safeParse(value).success; } +// The block-leaf guard behind PackageBlockLeaf above, and the reason that exclusion is enforced at runtime rather than left to the types: these predicates are the untrusted-input boundary, so a marker smuggled into a tree by a producer that flattened one encoding into the other has to fail here, where documentFromJson can report it, rather than parse and quietly leave a package no round trip can reproduce. +export function isPackageBlockLeaf(value: unknown): value is PackageBlockLeaf { + if (isContentConstructStart(value) || isContentConstructEnd(value)) return false; + return isContentBlock(value); +} + +export const PackageBlockLeafSchema = z.custom(isPackageBlockLeaf); + function isSectionChild(value: unknown): value is SectionChild { return ( isHeadingGroupNode(value) || isListGroupNode(value) || isSectionConstructGroupNode(value) || - isLeafChild(ContentBlockSchema, value) + isLeafChild(PackageBlockLeafSchema, value) ); } function isShapeChild(value: unknown): value is ShapeChild { - return isListGroupNode(value) || isShapeConstructGroupNode(value) || isLeafChild(ContentBlockSchema, value); + return isListGroupNode(value) || isShapeConstructGroupNode(value) || isLeafChild(PackageBlockLeafSchema, value); } function isListChild(value: unknown): value is ListChild { - return isListGroupNode(value) || isShapeConstructGroupNode(value) || isLeafChild(ContentBlockSchema, value); + return isListGroupNode(value) || isShapeConstructGroupNode(value) || isLeafChild(PackageBlockLeafSchema, value); } function isSheetChild(value: unknown): value is SheetChild { @@ -234,7 +253,7 @@ export function isShapeConstructGroupNode(value: unknown): value is ShapeConstru // Leaf validation delegates to the content model's own exported schemas rather than hand-rolling a second, parallel structural guard per payload -- the shapes are src/content.ts's to own, and a hand copy here would drift the first time a schema field changes. The union's first-match-wins order is safe because no leaf type is a structural subset of a later member that would change the verdict. const packageLeafUnion = z.union([ - ContentBlockSchema, + PackageBlockLeafSchema, ContentSheetImageSchema, ContentEmbeddedObjectSchema, ContentVectorSchema, @@ -263,7 +282,7 @@ export function isPackageNode(value: unknown): value is PackageNode { return isPackageGroup(value) || isPackageLeaf(value); } -// The zod faces of the guards above -- usable wherever a schema value is needed (array element, object property, safeParse of external input). Deliberately z.custom, not z.lazy: z.lazy() collapses the static type of a recursive schema to `unknown` under the pinned zod 4, so the recursion lives in the plain function guards instead (ContentBlockSchema in src/content.ts is the family precedent, OutlineNodeSchema in document-outline.js the direct one). +// The zod faces of the guards above -- usable wherever a schema value is needed (array element, object property, safeParse of external input). Deliberately z.custom, not z.lazy: z.lazy() collapses the static type of a recursive schema to `unknown` under the pinned zod 4, so the recursion lives in the plain function guards instead (ContentBlockSchema in src/content.ts is the family precedent, OutlineNodeSchema in document-outline.js the direct one). PackageBlockLeafSchema is the one face declared further up instead of here, beside its own guard, because the child predicates above consume it. export const SectionGroupSchema = z.custom(isSectionGroupNode); export const SlideGroupSchema = z.custom(isSlideGroupNode); export const SheetGroupSchema = z.custom(isSheetGroupNode); diff --git a/test/smoke.test.mjs b/test/smoke.test.mjs index 09783ca..1b00a60 100644 --- a/test/smoke.test.mjs +++ b/test/smoke.test.mjs @@ -78,12 +78,19 @@ describe('smoke: generated JSON Schema files', () => { // The tree fragments resolve file-locally: both published files carry the same $defs block (the same object emitted twice in one generator run). expect(Object.keys(documentPackage.$defs)).toContain('SectionGroup'); expect(Object.keys(documentPackage.$defs)).toContain('StyleEntry'); - // The recursion itself: a section group's children point back at the shared HeadingGroup/ListGroup/SectionConstructGroup definitions, and those at ContentBlock. + // The recursion itself: a section group's children point back at the shared HeadingGroup/ListGroup/SectionConstructGroup definitions, and those at PackageBlockLeaf -- the block union minus the two construct boundary markers, since a construct is a group at this position and never the flat form's marker pair. expect(documentPackage.$defs.SectionGroup.properties.children.items.oneOf).toEqual([ { $ref: '#/$defs/HeadingGroup' }, { $ref: '#/$defs/ListGroup' }, { $ref: '#/$defs/SectionConstructGroup' }, - { $ref: '#/$defs/ContentBlock' }, + { $ref: '#/$defs/PackageBlockLeaf' }, + ]); + expect(documentPackage.$defs.PackageBlockLeaf.oneOf).toEqual([ + { $ref: '#/$defs/ContentParagraph' }, + { $ref: '#/$defs/ContentTable' }, + { $ref: '#/$defs/ContentImageBlock' }, + { $ref: '#/$defs/ContentPageBreak' }, + { $ref: '#/$defs/ContentEmbeddedObjectBlock' }, ]); // Style entries enforce the ban list by shape: additionalProperties false on entry and both halves, with no frames/sourcePath/styleId field anywhere. expect(documentPackage.$defs.StyleEntry.additionalProperties).toBe(false); @@ -144,7 +151,20 @@ describe('smoke: generated JSON Schema files', () => { ); expect(contentDocument.$defs.ContentParagraph.properties.headingLevel.type).toBe('integer'); expect(contentDocument.$defs.ContentParagraph.properties.frames.items.$ref).toBe('#/$defs/LayoutFrame'); - expect(contentDocument.$defs.ContentBlock.oneOf).toHaveLength(5); + // The flat block union, listed rather than counted: the five content kinds plus the two construct boundary markers that let a codec emit a construct into the only shape it produces. + expect(contentDocument.$defs.ContentBlock.oneOf).toEqual([ + { $ref: '#/$defs/ContentParagraph' }, + { $ref: '#/$defs/ContentTable' }, + { $ref: '#/$defs/ContentImageBlock' }, + { $ref: '#/$defs/ContentPageBreak' }, + { $ref: '#/$defs/ContentEmbeddedObjectBlock' }, + { $ref: '#/$defs/ContentConstructStart' }, + { $ref: '#/$defs/ContentConstructEnd' }, + ]); + expect(contentDocument.$defs.ContentConstructStart.properties.descriptor.$ref).toBe('#/$defs/ConstructDescriptor'); + expect(contentDocument.$defs.ContentConstructStart.required).toEqual(['kind', 'descriptor']); + // The close marker's kind is its whole payload -- no id, no frames, no style ref. + expect(Object.keys(contentDocument.$defs.ContentConstructEnd.properties)).toEqual(['kind']); // The embedded-object cycle is the one deliberate cross-file pointer. expect(contentDocument.$defs.ContentEmbeddedObjectBlock.properties.document.$ref).toBe( `https://cdn.jsdelivr.net/npm/document-schema.js@${packageVersion}/schemas/content-document.schema.json`, diff --git a/test/workers/document-schema.test.ts b/test/workers/document-schema.test.ts index c595eca..0976670 100644 --- a/test/workers/document-schema.test.ts +++ b/test/workers/document-schema.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import { ContentDocumentSchema, DocumentPackageSchema, resolveStyleChain } from '../../src'; +import { ContentDocumentSchema, DocumentPackageSchema, findConstructMarkerImbalance, resolveStyleChain } from '../../src'; // Proves document-schema.js's Zod schemas and helpers parse inside a Cloudflare Workers isolate (workerd, via @cloudflare/vitest-pool-workers) with no Node-only APIs. The package is pure Zod by design -- no node:fs, no Buffer, no process -- and zod is isomorphic, so if any schema (or its zod dependency) touched a Node-only API the workerd isolate would throw rather than these passing. This is the runtime complement to the static node test suite. describe('document-schema.js under the Cloudflare Workers runtime', () => { @@ -89,4 +89,27 @@ describe('document-schema.js under the Cloudflare Workers runtime', () => { expect(parsed.destinations?.ch1).toEqual({ kind: 'destination', pageIndex: 0 }); expect(parsed.children[0]?.children).toHaveLength(1); }); + + it('ContentDocumentSchema parses the flat form of the same construct, and findConstructMarkerImbalance runs inside the isolate', () => { + const blocks = [ + { kind: 'constructStart', descriptor: { kind: 'division', name: 'Chapter1' } }, + { kind: 'paragraph', runs: [{ text: 'Chapter body.' }] }, + { kind: 'constructEnd' }, + ]; + const parsed = ContentDocumentSchema.parse({ + kind: 'wordprocessing', + metadata: {}, + sections: [ + { + pageSize: { widthPt: 612, heightPt: 792 }, + margins: { topPt: 0, rightPt: 0, bottomPt: 0, leftPt: 0 }, + blocks, + }, + ], + }); + const parsedBlocks = parsed.kind === 'wordprocessing' ? (parsed.sections[0]?.blocks ?? []) : []; + expect(parsedBlocks).toHaveLength(3); + expect(findConstructMarkerImbalance(parsedBlocks)).toBeUndefined(); + expect(findConstructMarkerImbalance(parsedBlocks.slice(1))).toStrictEqual({ kind: 'unmatchedEnd', index: 1 }); + }); });