diff --git a/docs/decisions.log.md b/docs/decisions.log.md index 8e6b082..d3dfedc 100644 --- a/docs/decisions.log.md +++ b/docs/decisions.log.md @@ -2733,3 +2733,85 @@ Architectural decisions go to [`adr/`](adr/) instead. Guarding against `limits::` in a string would need a Rust parser, which costs more than this boundary is worth; the trade is stated in the test rather than left for someone to discover. + +- 2026-09-02 — In the context of SWG-4A-06's level-2 lexer, facing spec + §5.11's contract that a retained token costs at most twelve bytes on + `wasm32`, we decided to **retain a kind and a span and slice the lexeme + from the source on demand**, to achieve a token that cannot silently grow + a heap allocation per lexeme, accepting that every consumer of a token's + text must carry the source alongside it. Prior art, since the technique is + not this task's invention: `rustc_lexer` emits tokens carrying a kind and + a length and leaves the text in the source buffer; `rust-analyzer`'s + `rowan` keeps green-tree tokens interned rather than owned per + occurrence; `logos` generates lexers whose token payload is a `Span` the + caller slices. What none of them supplies is the *bound* — twelve bytes is + §5.11's own derivation from `MAX_TOKENS` and the cockpit's `wasm32` heap, + and the compile-time assertion that enforces it is this task's. Level 1's + `String`-owning `Token` is untouched: it is frozen, it is not the level-2 + representation, and the two now differ on purpose. + +- 2026-09-02 — In the context of the level-1 parser meeting a `swang 2` + header once the build supports level 2, we decided to **refuse it in the + level-1 entry point with `SWG0401`**, to achieve §5.4's "routes to one of + them and never mixes them", accepting that a caller who reaches for the + frozen entry point with level-2 text gets a structural refusal rather than + a routing hint. Without the guard, `Level::new(2)` succeeded and a level-1 + `Program` carried level 2, so the formatter emitted `swang 2` above a + `pattern` block — a document neither level would read back. `SWG0001` was + rejected for it: this build *does* support level 2, and §5.10 forbids one + number carrying a second meaning. The arm cannot fire for a `swang 1` + source, so Law A is untouched by its existence, and the frozen baseline + proves it rather than the claim resting on inspection. + +- 2026-09-02 — In the context of SWG-4A-06 reading exactly one scalar, we + decided to **implement `SWG0505` and `SWG0506` for `ppqn` alone**, to + achieve a level-2 acceptance set that contains no text §6.6 declares + invalid, accepting that two codes belonging to 4A-07's scalar layer appear + one task early. The alternative was `parse("swang 2\n\nscore { ppqn 0 }")` + returning `Ok` — a permissive fallback in the accepted set of a level that + has not frozen, which 4A-07 would then have to *narrow*. Refusing early is + the cheaper mistake to correct: widening an unfrozen level is routine, + and the two checks are four lines scoped to the one word this slice reads. + +- 2026-09-02 — In the context of SWG-4A-06's depth accounting, facing a + falsification probe that **survived**, we decided to **have the parse + spend a budget its caller owns**, to achieve a wiring that can be + falsified today rather than when 4A-08 makes nesting reachable, accepting + a crate-internal signature that hands the budget in. Deleting + `enter_block` from the level-2 parser was caught by nothing at all: the + minimal score has one block, so no end-to-end breach can witness the + counter. The witness now reads it after a refusal *inside* the block, + where a depth still held is proof of entry — a balanced parse returns to + zero whether the block was entered or not, and would have passed either + way. This is not a side channel: there is no second entry that can succeed + without spending the same budget, and `Level2ResourceLimits` still has + private fields and `declared()` as its only production constructor. + + +- 2026-09-02 — In the context of `ast::v1::Level::new` being pinned to level + 1 while `LANGUAGE_LEVEL` rose to 2, we decided to **stop the level-1 AST's + refusal from quoting the build's supported range**, to achieve a public + error that describes the constraint the code actually enforces, accepting + that two neighbouring types now word "unsupported level" differently on + purpose. The message interpolated `LANGUAGE_LEVEL`, so refusing level 2 + read `language level 2 is not supported (1..=2)` — a sentence that lists + the level it is refusing, and sends the reader hunting for a bug anywhere + but where it is. The two doc comments carried the same stale claim. This + is the same error the constructor bug was, one layer out: the level-1 tree + asking the *build* what it may spell. It may spell level 1 (§5.4), and the + header's own `SWG0001` remains the place where the build-wide range is + named, because there the range is the subject. + +- 2026-09-03 — In the context of `syntax/document.rs` documenting a + delegation it did not implement, we decided to **make `parse_document` + literally `parse_document_with_source_map` with the map dropped**, to + achieve one level decision per build rather than two kept in step by hand, + accepting that a caller wanting no map still pays for one — the price + `v1::parse` already pays a layer down. The module held two `match level` + blocks while its doc said the two "cannot drift"; the property was real + but maintained by discipline. Reintroducing the duplicate is invisible to + behaviour — 287 tests agreed with it — because a fresh copy of a router + does agree, and agrees right up until someone edits one copy. So the + witness is lexical: the level is read once in that file. `v2::parse_exact` + lost its only caller and is deleted rather than kept for symmetry, since a + `pub(crate)` wrapper with nobody behind it is not an API. diff --git a/docs/swang/foundation-backlog.md b/docs/swang/foundation-backlog.md index 15335c7..7700bad 100644 --- a/docs/swang/foundation-backlog.md +++ b/docs/swang/foundation-backlog.md @@ -117,7 +117,7 @@ recorded in `decisions.log.md` if reversed. | SWG-4A-03 | Writer: transport and master timeline | code | 4A-01 | | SWG-4A-04 | Writer: tracks, voices, groups, atoms | code | 4A-03 | | SWG-4A-05 | Writer: techniques, positions, evidence, losses *(done)* | code | 4A-04, CORE-01 | -| SWG-4A-06 | Parser skeleton and level/root dispatch | code | 4A-01, INF-04, INF-06 | +| SWG-4A-06 | Parser skeleton and level/root dispatch *(done)* | code | 4A-01, INF-04, INF-06 | | SWG-4A-07 | Parser: exact scalar types | code | 4A-06 | | SWG-4A-08 | Parser: structural tree | code | 4A-07, 4A-02 | | SWG-4A-09 | Checked `ScoreBuilder` | code | 4A-08 | @@ -833,6 +833,52 @@ Acceptance: Preregistered falsification probe: *adding owned lexeme text to the level-2 token* **must be CAUGHT**. +**Delivered.** `LANGUAGE_LEVEL` is 2. `syntax/document.rs` routes on the +level and holds no grammar of its own; `parser/v2.rs` owns the level-2 +entry, `format/v2.rs` the level-2 canonical text. Level 1's entry point +gained a guard that fires only for a level it does not own, because without +it a `swang 2` header flowed into a level-1 `Program` and the formatter +emitted `swang 2` above a `pattern` block. + +**What this slice accepts, exactly:** `score { ppqn }`. `ppqn` is +`score`'s only `1` word and the structural blocks are `*`/`?`, so the +minimal empty score is a complete level-2 program rather than a stub. + +**What it refuses, and who owns the refusal.** `master_bar`, `track`, +`source` and `loss` are real `score` words (§6.4b) owned by SWG-4A-08; they +are refused here with a message that says so, never skipped, because a +parser that ignores a word it does not implement is how exact text stops +being exact. Two registry codes are implemented, `SWG0505` and `SWG0506`, +and only for `ppqn` — the scalar layer proper stays 4A-07's. The alternative +was to accept `ppqn 0960` and `ppqn 0`, text §6.6 declares invalid, into the +accepted set of a level that has not frozen. + +**The three inherited SWG-INF-06 obligations, discharged:** + +- the budget is constructed in the level-2 entry and nowhere else, and + consulted before the work it bounds — source bytes pre-lex, tokens as each + is retained, depth as the block is entered. `level_two_budget_boundary.rs` + still passes, with `parser/v2.rs` and `parser/v2/lexer.rs` added to its + exempt list one at a time and the dispatcher deliberately left off it; +- `swang_parse` gains the dispatched arm, so a budget breach reaching a + fuzzed input is a typed `SWG0509`. Two deterministic breaches drive the + declared limits through the public path with no scaling: 16 MiB of + whitespace padding a *grammatically valid* score (so it would be accepted + if the byte check were not consulted first), and four million tokens plus + one spelled under the byte cap; +- the retained level-2 token is a kind and a `Span` — twelve bytes, no + `String`, text sliced from the source. The bound is a `const` assertion in + the lexer, so it travels to every target the crate builds for; the + preregistered probe (owned lexeme text) is **CAUGHT-BY-COMPILE** on + `wasm32-unknown-unknown`. + +**Inherited by SWG-4A-08**, found by falsification rather than assumed: the +depth axis is wired but cannot breach end-to-end in a grammar with one +block. Deleting `enter_block` was caught by nothing until a witness read the +counter directly on the real parse. When the structural tree lands, the +depth cap becomes reachable from text and deserves an end-to-end breach of +its own. + ### SWG-4A-07 — Parser: exact scalar types **Kind:** code. **Depends on:** 4A-06 diff --git a/fuzz/fuzz_targets/swang_parse.rs b/fuzz/fuzz_targets/swang_parse.rs index b0306e3..a50396c 100644 --- a/fuzz/fuzz_targets/swang_parse.rs +++ b/fuzz/fuzz_targets/swang_parse.rs @@ -15,8 +15,22 @@ //! `SWG\d{4}` and a span inside the source (`start <= end <= len`). //! * On `Ok`: `format` emits canonical text that reparses to the same AST //! (law 3) and is its own fixed point (law 2). +//! * `parse_document`: the dispatched path — level 1 or level 2 — is +//! `Ok(Document)` xor a non-empty `Vec`, every diagnostic +//! carries a registry code and an in-source span, and `format_document` +//! obeys the same two laws whichever level answered. +//! +//! SWG-INF-06 declared level 2's input bounds but could not honestly claim +//! an end-to-end breach oracle: public parsing could not reach a level-2 +//! parser at all, so the oracle would have covered a path the binary could +//! not enter. SWG-4A-06 is the task that opens it, so the oracle lands +//! here. A budget breach is a typed `SWG0509` reaching the caller — not an +//! abort, not an allocation death, not a silent success. -use griff_swang::syntax::{format, header_level, parse, Diagnostic, LANGUAGE_LEVEL}; +use griff_swang::syntax::{ + format, format_document, header_level, parse, parse_document, Diagnostic, Document, + LANGUAGE_LEVEL, +}; use libfuzzer_sys::fuzz_target; /// The one diagnostic contract, applied to the header pre-parser and the @@ -74,4 +88,35 @@ fuzz_target!(|source: &str| { } } } + + match parse_document(source) { + Ok(document) => { + let canonical = format_document(&document); + let reparsed = parse_document(&canonical) + .unwrap_or_else(|d| panic!("canonical text must reparse (law 2): {d:?}")); + assert_eq!( + format_document(&reparsed), + canonical, + "format_document is its own fixed point (law 2), at either level" + ); + match (&document, &reparsed) { + (Document::Pattern(a), Document::Pattern(b)) => { + assert_eq!(a, b, "parse(format(ast)) == ast (law 3)"); + } + (Document::Score(a), Document::Score(b)) => { + assert_eq!(a, b, "parse(format(ast)) == ast (law 3)"); + } + _ => panic!("the canonical text of a document keeps its root"), + } + } + Err(diagnostics) => { + assert!( + !diagnostics.is_empty(), + "a dispatched refusal names at least one diagnostic" + ); + for d in &diagnostics { + assert_diagnostic(d, len); + } + } + } }); diff --git a/swang/src/syntax.rs b/swang/src/syntax.rs index d03c274..b668cdf 100644 --- a/swang/src/syntax.rs +++ b/swang/src/syntax.rs @@ -55,6 +55,7 @@ mod ast; mod diagnostic; +mod document; mod format; mod header; mod lexer; @@ -69,9 +70,11 @@ pub use ast::v1::{ MapRhythm, PatternDef, Program, Prune, StrategyName, StrategyPolicy, StringLiteral, Unit, }; pub use diagnostic::Diagnostic; +pub use document::{format_document, parse_document, parse_document_with_source_map, Document}; pub use format::v1::format; pub use header::{header_level, LANGUAGE_LEVEL}; pub use parser::v1::{parse, parse_with_source_map}; +pub use parser::v2::ExactScore; pub use source_map::{AstId, FieldKind, FieldRef, Parsed, SourceMap}; pub use span::Span; #[cfg(test)] diff --git a/swang/src/syntax/ast/v1.rs b/swang/src/syntax/ast/v1.rs index 8a8ac0c..0faada8 100644 --- a/swang/src/syntax/ast/v1.rs +++ b/swang/src/syntax/ast/v1.rs @@ -5,7 +5,6 @@ use std::fmt; use griff_pattern::{DensityBps, Traversal}; -use crate::syntax::header::LANGUAGE_LEVEL; use crate::TailPolicy; /// Why an AST value refused to exist. @@ -36,7 +35,10 @@ pub enum AstError { }, /// A unit part is zero — no note value has a zero side. ZeroUnitPart, - /// Level zero, or newer than [`LANGUAGE_LEVEL`]. + /// Any level but 1 — this is the level-1 AST, and it spells one level + /// (spec §5.4). Deliberately not the build-wide range that + /// [`LANGUAGE_LEVEL`](crate::syntax::LANGUAGE_LEVEL) reports: a build + /// understanding a level says nothing about which tree carries it. UnsupportedLevel { /// The rejected level. level: u32, @@ -60,7 +62,7 @@ impl fmt::Display for AstError { Self::ZeroUnitPart => write!(f, "a unit part is zero"), Self::UnsupportedLevel { level } => write!( f, - "language level {level} is not supported (1..={LANGUAGE_LEVEL})" + "the level-1 AST spells level 1; it cannot carry level {level}" ), } } @@ -68,18 +70,32 @@ impl fmt::Display for AstError { impl Error for AstError {} -/// A pinned language level, valid by construction: nonzero and at most -/// [`LANGUAGE_LEVEL`]. +/// A pinned language level, valid by construction: level 1, the one this +/// AST spells — not every level +/// [`LANGUAGE_LEVEL`](crate::syntax::LANGUAGE_LEVEL) admits, for the reason +/// [`Level::new`] gives. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct Level(u32); +/// The one language level this AST spells (spec §5.4: each released level +/// owns its own parser, formatter and tree). +const LEVEL_ONE: u32 = 1; + impl Level { - /// Validates the level against this build's supported range. + /// Validates the level against the one level this AST spells. + /// + /// Level 1's AST carries level 1 and nothing else. Validating against + /// `LANGUAGE_LEVEL` instead would mean that raising the constant — which + /// SWG-4A-06 did — silently admits a `Program` whose header says + /// `swang 2` above a `pattern` root: text the formatter emits happily + /// and both entry points refuse, so `parse(format(ast)) == ast` fails + /// for an AST a caller can build from public fields. The mixed value is + /// unconstructible instead of merely unreachable. /// /// # Errors - /// [`AstError::UnsupportedLevel`] for zero or a newer level. + /// [`AstError::UnsupportedLevel`] for any level but 1. pub const fn new(level: u32) -> Result { - if level == 0 || level > LANGUAGE_LEVEL { + if level != LEVEL_ONE { return Err(AstError::UnsupportedLevel { level }); } Ok(Self(level)) diff --git a/swang/src/syntax/document.rs b/swang/src/syntax/document.rs new file mode 100644 index 0000000..0aae778 --- /dev/null +++ b/swang/src/syntax/document.rs @@ -0,0 +1,112 @@ +//! Level dispatch: one header, one entry point, no shared grammar. +//! +//! ```text +//! header_level -> level dispatch -> root dispatch (pattern | score) +//! ``` +//! +//! Spec §5.4 requires that "each released level owns its own parser and +//! formatter entry point. A build routes to one of them and never mixes +//! them: there is no single grammar with level-conditioned branches, because +//! such a grammar has no way to prove that level 1's behaviour survived the +//! addition of level 2." +//! +//! So this module is a router and nothing else. It holds no grammar, no +//! token, no budget and no diagnostic of its own beyond the one defensive +//! arm below: everything it can say, it says by choosing which parser to +//! call. That is also why it is not on `level_two_budget_boundary.rs`'s +//! exempt list — a budget consulted here, before the branch, would be a +//! level-1 bound whatever file it lived in. + +use super::ast::v1::Program; +use super::diagnostic::Diagnostic; +use super::format::v1::format as format_pattern; +use super::format::v2::format_exact; +use super::header::{header_level, LANGUAGE_LEVEL}; +use super::parser; +use super::parser::v2::ExactScore; +use super::source_map::Parsed; +use super::span::span_of; + +/// A parsed Swang document, tagged by the root its level admits. +/// +/// The two roots are two levels' (spec §5.7): `swang 1` writes `pattern`, +/// `swang 2` writes `score`, and nothing has yet earned a document that +/// holds both at once. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Document { + /// A level-1 pattern program. + Pattern(Program), + /// A level-2 exact score. + Score(ExactScore), +} + +/// Parses a source at whichever level its header pins, returning the +/// [`SourceMap`] side table alongside it. +/// +/// **The one router.** Every level decision in this build is this call: +/// [`parse_document`] is this function with the map dropped, so the two +/// cannot drift in what they accept or how they refuse — not by discipline, +/// but because there is only one of them to keep in step. Each level then +/// contributes its own level's map — level 1 the `Program`/`Pattern` ids it +/// already recorded, level 2 the `Score` root and its `Ppqn` field — because +/// a map is a location table, and a location belongs to the grammar that +/// read it. +/// +/// # Errors +/// The frozen §1.1 pre-parser's diagnostics for a header this build cannot +/// read, and otherwise exactly the chosen level's own. +/// +/// [`SourceMap`]: super::SourceMap +pub fn parse_document_with_source_map(source: &str) -> Result, Vec> { + let level = header_level(source).map_err(|d| vec![d])?; + match level { + 1 => parser::v1::parse_with_source_map(source).map(|parsed| Parsed { + value: Document::Pattern(parsed.value), + source_map: parsed.source_map, + }), + 2 => parser::v2::parse_exact_with_source_map(source).map(|parsed| Parsed { + value: Document::Score(parsed.value), + source_map: parsed.source_map, + }), + other => Err(vec![unsupported_level(other)]), + } +} + +/// Unreachable: the pre-parser already refused anything outside +/// `1..=LANGUAGE_LEVEL`. Defense in depth, so that raising the constant +/// without adding an arm is a refusal rather than a panic or, worse, a +/// silent fall-through to the wrong grammar. +fn unsupported_level(level: u32) -> Diagnostic { + Diagnostic { + code: "SWG0001", + span: span_of(6, 6), + message: format!( + "language level {level} has no parser in this build (1..={LANGUAGE_LEVEL})" + ), + } +} + +/// [`parse_document_with_source_map`] with the map dropped. +/// +/// The shape level 1's own pair already has — `v1::parse` is written this +/// way too — so the map costs a caller who does not want it exactly what it +/// costs them one layer down, and dispatch cannot be changed for one entry +/// point without being changed for the other. +/// +/// # Errors +/// Exactly [`parse_document_with_source_map`]'s. +pub fn parse_document(source: &str) -> Result> { + parse_document_with_source_map(source).map(|parsed| parsed.value) +} + +/// Emits the canonical text of a document, at its own level. +/// +/// Level 1's output is byte-identical to the frozen level-1 formatter's, +/// permanently (spec §5.8, Law A observable 3). +#[must_use] +pub fn format_document(document: &Document) -> String { + match document { + Document::Pattern(program) => format_pattern(program), + Document::Score(score) => format_exact(score), + } +} diff --git a/swang/src/syntax/format.rs b/swang/src/syntax/format.rs index 5adebf6..ffac2df 100644 --- a/swang/src/syntax/format.rs +++ b/swang/src/syntax/format.rs @@ -4,3 +4,4 @@ //! than editing it. pub(crate) mod v1; +pub(crate) mod v2; diff --git a/swang/src/syntax/format/v2.rs b/swang/src/syntax/format/v2.rs new file mode 100644 index 0000000..49f983e --- /dev/null +++ b/swang/src/syntax/format/v2.rs @@ -0,0 +1,37 @@ +//! The level-2 canonical formatter. +//! +//! Spec §5.8: for `swang 2` the formatter preserves `swang 2`. It never +//! downgrades a document to an older level and never promotes one to a newer +//! level — changing a level is an authoring act, never a formatting act. +//! +//! The canonical layout is `exact-score-text.md` §6.1's, restricted to what +//! SWG-4A-06 parses: the header, one blank line, and the `score` block with +//! a four-space field indent. The blocks that fill a fuller score are +//! SWG-4A-08's, and this slice cannot construct one that holds them. + +use crate::syntax::ast::v2::ExactScoreDocument; +use crate::syntax::parser::v2::ExactScore; + +/// Emits the one canonical text for `score`. +/// +/// `format_exact(parse(t))` is idempotent and `parse(format_exact(s))` +/// recovers the same score, for the level-2 parse. +pub(crate) fn format_exact(score: &ExactScore) -> String { + // Destructured with no `..` on purpose: when SWG-4A-08 starts filling + // the structural slots, this function stops compiling until it learns to + // emit them. A formatter that silently drops what it does not recognise + // is the one defect a canonical writer must not have. + let ExactScoreDocument { + ppqn, + master_bars, + tracks, + source, + loss, + } = score.document(); + debug_assert!( + master_bars.is_empty() && tracks.is_empty() && source.is_none() && loss.is_empty(), + "SWG-4A-06 parses only the minimal score; a document holding more \ + than that has arrived from somewhere this formatter cannot serve" + ); + format!("swang 2\n\nscore {{\n ppqn {ppqn}\n}}\n") +} diff --git a/swang/src/syntax/header.rs b/swang/src/syntax/header.rs index f68c3e8..a9198d9 100644 --- a/swang/src/syntax/header.rs +++ b/swang/src/syntax/header.rs @@ -8,9 +8,13 @@ use std::str::from_utf8; use super::diagnostic::Diagnostic; use super::span::span_of; -/// The language level this build parses (spec §1.1). Levels are additive-only -/// and never enter any content hash. -pub const LANGUAGE_LEVEL: u32 = 1; +/// The newest language level this build parses (spec §1.1). Levels are +/// additive-only and never enter any content hash. +/// +/// SWG-4A-06 raised this to 2. The pre-parser below is unchanged — that is +/// what "frozen" means — and what moved is the range it reports, which has +/// to be the range the build actually has or `SWG0001` would name a lie. +pub const LANGUAGE_LEVEL: u32 = 2; /// The frozen §1.1 pre-parser: reads at most 64 bytes of the first line and /// returns the pinned language level. diff --git a/swang/src/syntax/parser.rs b/swang/src/syntax/parser.rs index 7f06834..2f4da57 100644 --- a/swang/src/syntax/parser.rs +++ b/swang/src/syntax/parser.rs @@ -4,3 +4,4 @@ //! than editing it. pub(crate) mod v1; +pub(crate) mod v2; diff --git a/swang/src/syntax/parser/v1.rs b/swang/src/syntax/parser/v1.rs index 4e95e05..3ce2c49 100644 --- a/swang/src/syntax/parser/v1.rs +++ b/swang/src/syntax/parser/v1.rs @@ -22,9 +22,30 @@ use crate::TailPolicy; /// # Errors /// Exactly [`parse`]'s errors. pub fn parse_with_source_map(source: &str) -> Result, Vec> { + let pinned = header_level(source).map_err(|d| vec![d])?; + // §5.4: each released level owns its own entry point, and a build "routes + // to one of them and never mixes them". This is level 1's, so a source + // pinning any other level is refused here rather than half-read — which + // also keeps a level-2 header out of a level-1 `Program`, where the + // formatter would emit `swang 2` above a `pattern` block. + // + // `SWG0401` because that is what it is: a structural violation of the + // level-1 document contract, the same class as any other. It is not + // `SWG0001` — this build does support level 2, and §5.10 forbids one + // number carrying a second meaning. This arm cannot fire for a `swang 1` + // source, so Law A (§5.5) is untouched by its existence. + if pinned != 1 { + return Err(vec![Diagnostic { + code: "SWG0401", + span: level_span(source), + message: format!( + "the level-1 parser reads `swang 1`; this source pins `swang {pinned}`" + ), + }]); + } // `header_level` already enforced 1..=LANGUAGE_LEVEL; the map_err is // defense in depth, not a reachable path. - let level = Level::new(header_level(source).map_err(|d| vec![d])?).map_err(|e| { + let level = Level::new(pinned).map_err(|e| { vec![Diagnostic { code: "SWG0002", span: span_of(0, 0), diff --git a/swang/src/syntax/parser/v2.rs b/swang/src/syntax/parser/v2.rs new file mode 100644 index 0000000..17bbe67 --- /dev/null +++ b/swang/src/syntax/parser/v2.rs @@ -0,0 +1,373 @@ +//! The level-2 exact-score parser: root dispatch and the minimal score. +//! +//! Spec §5.4 gives each released level its own parser entry point, and this +//! is level 2's. It never sees a `swang 1` source and never branches on a +//! level — the branch happened one module up, in the dispatcher. +//! +//! # What this slice parses +//! +//! Exactly the minimal empty score: +//! +//! ```text +//! swang 2 +//! +//! score { +//! ppqn 960 +//! } +//! ``` +//! +//! `ppqn` is `score`'s only `1` word; `master_bar`, `track`, `source` and +//! `loss` are `*`/`?` words (`exact-score-text.md` §6.4b) whose omission is +//! §6.2's, not a missing required word. So the minimal score is a complete +//! level-2 program, not a stub — and the structural tree that fills those +//! slots is SWG-4A-08's. +//! +//! Words this slice does not implement are **refused**, never skipped. A +//! parser that ignores a word it does not understand is how exact text stops +//! being exact, and the refusal is a `SWG0401` that says which task owns the +//! word rather than pretending the grammar has no such thing. +//! +//! # Where the budget is spent +//! +//! The level-2 budget (SWG-INF-06, spec §5.11) is constructed here and +//! nowhere else, and it is consulted before the work it bounds: the source +//! byte cap before lexing, the token cap inside the lexer as each token is +//! retained, and structural depth as each block is entered. There is no +//! successful `swang 2` result that does not pass through all three. +//! +//! `admit_diagnostic` is deliberately not called. The diagnostic cap governs +//! what a *recovering* parser returns, and recovery is SWG-INF-05's; this +//! parser stops at the first refusal, so it can no more exhaust that axis +//! than level 1 can. + +pub(crate) mod lexer; + +use self::lexer::{lex_level_two, Level2Token, Level2TokenKind}; +use crate::syntax::ast::v2::ExactScoreDocument; +use crate::syntax::diagnostic::Diagnostic; +use crate::syntax::header::HEADER_WINDOW; +use crate::syntax::limits::Level2Budget; +use crate::syntax::source_map::{AstId, FieldKind, Parsed, SourceMap}; +use crate::syntax::span::{span_of, Span}; + +/// A parsed level-2 exact score. +/// +/// Opaque on purpose. SWG-4A-02 made the exact document a transient syntax +/// form and fenced it off the public surface so it could not quietly become +/// a second durable model beside `griff_core::Score`; the fence holds here. +/// This handle exists so that a level-2 parse has a public result at all — +/// which is what lets the fuzz oracle reach the real path — and it exposes +/// nothing of what it wraps. SWG-4A-09's builder is what opens it, into a +/// `Score` and nothing else. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ExactScore(ExactScoreDocument); + +impl ExactScore { + /// The wrapped document, for the crate-internal formatter. + pub(crate) const fn document(&self) -> &ExactScoreDocument { + &self.0 + } +} + +/// Parses a level-2 source, returning the [`SourceMap`] side table with it. +/// The caller has already proved the header says `swang 2`; this function +/// owns everything after it. +/// +/// The level-2 entry point, singular. Level 1 additionally offers `v1::parse` +/// as its map-dropping wrapper because callers of the frozen level want one; +/// level 2's only caller is the router, which takes the map, so a wrapper +/// here would be a second entry with no one behind it. +/// +/// # Errors +/// One diagnostic. `SWG0509` for a declared budget breach, `SWG0403` for a +/// missing required word, `SWG0404` for a repeated singleton, `SWG0505` for +/// a non-canonical spelling, `SWG0506` for a canonical-model invariant, and +/// `SWG0401` for everything structural. +pub(crate) fn parse_exact_with_source_map( + source: &str, +) -> Result, Vec> { + let mut budget = Level2Budget::declared(); + let mut map = SourceMap::default(); + parse_score(source, &mut budget, &mut map) + .map(|document| Parsed { + value: ExactScore(document), + source_map: map, + }) + .map_err(|d| vec![d]) +} + +/// The parse itself, spending a budget the caller owns. +/// +/// The caller owning it is what makes the accounting observable: the +/// counters can be read after the parse — including after a refusal, where +/// a depth still held is proof the block was entered. The budget is not an +/// argument a caller may weaken, because `Level2ResourceLimits` has private +/// fields and `declared()` as its only production constructor. +/// +/// **One budget, one parse.** A refusal returns with whatever the parse had +/// spent, depth included — deliberately, since that is the evidence the +/// block was entered — so a budget carried into a second parse would start +/// it part-spent. [`parse_exact_with_source_map`] constructs a fresh one +/// every time, which +/// is why no production path can reuse one; a future caller that wants to +/// bound a *sequence* of parses is declaring a different bound than §5.11's +/// per-source one and should say so explicitly rather than by accident. +/// +/// # Errors +/// Exactly [`parse_exact_with_source_map`]'s, unwrapped from the vector. +pub(crate) fn parse_score( + source: &str, + budget: &mut Level2Budget, + map: &mut SourceMap, +) -> Result { + budget.admit_source(source, level_span(source))?; + let tokens = lex_level_two(source, body_offset(source), budget)?; + let mut parser = Parser { + source, + tokens, + pos: 0, + eof: span_of(source.len(), source.len()), + budget, + map, + }; + let document = parser.score()?; + parser.expect_end()?; + Ok(document) +} + +/// The header's level digits: everything after `swang ` up to the line +/// break, trimmed. +/// +/// Spec §5.11's breach-location table sends a source-byte breach here — +/// "the level/header span — no body token has been admitted" — because at +/// that point nothing else has a location. `0..source.len()` would +/// highlight the whole oversized document and call that a location. +fn level_span(source: &str) -> Span { + let line_end = source + .as_bytes() + .iter() + .take(HEADER_WINDOW) + .position(|&b| b == b'\n') + .unwrap_or(source.len()); + let line = source.get(..line_end).unwrap_or_default(); + let end = line.trim_end().len(); + let start = HEADER.len().min(end); + span_of(start, end) +} + +/// The frozen §1.1 header prefix, whose length is where the level digits +/// begin. +const HEADER: &str = "swang "; + +/// The first byte after the header line, matching level 1's own arithmetic. +fn body_offset(source: &str) -> usize { + source + .as_bytes() + .iter() + .take(HEADER_WINDOW) + .position(|&b| b == b'\n') + .map_or(source.len(), |lf| lf.saturating_add(1)) +} + +/// The `score` words this grammar defines but this slice does not yet parse +/// (`exact-score-text.md` §6.4b). Named so the refusal can say which task +/// owns them instead of claiming they are not words at all. +const DEFERRED_SCORE_WORDS: &[&str] = &["master_bar", "track", "source", "loss"]; + +/// The one `score` root a level-2 document holds (§5.7). Numbered like level +/// 1's ids so the map has one shape across levels. +const SCORE: AstId = AstId::Score(0); + +struct Parser<'a> { + source: &'a str, + tokens: Vec, + pos: usize, + eof: Span, + budget: &'a mut Level2Budget, + map: &'a mut SourceMap, +} + +impl Parser<'_> { + fn next(&mut self) -> Option { + let token = self.tokens.get(self.pos).copied(); + if token.is_some() { + self.pos = self.pos.saturating_add(1); + } + token + } + + fn peek(&self) -> Option { + self.tokens.get(self.pos).copied() + } + + fn text(&self, token: Level2Token) -> &str { + token.text_in(self.source) + } + + fn unexpected_end(&self) -> Diagnostic { + Diagnostic { + code: "SWG0401", + span: self.eof, + message: "unexpected end of input".to_owned(), + } + } + + /// A structural refusal at a token: `SWG0401`, the class level 1 already + /// uses for an unexpected token or a violated shape. + const fn structural(token: Level2Token, message: String) -> Diagnostic { + Diagnostic { + code: "SWG0401", + span: token.span, + message, + } + } + + fn expect_kind( + &mut self, + kind: Level2TokenKind, + what: &str, + ) -> Result { + let token = self.next().ok_or_else(|| self.unexpected_end())?; + if token.kind == kind { + Ok(token) + } else { + let found = self.text(token).to_owned(); + Err(Self::structural( + token, + format!("expected {what}, found `{found}`"), + )) + } + } + + /// `score { … }` — the one level-2 root (spec §5.7). + fn score(&mut self) -> Result { + let root = self.expect_kind(Level2TokenKind::Word, "a root construct")?; + if self.text(root) != "score" { + let found = self.text(root).to_owned(); + return Err(Self::structural( + root, + format!("expected the `score` root, found `{found}`; level 2 admits no other root"), + )); + } + let open = self.expect_kind(Level2TokenKind::OpenBrace, "`{`")?; + self.budget.enter_block(open.span)?; + let (ppqn, close) = self.score_body(root.span)?; + self.budget.leave_block(); + // The construct is its keyword through its closing brace, and the + // field is the value rather than the word that introduces it — the + // convention level 1's map already follows. + self.map + .insert_node(SCORE, span_of(root.span.start as usize, close.end as usize)); + Ok(ExactScoreDocument { + ppqn, + master_bars: Vec::new(), + tracks: Vec::new(), + source: None, + loss: Vec::new(), + }) + } + + /// The `score` body up to its closing brace, returning the one value + /// this slice reads. + fn score_body(&mut self, root: Span) -> Result<(u16, Span), Diagnostic> { + let mut ppqn: Option = None; + let close = loop { + let token = self.next().ok_or_else(|| self.unexpected_end())?; + match token.kind { + Level2TokenKind::CloseBrace => break token.span, + Level2TokenKind::Word => { + let word = self.text(token).to_owned(); + if word != "ppqn" { + return Err(Self::unknown_word(token, &word)); + } + if ppqn.is_some() { + return Err(Diagnostic { + code: "SWG0404", + span: token.span, + message: "`score` takes one `ppqn` word".to_owned(), + }); + } + let (value, at) = self.ppqn_value()?; + self.map.insert_field(SCORE, FieldKind::Ppqn, at); + ppqn = Some(value); + } + Level2TokenKind::Number | Level2TokenKind::OpenBrace => { + let found = self.text(token).to_owned(); + return Err(Self::structural( + token, + format!("expected a word, found `{found}`"), + )); + } + } + }; + let value = ppqn.ok_or_else(|| Diagnostic { + code: "SWG0403", + span: root, + message: "`score` requires a `ppqn` word".to_owned(), + })?; + Ok((value, close)) + } + + /// A word `score` does not take here, said honestly: a word the grammar + /// defines but this slice has not implemented is not the same failure as + /// a word nobody has ever defined, and the message says which it is. + fn unknown_word(token: Level2Token, word: &str) -> Diagnostic { + let message = if DEFERRED_SCORE_WORDS.contains(&word) { + format!( + "`score` takes a `{word}` word, but this build does not parse \ + it yet (SWG-4A-08); it is refused rather than ignored" + ) + } else { + format!("`score` does not take a `{word}` word") + }; + Diagnostic { + code: "SWG0401", + span: token.span, + message, + } + } + + /// `ppqn ` — the one scalar this slice reads. + /// + /// The scalar layer proper is SWG-4A-07's. The two registry refusals + /// here are the ones reachable from this single word, and they are + /// implemented rather than deferred because the alternative is to accept + /// `ppqn 0960` and `ppqn 0` — text §6.6 declares invalid — into the + /// accepted set of a level that has not frozen. + fn ppqn_value(&mut self) -> Result<(u16, Span), Diagnostic> { + let token = self.expect_kind(Level2TokenKind::Number, "a `ppqn` value")?; + let text = self.text(token); + if text.len() > 1 && text.starts_with('0') { + return Err(Diagnostic { + code: "SWG0505", + span: token.span, + message: format!("`{text}` has a leading zero; the canonical spelling has none"), + }); + } + let value: u16 = text.parse().map_err(|_| Diagnostic { + code: "SWG0401", + span: token.span, + message: format!("`{text}` does not fit the `ppqn` field"), + })?; + if value == 0 { + return Err(Diagnostic { + code: "SWG0506", + span: token.span, + message: "`ppqn` is the tick resolution and cannot be zero".to_owned(), + }); + } + Ok((value, token.span)) + } + + /// One document holds one root: anything after the root's closing brace + /// is a second document trying to arrive inside the first. + fn expect_end(&self) -> Result<(), Diagnostic> { + self.peek().map_or(Ok(()), |token| { + let found = self.text(token).to_owned(); + Err(Self::structural( + token, + format!("expected the end of the document, found `{found}`"), + )) + }) + } +} diff --git a/swang/src/syntax/parser/v2/lexer.rs b/swang/src/syntax/parser/v2/lexer.rs new file mode 100644 index 0000000..608a851 --- /dev/null +++ b/swang/src/syntax/parser/v2/lexer.rs @@ -0,0 +1,135 @@ +//! The level-2 lexer and the token it retains. +//! +//! Level 1's [`Token`] owns a `String` per lexeme. That is not the level-2 +//! storage representation, and saying so is an obligation rather than a +//! preference: spec §5.11 declares `MAX_TOKENS = 4_000_000`, and the +//! derivation behind that number budgets **at most 12 bytes per retained +//! token on `wasm32`**. A token owning a `String` costs 12 bytes for the +//! `String` alone on a 32-bit target, before its kind, its span, and the +//! heap block each lexeme would allocate. +//! +//! So a level-2 token is a kind and a [`Span`], and the lexeme is sliced +//! back out of the source on demand ([`Level2Token::text_in`]). The bound is +//! asserted at compile time rather than in a host test, because the claim is +//! about `wasm32` and a runtime assertion on `x86_64` proves `x86_64`. +//! +//! [`Token`]: crate::syntax::token::Token + +use std::iter::Peekable; +use std::mem::size_of; +use std::str::CharIndices; + +use crate::syntax::diagnostic::Diagnostic; +use crate::syntax::limits::Level2Budget; +use crate::syntax::span::{span_of, Span}; + +/// One retained level-2 lexeme: what it is, and where it is. Never what it +/// says — that is [`Level2Token::text_in`]'s job, and the reason this type +/// fits the budget. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct Level2Token { + /// Which lexeme class this is. + pub(crate) kind: Level2TokenKind, + /// The full source range of the lexeme. + pub(crate) span: Span, +} + +/// The lexeme classes the level-2 grammar reads. Strings, and with them the +/// escape policy and `SWG0508`, are SWG-4A-07's. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Level2TokenKind { + /// `[A-Za-z_][A-Za-z0-9_]*` + Word, + /// A run of ASCII digits. Its spelling laws belong to the construct that + /// reads it, not to the lexer. + Number, + /// `{` + OpenBrace, + /// `}` + CloseBrace, +} + +/// The retained-token budget from spec §5.11, asserted where every target +/// this crate builds for must satisfy it — `wasm32` included. +/// +/// The preregistered SWG-INF-06 falsification probe is *adding owned lexeme +/// text to the level-2 token*: a `String` field takes the type to 24 bytes +/// on `wasm32`, and this assertion is what refuses to compile. +const _RETAINED_TOKEN_FITS_THE_BUDGET: () = assert!( + size_of::() <= 12, + "a retained level-2 token must fit spec §5.11's 12-byte wasm32 budget; \ + owning lexeme text is what breaks this" +); + +impl Level2Token { + /// The lexeme, sliced out of the source this token was lexed from. + pub(crate) fn text_in<'a>(&self, source: &'a str) -> &'a str { + source + .get(self.span.start as usize..self.span.end as usize) + .unwrap_or_default() + } +} + +/// Lexes `source` from byte `from` on, spending one token of `budget` for +/// every token retained. +/// +/// Whitespace is ASCII only, as at level 1: the determinism law (spec §1.2) +/// keeps Unicode classification out of anything semantics can observe. +/// +/// # Errors +/// `SWG0401` for a character the level-2 grammar does not read, and +/// `SWG0509` when the token allowance is spent — the breach is typed and +/// reaches the caller rather than being discovered by the allocator. +pub(crate) fn lex_level_two( + source: &str, + from: usize, + budget: &mut Level2Budget, +) -> Result, Diagnostic> { + let tail = source.get(from..).unwrap_or_default(); + let mut tokens = Vec::new(); + let mut chars = tail.char_indices().peekable(); + while let Some((at, c)) = chars.next() { + let start = from.saturating_add(at); + let kind = match c { + ' ' | '\t' | '\r' | '\n' => continue, + '{' => Level2TokenKind::OpenBrace, + '}' => Level2TokenKind::CloseBrace, + 'A'..='Z' | 'a'..='z' | '_' => Level2TokenKind::Word, + '0'..='9' => Level2TokenKind::Number, + other => { + return Err(Diagnostic { + code: "SWG0401", + span: span_of(start, start.saturating_add(other.len_utf8())), + message: format!("unexpected character {other:?}"), + }) + } + }; + let end = match kind { + Level2TokenKind::Word => run(tail, &mut chars, |ch| { + ch.is_ascii_alphanumeric() || ch == '_' + }), + Level2TokenKind::Number => run(tail, &mut chars, |ch| ch.is_ascii_digit()), + Level2TokenKind::OpenBrace | Level2TokenKind::CloseBrace => at.saturating_add(1), + }; + let span = span_of(start, from.saturating_add(end)); + // Admitted before it is retained: a token that would exceed the + // allowance is never stored, so the refusal costs one token's worth + // of memory rather than the whole tail of the file. + budget.admit_token(span)?; + tokens.push(Level2Token { kind, span }); + } + Ok(tokens) +} + +/// Consumes characters while `keep` holds; returns the end byte offset +/// relative to `tail`. +fn run(tail: &str, chars: &mut Peekable>, keep: impl Fn(char) -> bool) -> usize { + while let Some(&(_, c)) = chars.peek() { + if keep(c) { + chars.next(); + } else { + break; + } + } + chars.peek().map_or(tail.len(), |&(next, _)| next) +} diff --git a/swang/src/syntax/source_map.rs b/swang/src/syntax/source_map.rs index 54fdef7..318ee48 100644 --- a/swang/src/syntax/source_map.rs +++ b/swang/src/syntax/source_map.rs @@ -69,6 +69,8 @@ pub enum AstId { Generate(u32), /// An `export …` pipeline step. Export(u32), + /// A level-2 `score { … }` root (SWG-4A-06). + Score(u32), } /// Which value of a construct a location belongs to. @@ -117,6 +119,8 @@ pub enum FieldKind { Format, /// The quoted `export` path literal. Path, + /// A level-2 `score { ppqn … }` tick resolution (SWG-4A-06). + Ppqn, } /// One value's address: which construct, and which of its words. diff --git a/swang/src/syntax/tests.rs b/swang/src/syntax/tests.rs index a58fd28..7555c28 100644 --- a/swang/src/syntax/tests.rs +++ b/swang/src/syntax/tests.rs @@ -109,7 +109,10 @@ fn first_error(source: &str) -> Diagnostic { #[test] fn the_frozen_header_form_pins_the_level() { assert_eq!(header_level("swang 1\nrest").expect("frozen form"), 1); - assert_eq!(LANGUAGE_LEVEL, 1, "this build parses level 1"); + // The frozen form is frozen; the range this build supports is not, and + // SWG-4A-06 moved it. Level 1 still reads exactly as it did — that is + // the line above, and Law A's baseline is the fuller proof. + assert_eq!(LANGUAGE_LEVEL, 2, "SWG-4A-06 admits level 2"); } #[test] @@ -157,13 +160,21 @@ fn the_pre_parser_reads_at_most_64_bytes() { #[test] fn a_newer_level_is_swg0001_naming_the_supported_range() { - let d = header_level("swang 2\n").expect_err("newer than this build"); + // `swang 2` was this case's fixture until SWG-4A-06 made it supported. + // The refusal is not what changed — the range is — so the case moves up + // to the next unsupported level and now pins the range exactly, rather + // than pinning that the message contains the digit one. + let d = header_level("swang 3\n").expect_err("newer than this build"); assert_eq!(d.code, "SWG0001"); assert!( - d.message.contains('1'), + d.message.contains("1..=2"), "the message names the supported range: {}", d.message ); + assert_eq!( + header_level("swang 2\n").expect("level 2 is this build's"), + 2 + ); } #[test] @@ -897,3 +908,159 @@ mod level_two_budget { assert_eq!(budget.diagnostics(), 2, "the terminal one is counted"); } } + +/// SWG-4A-06: the depth axis is wired, and the wiring is falsifiable. +/// +/// The minimal score has exactly one block, so nesting can never approach +/// `MAX_NESTING_DEPTH` and no end-to-end breach can witness the accounting. +/// A falsification probe found that out the honest way: deleting +/// `enter_block` from the parser was caught by nothing at all. So the +/// witness observes the counter directly, on the real parse function with +/// the real budget — not on a reconstruction of it. +mod level_two_depth_accounting { + use crate::syntax::limits::Level2Budget; + use crate::syntax::parser::v2::parse_score; + use crate::syntax::source_map::SourceMap; + use crate::syntax::span::span_of; + + fn budget_for(source: &str) -> Level2Budget { + let budget = Level2Budget::declared(); + budget + .admit_source(source, span_of(0, source.len())) + .expect("far under the source cap"); + budget + } + + #[test] + fn the_root_block_is_entered_on_the_caller_s_budget() { + // A refusal *inside* the block is what makes this observable: the + // parse returns before `leave_block`, so a depth of one is proof the + // block was entered, where a balanced parse would have returned the + // counter to zero either way. + let source = "swang 2 + +score { + nope 1 +} +"; + let mut budget = budget_for(source); + let mut map = SourceMap::default(); + let refusal = + parse_score(source, &mut budget, &mut map).expect_err("`nope` is not a `score` word"); + assert_eq!(refusal.code, "SWG0401"); + assert_eq!( + budget.depth(), + 1, + "the root block was entered on the budget the caller owns" + ); + } + + #[test] + fn a_balanced_parse_returns_the_depth_it_borrowed() { + let source = "swang 2\n\nscore {\n ppqn 960\n}\n".replace("\\n", "\n"); + let mut budget = budget_for(&source); + let mut map = SourceMap::default(); + parse_score(&source, &mut budget, &mut map).expect("the minimal score"); + assert_eq!(budget.depth(), 0, "what was entered was left"); + } +} + +/// SWG-4A-06's inherited SWG-INF-06 obligation: the level-2 token proves the +/// `MAX_TOKENS` heap derivation instead of asserting it in prose. +mod level_two_token_storage { + use std::mem::size_of; + + use crate::syntax::limits::Level2Budget; + use crate::syntax::parser::v2::lexer::{lex_level_two, Level2Token, Level2TokenKind}; + use crate::syntax::span::span_of; + + /// The budget derivation in spec §5.11 is `<= 12` bytes per retained + /// token on `wasm32`. `Span` is two `u32` by the determinism law, so the + /// only way to exceed it is to start owning something. + const MAX_RETAINED_TOKEN_BYTES: usize = 12; + + #[test] + fn a_retained_level_two_token_fits_the_declared_budget() { + assert!( + size_of::() <= MAX_RETAINED_TOKEN_BYTES, + "a retained token is at most {MAX_RETAINED_TOKEN_BYTES} bytes; \ + this one is {}", + size_of::() + ); + } + + #[test] + fn the_size_bound_is_asserted_at_compile_time_so_it_travels_to_wasm32() { + // This test runs on the host only. The binding obligation is the + // `const` assertion in the lexer itself, which every target this + // crate builds for must satisfy — including the `wasm32` frontend + // CI builds. A host-only runtime check would prove the host and + // call it a platform proof. + let shipped = include_str!("parser/v2/lexer.rs"); + assert!( + shipped.contains("const _RETAINED_TOKEN_FITS_THE_BUDGET"), + "the compile-time assertion is what reaches wasm32; a runtime \ + test on x86_64 does not" + ); + } + + #[test] + fn level_two_token_text_is_recovered_from_the_source_span() { + // "Token text is recovered from the source span." Level 1's + // `String`-owning `Token` is not the level-2 storage representation, + // so the lexeme has to be sliced back out on demand. + let source = "swang 2\n\nscore {\n ppqn 960\n}\n"; + let tokens = lex_level_two(source, 9, &mut budget_for(source)).expect("lexes"); + let words: Vec<&str> = tokens + .iter() + .filter(|t| t.kind == Level2TokenKind::Word) + .map(|t| t.text_in(source)) + .collect(); + assert_eq!(words, ["score", "ppqn"]); + let numbers: Vec<&str> = tokens + .iter() + .filter(|t| t.kind == Level2TokenKind::Number) + .map(|t| t.text_in(source)) + .collect(); + assert_eq!(numbers, ["960"]); + } + + #[test] + fn every_level_two_token_carries_a_span_that_slices_its_own_lexeme() { + let source = "swang 2\n\nscore {\n ppqn 960\n}\n"; + let tokens = lex_level_two(source, 9, &mut budget_for(source)).expect("lexes"); + assert!(!tokens.is_empty()); + for token in &tokens { + let text = token.text_in(source); + assert!(!text.is_empty(), "a token spans at least one byte"); + let start = token.span.start as usize; + assert_eq!( + source.get(start..token.span.end as usize), + Some(text), + "the span is the lexeme, not an approximation of it" + ); + } + } + + #[test] + fn the_lexer_spends_the_token_budget_it_is_given() { + // Accounting attached to the real transition it constrains: one + // admitted token per token retained, on the budget the caller owns. + let source = "swang 2\n\nscore {\n ppqn 960\n}\n"; + let mut budget = budget_for(source); + let tokens = lex_level_two(source, 9, &mut budget).expect("lexes"); + assert_eq!( + budget.tokens(), + tokens.len() as u64, + "no token is retained without being admitted first" + ); + } + + fn budget_for(source: &str) -> Level2Budget { + let budget = Level2Budget::declared(); + budget + .admit_source(source, span_of(0, source.len())) + .expect("this fixture is far under the source cap"); + budget + } +} diff --git a/swang/tests/level_two_budget_boundary.rs b/swang/tests/level_two_budget_boundary.rs index cc34c83..6371092 100644 --- a/swang/tests/level_two_budget_boundary.rs +++ b/swang/tests/level_two_budget_boundary.rs @@ -340,7 +340,18 @@ fn every_shipped_module_but_the_budget_itself_is_scanned() { /// like), those go on this list explicitly, one at a time. Shared dispatch /// never joins it: a budget consulted before the level branch is a level-1 /// bound, whatever file it lives in. -const EXEMPT: &[&[&str]] = &[&["syntax", "limits.rs"], &["syntax", "tests.rs"]]; +const EXEMPT: &[&[&str]] = &[ + &["syntax", "limits.rs"], + &["syntax", "tests.rs"], + // SWG-4A-06's level-2 modules, added one at a time as the comment above + // requires. Both are level-2-only: the entry point that constructs the + // budget, and the lexer that spends its token allowance. Neither is on + // any level-1 path, and `syntax.rs` and the dispatcher stay off this + // list, because a budget consulted before the level branch is a level-1 + // bound whatever file it lives in. + &["syntax", "parser", "v2.rs"], + &["syntax", "parser", "v2", "lexer.rs"], +]; /// Whether a path relative to `swang/src` is one of the exempt modules. /// diff --git a/swang/tests/level_two_dispatch.rs b/swang/tests/level_two_dispatch.rs new file mode 100644 index 0000000..c202dc8 --- /dev/null +++ b/swang/tests/level_two_dispatch.rs @@ -0,0 +1,570 @@ +//! SWG-4A-06: level dispatch, root dispatch, and the level-2 skeleton. +//! +//! ```text +//! header_level -> level dispatch -> root dispatch (pattern | score) +//! ``` +//! +//! Spec §5.4 says a build "routes to one of them and never mixes them: +//! there is no single grammar with level-conditioned branches, because such +//! a grammar has no way to prove that level 1's behaviour survived the +//! addition of level 2". These tests are that proof obligation, written +//! down: every case here either pins where a source is routed, or pins that +//! a source which routes nowhere is refused rather than half-read. +//! +//! # The slice boundary +//! +//! This task accepts exactly one level-2 program shape — the minimal empty +//! score, `score { ppqn }`. Every other grammatical `score` word +//! (`master_bar`, `track`, `source`, `loss`) is real grammar owned by +//! SWG-4A-08 and is **refused here**, not tolerated: a parser that ignores +//! a word it does not implement is how exact text stops being exact. The +//! scalar layer is SWG-4A-07's; the two refusals this slice implements +//! (`SWG0505`, `SWG0506`) are the ones reachable from the single word it +//! reads, because the alternative is to accept `ppqn 0` — text §6.6 +//! declares invalid — into the accepted set of a level that has not frozen. + +// Reason: integration-test code. `unwrap`/`expect`/`panic` abort loudly with +// a clear message, which is exactly what a test harness wants. +#![allow( + clippy::unwrap_used, + clippy::expect_used, + clippy::panic, + clippy::missing_assert_message +)] + +use griff_swang::syntax::{ + format, format_document, header_level, parse, parse_document, parse_document_with_source_map, + AstId, Diagnostic, Document, FieldKind, FieldRef, Level, Span, LANGUAGE_LEVEL, +}; + +/// The one level-1 program every dispatch test measures level 1 against: +/// spec §3.1's reference program, byte for byte, so "level 1 is unchanged" +/// is measured against the text level 1 froze rather than a sketch of it. +const LEVEL_ONE: &str = r#"swang 1 + +pattern dgd_fractal { + ascii "X.X/XX./.XX" + |> fractalize depth 1 max_cells 4096 density 9500bps seed 4 + |> linearize snake + |> map_rhythm unit 1/16 tail rest_pad + |> generate { + source "corpus/Dance Gavin Dance - The Robot With Human Hair Part 2.gp5" + bars 8 + seed 42 + candidates 2 + strategy repeat_variation + corpus "corpus" + } + |> export midi "dgd_fractal_dense.mid" +} +"#; + +/// The same program under a level-2 header: a level-1 body is not level-2 +/// text, however well formed it is at its own level. +const LEVEL_ONE_BODY_UNDER_TWO: &str = r#"swang 2 + +pattern dgd_fractal { + ascii "X.X/XX./.XX" +} +"#; + +/// The minimal level-2 score: `ppqn` is the construct's only `1` word, and +/// `master_bar`/`track`/`source`/`loss` are omissible (§6.2, §6.4b). +const MINIMAL_SCORE: &str = "swang 2\n\nscore {\n ppqn 960\n}\n"; + +fn codes(diagnostics: &[Diagnostic]) -> Vec<&str> { + diagnostics.iter().map(|d| d.code).collect() +} + +fn refusal(source: &str) -> Vec { + parse_document(source).err().unwrap_or_else(|| { + panic!("this source must be refused, and it was accepted:\n{source}"); + }) +} + +// ── the level surface ─────────────────────────────────────────────────────── + +#[test] +fn this_build_supports_level_two() { + assert_eq!( + LANGUAGE_LEVEL, 2, + "SWG-4A-06 is the task that admits level 2" + ); + assert_eq!(header_level(MINIMAL_SCORE), Ok(2)); + assert_eq!(header_level(LEVEL_ONE), Ok(1)); +} + +#[test] +fn an_unknown_newer_level_is_still_refused_by_the_frozen_pre_parser() { + // §1.1: the pre-parser never changes across releases. What changes is + // the supported range it names, and it must name the range this build + // actually has. + let d = header_level("swang 3\n\nscore {\n}\n").expect_err("level 3 is not supported"); + assert_eq!(d.code, "SWG0001"); + assert!( + d.message.contains("1..=2"), + "SWG0001 reports the supported range: {}", + d.message + ); + assert_eq!(codes(&refusal("swang 3\n\nscore {\n}\n")), ["SWG0001"]); + assert_eq!( + codes(&refusal("swang 999999999\n\nscore {\n}\n")), + ["SWG0001"] + ); +} + +#[test] +fn a_malformed_or_truncated_header_is_still_swg0002() { + for source in [ + "swang\n\nscore {\n}\n", + "swang 2", + "swang 2\n", + "swang 02\n", + " swang 2\n", + "SWANG 2\n", + "swang 2x\n", + "", + ] { + assert_eq!( + codes(&refusal(source)), + ["SWG0002"], + "a malformed header is refused before any grammar sees it: {source:?}" + ); + } + assert_eq!(codes(&refusal("\u{feff}swang 2\n")), ["SWG0003"]); +} + +// ── level dispatch ────────────────────────────────────────────────────────── + +#[test] +fn a_level_one_source_routes_to_the_frozen_level_one_parser() { + let Ok(Document::Pattern(program)) = parse_document(LEVEL_ONE) else { + panic!("a valid `swang 1` source routes to the level-1 parser"); + }; + let direct = parse(LEVEL_ONE).expect("the frozen entry point still accepts it"); + assert_eq!(program, direct, "dispatch adds no reinterpretation"); +} + +#[test] +fn dispatching_a_level_one_source_formats_byte_for_byte_as_before() { + // Law A observable 3, through the new entry point. + let document = parse_document(LEVEL_ONE).expect("accepted"); + let program = parse(LEVEL_ONE).expect("accepted"); + assert_eq!(format_document(&document), format(&program)); +} + +#[test] +fn a_minimal_level_two_score_reaches_the_exact_parser() { + let Ok(Document::Score(_)) = parse_document(MINIMAL_SCORE) else { + panic!("`swang 2` with a `score` root routes to the level-2 parser"); + }; +} + +#[test] +fn the_level_one_entry_point_does_not_read_level_two() { + // §5.4: one build routes to one entry point and never mixes them. The + // frozen entry point must not half-read a level it does not own — and + // must never hand back a level-1 `Program` carrying level 2. + let diagnostics = parse(MINIMAL_SCORE).expect_err("the level-1 parser does not read level 2"); + assert_eq!(codes(&diagnostics), ["SWG0401"]); + let under_two = parse(LEVEL_ONE_BODY_UNDER_TWO) + .expect_err("not even a level-1 body under a level-2 header"); + assert_eq!(codes(&under_two), ["SWG0401"]); +} + +// ── root dispatch ─────────────────────────────────────────────────────────── + +#[test] +fn a_level_two_pattern_root_is_refused() { + // §5.7: "Level 2 does not admit the level-1 `pattern` root." + let diagnostics = refusal("swang 2\n\npattern riff {\n}\n"); + assert_eq!(codes(&diagnostics), ["SWG0401"]); + let first = diagnostics.first().expect("one diagnostic"); + assert!( + first.message.contains("score") && first.message.contains("pattern"), + "the refusal names the root it wanted and the root it found: {}", + first.message + ); + // And a real level-1 body under a level-2 header is refused too. Its + // string literal reaches the level-2 lexer before the root check does, + // so the message is the lexer's — the verdict is what this case pins, + // and the verdict is closed either way. + assert_eq!(codes(&refusal(LEVEL_ONE_BODY_UNDER_TWO)), ["SWG0401"]); +} + +#[test] +fn a_level_one_score_root_keeps_its_frozen_refusal() { + // Law A's invalid-body half (§5.5): a level-2 keyword in a `swang 1` + // script raises exactly what level 1 already raised — not a friendlier + // "`score` requires language level 2". + let diagnostics = refusal("swang 1\n\nscore {\n ppqn 960\n}\n"); + assert_eq!(codes(&diagnostics), ["SWG0401"]); + let through_frozen = + parse("swang 1\n\nscore {\n ppqn 960\n}\n").expect_err("level 1 refuses a `score` root"); + assert_eq!( + diagnostics, through_frozen, + "identical code, message, span and order — the frozen verdict is not \ + rewritten by the arrival of level 2" + ); +} + +#[test] +fn a_malformed_prefix_is_not_accepted_as_a_root() { + for source in [ + "swang 2\n\nscor {\n ppqn 960\n}\n", + "swang 2\n\nscores {\n ppqn 960\n}\n", + "swang 2\n\nscore\n", + "swang 2\n\nscore {\n", + "swang 2\n\nscore {\n ppqn 960\n", + "swang 2\n\n{\n ppqn 960\n}\n", + "swang 2\n\n", + ] { + assert_eq!( + codes(&refusal(source)), + ["SWG0401"], + "a prefix of the root is not the root: {source:?}" + ); + } +} + +#[test] +fn trailing_material_cannot_bypass_the_root_contract() { + for source in [ + "swang 2\n\nscore {\n ppqn 960\n}\nscore {\n ppqn 480\n}\n", + "swang 2\n\nscore {\n ppqn 960\n}\npattern riff {\n ascii \"x-\"\n}\n", + "swang 2\n\nscore {\n ppqn 960\n}\n}\n", + "swang 2\n\nscore {\n ppqn 960\n} ppqn 480\n", + ] { + assert_eq!( + codes(&refusal(source)), + ["SWG0401"], + "one document holds one root: {source:?}" + ); + } +} + +// ── the score body, and what this slice does not yet parse ────────────────── + +#[test] +fn ppqn_is_the_one_required_word() { + assert_eq!(codes(&refusal("swang 2\n\nscore {\n}\n")), ["SWG0403"]); + assert_eq!( + codes(&refusal( + "swang 2\n\nscore {\n ppqn 960\n ppqn 480\n}\n" + )), + ["SWG0404"], + "`ppqn` is a `1` word, so repeating it is SWG0404 (§6.4b)" + ); +} + +#[test] +fn an_unknown_score_word_is_refused() { + assert_eq!( + codes(&refusal( + "swang 2\n\nscore {\n ppqn 960\n nope 1\n}\n" + )), + ["SWG0401"], + "§6.4b: an unknown field word is SWG0401" + ); +} + +#[test] +fn grammatical_words_this_slice_does_not_parse_fail_closed() { + // `master_bar`, `track`, `source` and `loss` are real `score` words + // (§6.4b) owned by SWG-4A-08. Until it lands they are refused, never + // skipped: silently ignoring a word is how exact text stops being exact. + for word in [ + "master_bar {\n }", + "track {\n }", + "source {\n }", + "loss {\n }", + ] { + let source = format!("swang 2\n\nscore {{\n ppqn 960\n {word}\n}}\n"); + assert_eq!( + codes(&refusal(&source)), + ["SWG0401"], + "not yet parsed means refused, not ignored: {word}" + ); + } +} + +#[test] +fn the_one_scalar_this_slice_reads_refuses_what_the_registry_says_it_must() { + // Scoped deliberately to `ppqn`. The scalar layer — widths, non-zero + // types, rational tempo, ranges, pitch, velocity, meter, confidence, + // enums, escapes — is SWG-4A-07's and is absent here. + assert_eq!( + codes(&refusal("swang 2\n\nscore {\n ppqn 0960\n}\n")), + ["SWG0505"], + "§6.6: a leading zero is a non-canonical spelling" + ); + assert_eq!( + codes(&refusal("swang 2\n\nscore {\n ppqn 0\n}\n")), + ["SWG0506"], + "§6.6 names zero `ppqn` as a canonical-model invariant violation" + ); + for over in ["ppqn 70000", "ppqn 99999999999999999999"] { + assert_eq!( + codes(&refusal(&format!("swang 2\n\nscore {{\n {over}\n}}\n"))), + ["SWG0401"], + "a value that does not fit the field is SWG0401, as at level 1" + ); + } + for shape in ["ppqn", "ppqn x", "ppqn \"960\"", "ppqn 9/6", "ppqn 960bps"] { + assert_eq!( + codes(&refusal(&format!("swang 2\n\nscore {{\n {shape}\n}}\n"))), + ["SWG0401"], + "and a value of the wrong shape is SWG0401: {shape}" + ); + } +} + +// ── the formatter, dispatched ─────────────────────────────────────────────── + +#[test] +fn the_level_two_formatter_preserves_the_level_and_is_its_own_fixed_point() { + // §5.8: the formatter never downgrades or promotes a document. + let document = parse_document(MINIMAL_SCORE).expect("accepted"); + let canonical = format_document(&document); + assert!( + canonical.starts_with("swang 2\n"), + "the level travels with the document: {canonical:?}" + ); + let reparsed = parse_document(&canonical).expect("canonical text reparses"); + assert_eq!(format_document(&reparsed), canonical, "law 2: fixed point"); + assert_eq!(reparsed, document, "law 3: and it is the same document"); + let Document::Score(_) = reparsed else { + panic!("canonical level-2 text is still a score"); + }; +} + +#[test] +fn a_level_two_round_trip_preserves_the_value_it_parsed() { + // Law 3, `parse(format(ast)) == ast`, which the fixed-point law does not + // imply: a formatter emitting a *constant* `ppqn` is its own fixed point + // and reparses to something that is not what it was given. A probe found + // that hole the honest way — replacing the interpolated `ppqn` with the + // literal 960 was caught by nothing, because every fixture used 960. + // + // So the values here are deliberately not the one the other fixtures use. + for ppqn in ["1", "480", "65535"] { + let source = format!("swang 2\n\nscore {{\n ppqn {ppqn}\n}}\n"); + let document = parse_document(&source).expect("a valid minimal score"); + let canonical = format_document(&document); + assert_eq!(canonical, source, "this fixture is already canonical"); + let reparsed = parse_document(&canonical).expect("canonical text reparses"); + assert_eq!( + reparsed, document, + "parse(format(ast)) == ast: the value survives the round trip" + ); + } +} + +#[test] +fn canonical_level_two_text_is_the_one_spelling() { + let document = parse_document("swang 2\n\nscore {\nppqn 960\n}\n").expect("accepted"); + assert_eq!(format_document(&document), MINIMAL_SCORE); +} + +// -- the level a constructed AST may carry ---------------------------------- + +#[test] +fn a_level_one_ast_cannot_carry_level_two() { + // The parser guard closed the *parse* path; this closes the + // *construction* path, which `Program`'s public fields leave open. + // `ast::v1::Level` validated against `LANGUAGE_LEVEL`, so raising the + // constant silently made `Level::new(2)` succeed — and a caller could + // then build a `Program` the formatter renders as `swang 2` above a + // `pattern` root. Both entry points refuse that text, so + // `parse(format(ast)) == ast` would fail for an AST anyone can build. + Level::new(1).expect("level 1 is the level this AST spells"); + for other in [0, 2, 3, LANGUAGE_LEVEL, u32::MAX] { + assert!( + other == 1 || Level::new(other).is_err(), + "the level-1 AST spells no level but 1; `Level::new({other})` must \ + not succeed merely because the build understands that level" + ); + } +} + +#[test] +fn the_level_one_ast_refusal_names_the_level_it_spells() { + // Codex, on 6d1775d: pinning the constructor to level 1 left its refusal + // still printing the *build's* range. With `LANGUAGE_LEVEL` at 2 that + // renders "language level 2 is not supported (1..=2)" — a message that + // refuses a level while listing it among the supported ones. A public + // error contradicting itself is a defect on its own: it tells the caller + // the call it just lost should have succeeded, and sends them looking for + // the bug anywhere but where it is. + for level in [0_u32, 2, 3, LANGUAGE_LEVEL, u32::MAX] { + if level == 1 { + continue; + } + let refusal = Level::new(level) + .expect_err("the level-1 AST spells no level but 1") + .to_string(); + assert!( + refusal.contains(&level.to_string()), + "the refusal must name the level it rejected; got {refusal:?}" + ); + assert!( + !refusal.contains(&format!("1..={LANGUAGE_LEVEL}")), + "the level-1 AST's refusal must describe the level *it* spells, \ + not the build's supported range — a range that now contains the \ + level just refused; got {refusal:?}" + ); + } +} + +// -- the source map the skeleton ships with --------------------------------- + +#[test] +fn a_level_two_parse_carries_a_source_map() { + // The SWG-4A-06 contract: "the skeleton ships complete: a typed root + // enum, the source map, deterministic diagnostics, resource budgets, and + // formatter dispatch." Without it a caller cannot locate a successfully + // parsed level-2 construct, and 4A-09's builder diagnostics would have + // to reparse to point anywhere. + let parsed = parse_document_with_source_map(MINIMAL_SCORE).expect("accepted"); + let Document::Score(_) = parsed.value else { + panic!("it is a score"); + }; + let map = &parsed.source_map; + let root = map + .node_span(AstId::Score(0)) + .expect("the score block has a location"); + assert_eq!( + MINIMAL_SCORE.get(root.start as usize..root.end as usize), + Some("score {\n ppqn 960\n}"), + "the node span is the construct, from its keyword to its brace" + ); + let ppqn = map + .field_span(FieldRef::new(AstId::Score(0), FieldKind::Ppqn)) + .expect("`ppqn` has a location"); + assert_eq!( + MINIMAL_SCORE.get(ppqn.start as usize..ppqn.end as usize), + Some("960"), + "the field span is the value, not the word that introduces it" + ); +} + +#[test] +fn the_router_decides_the_level_once() { + // `parse_document_with_source_map`'s doc says `parse_document` "is this + // function with the map dropped, so the two cannot drift in what they + // accept or how they refuse". That sentence describes a delegation, and + // for a while it described one that did not exist: the module held two + // independent dispatches, each reading the level for itself, and the + // property was maintained by hand rather than established by + // construction — while one layer down, `v1::parse` really is its + // map-bearing twin with the map dropped, which is presumably where the + // sentence came from. + // + // This is lexical because the claim is structural. The test below can + // only sample inputs, and two routers agreeing on every input a suite + // happens to try is exactly what a second router looks like right up + // until it doesn't. One read of the level means one function owns the + // decision, and any other entry point has to go through it. + let shipped = include_str!("../src/syntax/document.rs"); + assert_eq!( + shipped.matches("header_level(").count(), + 1, + "the level is read once, by the one router; a second reader is a \ + second dispatch, free to drift from the first" + ); +} + +#[test] +fn the_dispatched_map_and_the_dispatched_parse_agree() { + // One parser, two entry points: the plain one is the mapped one with the + // map dropped, so the two cannot drift in what they accept. This samples + // that agreement; `the_router_decides_the_level_once` is what makes it + // structural rather than a coincidence this corpus happens to confirm. + for source in [MINIMAL_SCORE, LEVEL_ONE] { + let mapped = parse_document_with_source_map(source).expect("accepted"); + let plain = parse_document(source).expect("accepted"); + assert_eq!(mapped.value, plain); + } + for source in [ + "swang 2\n\nscore {\n}\n", + "swang 3\n", + "swang 2\n\npattern x {\n}\n", + ] { + assert_eq!( + parse_document_with_source_map(source).err(), + parse_document(source).err(), + "and they refuse identically: {source:?}" + ); + } +} + +// ── the inherited INF-06 obligations ──────────────────────────────────────── + +#[test] +fn the_source_byte_budget_breaches_end_to_end_before_any_success() { + // The declared limit, through the public path, with no scaling and no + // test-only reconstruction of the parser. The padding is whitespace, so + // the source is *grammatically* a valid minimal score and would be + // accepted if the byte check were not consulted first — which is what + // makes this a witness that no successful `swang 2` result precedes + // budget wiring, rather than merely a witness that a big string fails. + let padding = " ".repeat(16 * 1024 * 1024); + let source = format!("swang 2\n\nscore {{\n ppqn 960\n{padding}}}\n"); + let diagnostics = refusal(&source); + assert_eq!(codes(&diagnostics), ["SWG0509"]); + let first = diagnostics.first().expect("one diagnostic"); + assert!( + first.message.contains("source bytes") && first.message.contains("16777216"), + "the breach names its axis and the declared limit: {}", + first.message + ); + // Spec §5.11's breach-location table: "source bytes -> the level/header + // span — no body token has been admitted". Pointing at `0..len` would + // highlight sixteen megabytes and call it a location. + assert_eq!( + first.span, + Span { start: 6, end: 7 }, + "the breach points at the level digits, not at the whole document" + ); +} + +#[test] +fn the_token_budget_breaches_end_to_end_as_a_typed_refusal() { + // Four million tokens plus one, spelled in under the byte cap, so the + // token axis is the one that fires. A typed `SWG0509`, not an + // allocation death. + let source = format!( + "swang 2\n\nscore {{\n ppqn 960\n{}}}\n", + "a ".repeat(4_000_001) + ); + let diagnostics = refusal(&source); + assert_eq!(codes(&diagnostics), ["SWG0509"]); + assert!( + diagnostics + .first() + .is_some_and(|d| d.message.contains("tokens")), + "the breach names the token axis" + ); +} + +#[test] +fn a_level_one_source_never_reaches_the_level_two_budget() { + // The budget is level 2's alone (§5.11). A level-1 source larger than + // the level-2 source cap is still parsed by level 1's rules, because + // level 1's acceptance set is frozen and a bound that reached it would + // be a narrowing. + let padding = " ".repeat(16 * 1024 * 1024); + let source = LEVEL_ONE.replace( + "pattern dgd_fractal {", + &format!("pattern dgd_fractal {{\n{padding}"), + ); + assert!( + source.len() > 16 * 1024 * 1024, + "the fixture really is over the level-2 cap" + ); + let document = parse_document(&source).expect("level 1 knows no such bound"); + let Document::Pattern(_) = document else { + panic!("it is still a pattern"); + }; + assert!(parse(&source).is_ok(), "and the frozen entry point agrees"); +}