diff --git a/docs/decisions.log.md b/docs/decisions.log.md index d44b195..e11538e 100644 --- a/docs/decisions.log.md +++ b/docs/decisions.log.md @@ -2477,3 +2477,33 @@ Architectural decisions go to [`adr/`](adr/) instead. reach the evaluator at all: `mod ast` is private to `syntax`, so naming the type from outside is `E0603`. Accepting that the bullet stays open in the backlog until 4A-09 closes it. + +- 2026-08-29 — In the context of SWG-INF-04, facing the question of what an + `AstId` promises, we decided that it is a **parse-local structural + handle** and against any persistence guarantee, to achieve a source map + useful to diagnostics and editors today without pre-deciding a question + Phase 4C owns, accepting that an edit which inserts, deletes or reorders + repeated nodes may renumber every id after it. What is guaranteed: for two + sources that parse to equal ASTs, the maps have the same node-key set and + the same field-key set, though the spans differ. That is deterministic + under whitespace and under §3.2's legal word reordering, which is the + stability an editor actually needs to re-anchor after a reformat. What is + not guaranteed, stated so nobody has to guess: identity across + AST-changing edits, patch identity, a serialized form, a semantic-hash + input, or anything about Phase 4C selector identity. Pretending otherwise + would make 4C's real problem look solved by a side table that never + addressed it. + +- 2026-08-29 — In the context of SWG-INF-04's design, facing three pieces of + prior art, we decided to adopt two and refuse one, to achieve a source map + that is boring in the ways that matter. From `rustc`: byte-offset + locations, with line and column resolved only at render time, so nothing + stores a line number as semantic state. From `rust-analyzer`'s `AstIdMap`: + the separation between structural identity and position-dependent + location — the side-table architecture itself. From `rowan`'s + `SyntaxNodePtr`: the idea is sound prior art for transient source + pointers, but a lossless CST is **not** adopted, because SWG-UI-07 owns + that admission gate and it requires three demonstrated needs that have not + been demonstrated. The borrowed architecture stops short of + `rust-analyzer`'s incremental-IDE identity guarantees, which is the + distinction the entry above exists to make explicit. diff --git a/docs/swang/foundation-backlog.md b/docs/swang/foundation-backlog.md index 95aad3e..7e87a09 100644 --- a/docs/swang/foundation-backlog.md +++ b/docs/swang/foundation-backlog.md @@ -107,7 +107,7 @@ recorded in `decisions.log.md` if reversed. | SWG-INF-01 | Sync the S16 status block with reality *(done)* | docs | — | | SWG-INF-02 | Language level 2 admission contract *(done)* | docs | INF-03 | | SWG-INF-03 | Split `syntax.rs` without behaviour change *(done)* | code | INF-01 | -| SWG-INF-04 | Replace `ProgramSpans` with a `SourceMap` | code | INF-03 | +| SWG-INF-04 | Replace `ProgramSpans` with a `SourceMap` *(done)* | code | INF-03 | | SWG-INF-05 | Deterministic multi-error recovery | code | INF-04 | | SWG-INF-06 | Parser resource gate and differential harness | code | INF-02, INF-03 | | SWG-4A-01 | Normative exact-score-text grammar *(done)* | docs | INF-02 | @@ -288,40 +288,78 @@ than a description of something already coded: no `v2` module, no `Level2Root`, no `GrammarVersion`, no dispatch stub, no `Parsed`, no `SourceMap`, no recovery, no limits, no public token API. -### SWG-INF-04 — Replace `ProgramSpans` with a `SourceMap` +### SWG-INF-04 — Replace `ProgramSpans` with a `SourceMap` *(done)* **Kind:** code. **Depends on:** INF-03 Four spans cannot locate a diagnostic in an exact score text, cannot anchor -a patch selector, and cannot drive an editor action. Shape: +a patch selector, and cannot drive an editor action. As built: ```text struct SourceMap { - nodes: BTreeMap, - fields: BTreeMap, + nodes: BTreeMap, // private + fields: BTreeMap, // private } -enum AstId { Program(u32), Pattern(u32), PipelineStep(u32), Generate(u32) } -enum FieldRef { Node(AstId), Named { node: AstId, field: FieldKind } } +enum AstId { Program(u32), Pattern(u32), Fractalize(u32), Linearize(u32), + MapRhythm(u32), Generate(u32), Export(u32) } +struct FieldRef { node: AstId, field: FieldKind } ``` -Requirements: +Two departures from this entry's original sketch, both deliberate. `AstId` +gets one variant per construct rather than a shared `PipelineStep`, so a +field's owner is named rather than inferred from an ordinal. And `FieldRef` +has no `Node` variant: `SourceMap::node_span` already owns that relation, +and two ways to ask one question is one too many. Both enums are +`#[non_exhaustive]` so level 2 appends variants without breaking a match. + +`FieldKind` is not unique on its own — `Seed` names both the pruning and the +generation seed, and the owning `AstId` tells them apart. A variant per +(construct, word) pair would grow quadratically and say nothing the pair +does not already say. + +Requirements, all met: - spans stay out of AST equality — `parse(format(ast)) == ast` still holds; -- the formatter can consume an AST with no source map at all; +- the formatter consumes an AST with no source map at all, proved by + building a `Program` in Rust with no source text anywhere and reparsing + its output; - the parser returns `Parsed { value, source_map }`; -- a diagnostic points at the value the user must change, not at the - statement containing it; -- every semantically significant level-1 field has a span; +- a diagnostic points at the value, not the statement containing it; +- every semantically significant level-1 field has a span — seven nodes and + eighteen fields in the §3.1 reference, fifteen when pruning and `corpus` + are absent; - every span lies on a UTF-8 boundary inside the source. Acceptance: -- the four existing span tests pass through the new model; -- a witness test enumerates every level-1 AST field and asserts a span for - each — adding a field without a span fails to compile or fails the test; -- reordering words in the source moves the owning span with the value; -- the spec §3.5 formatter laws are unchanged. +- the existing span tests pass through the new model. There were **two**, + not the four this entry claimed, and they are carried across as + characterization rather than rewritten; +- a witness enumerates every level-1 AST field by exhaustive destructuring + with no `..`, so a new AST field without a location classification stops + the suite compiling; +- reordering words moves the owning span with the value while the node and + field key sets stay identical — the useful stability guarantee, stated in + the decision log; +- the spec §3.5 formatter laws are unchanged, and so are the four + diagnostic locations §3.5 released. The map now knows `bars`, + `candidates` and `strategy`; that is editor capability, not permission to + move a diagnostic somebody's tooling already parses; +- eleven mutations, none survived. Three of them were run first against + the suite as it stood at closure and survived it — a node span stretched + back to byte 0, and two swapped arms of the evaluator's flaw-to-location + match — which is how the two review witnesses earned their place rather + than merely occupying it. + +`ProgramSpans` and `parse_with_spans` are removed rather than wrapped: a +compatibility shim would have kept the four-field model alive indefinitely, +which is the thing this task exists to end. + +What this task did **not** do, and must not be read as having done: no +level-2 parsing, no level/root dispatch, no multi-error recovery, no +resource limits, no token API, no persistent identity. `LANGUAGE_LEVEL` +stays 1 and a `swang 2` source still takes the unsupported-level path. ### SWG-INF-05 — Deterministic multi-error recovery @@ -1304,9 +1342,10 @@ INF-01 status sync (done) │ surface over it, not another slice of it │ └─→ 4A-02 → INF-04 → INF-06 → 4A-06 parser skeleton - (done) ↑ - next — INF-04 can now design SourceMap, - AstId and FieldRef against a real v2 shape + (done) (done) ↑ + next — level 2's input bounds must + be declared before its first + accepted program (§5.11) -> 4A-02..4A-09 writer / parser / builder -> 4A-10..4A-14 dump / verify / laws / fuzz -> 4B corpus acceptance diff --git a/swang/src/eval.rs b/swang/src/eval.rs index 2f86ecd..7df7512 100644 --- a/swang/src/eval.rs +++ b/swang/src/eval.rs @@ -28,17 +28,17 @@ use crate::pattern_compile::{ compile_pattern_flaws, PatternFlaw, PatternPlan, RhythmPatternArgs, TailChoice, TraversalChoice, }; use crate::syntax::{ - self, Diagnostic, ExportFormat, PatternDef, Program, ProgramSpans, Span, StrategyName, - StrategyPolicy, + self, AstId, Diagnostic, ExportFormat, FieldKind, FieldRef, PatternDef, Program, SourceMap, + Span, StrategyName, StrategyPolicy, }; -/// A statically-checked program: the parsed AST plus the source spans a +/// A statically-checked program: the parsed AST plus the source locations a /// frontend renders diagnostics at. Text in, structure out — nothing /// resolved, nothing run. #[derive(Debug, Clone, PartialEq, Eq)] pub struct CompiledProgram { program: Program, - spans: ProgramSpans, + source_map: SourceMap, } impl CompiledProgram { @@ -48,10 +48,15 @@ impl CompiledProgram { &self.program } - /// The source-span table. + /// The source-location side table (SWG-INF-04). + /// + /// The whole map, not a four-word projection of it: a frontend that + /// wants to underline `bars` no longer needs a new parser field to do + /// it. What this does **not** license is moving a diagnostic §3.5 + /// already released — see [`flaw_to_diagnostic`]. #[must_use] - pub const fn spans(&self) -> &ProgramSpans { - &self.spans + pub const fn source_map(&self) -> &SourceMap { + &self.source_map } /// The seed-score path the program declares (`generate { source … }`). @@ -170,8 +175,11 @@ impl EvaluationResult { /// # Errors /// The parser's span diagnostics (`SWG0001`–`SWG0404`), never empty on `Err`. pub fn compile_program(source: &str) -> Result> { - let (program, spans) = syntax::parse_with_spans(source)?; - Ok(CompiledProgram { program, spans }) + let parsed = syntax::parse_with_source_map(source)?; + Ok(CompiledProgram { + program: parsed.value, + source_map: parsed.source_map, + }) } /// Runs a compiled program's pattern pipeline up to `map_rhythm`. @@ -190,7 +198,7 @@ pub fn expand_program( let args = rhythm_args(pattern); let bars = clamp_bars(pattern.generate.bars); compile_pattern_flaws(&args, source_score, bars) - .map_err(|flaw| vec![flaw_to_diagnostic(flaw, &compiled.spans)]) + .map_err(|flaw| vec![flaw_to_diagnostic(flaw, &compiled.source_map)]) } /// Runs a compiled program end to end against resolved inputs. @@ -238,7 +246,11 @@ pub fn evaluate_program( .map_err(|e| { vec![EvalDiagnostic { code: "SWG0310", - location: DiagLocation::Span(compiled.spans.source), + location: DiagLocation::Span(released( + &compiled.source_map, + AstId::Generate(0), + FieldKind::Source, + )), message: format!("the source score cannot seed generation: {e:?}"), }] })?; @@ -328,30 +340,50 @@ const fn strategy_kind(name: StrategyName) -> GenerationStrategy { } } +/// One of the four locations §3.5 released, resolved from the map. +/// +/// The map is total over a well-formed program — the source-map contract +/// suite proves all eighteen level-1 fields are present — so the fallback +/// is unreachable. It exists rather than a panic because a location bug +/// should degrade to a worse message, not to a crash in a frontend. +fn released(map: &SourceMap, node: AstId, field: FieldKind) -> Span { + map.field_span(FieldRef::new(node, field)) + .or_else(|| map.node_span(AstId::Program(0))) + .unwrap_or(Span { start: 0, end: 0 }) +} + /// Maps a pattern-compilation flaw to a layered [`EvalDiagnostic`] (spec /// §1.5): structural breaches keep their `NodePath`, score-borne facts sit at /// the `source` word, time-domain flaws at the value that must change. -fn flaw_to_diagnostic(flaw: PatternFlaw, spans: &ProgramSpans) -> EvalDiagnostic { +/// +/// SWG-INF-04 widened what the parser can locate; it deliberately did not +/// widen what this function points at. Each arm resolves exactly the word +/// §3.5 already names, and the richer map is editor capability rather than +/// permission to move a released diagnostic. +fn flaw_to_diagnostic(flaw: PatternFlaw, map: &SourceMap) -> EvalDiagnostic { let at = |span: Span, code: &'static str, message: String| EvalDiagnostic { code, location: DiagLocation::Span(span), message, }; + let kernel = released(map, AstId::Pattern(0), FieldKind::Kernel); + let unit = released(map, AstId::MapRhythm(0), FieldKind::Unit); + let source = released(map, AstId::Generate(0), FieldKind::Source); match flaw { - PatternFlaw::Kernel(d) | PatternFlaw::Density(d) => at(spans.kernel, d.code, d.message), - PatternFlaw::Unit(d) => at(spans.unit, d.code, d.message), - PatternFlaw::Score(d) => at(spans.source, d.code, d.message), + PatternFlaw::Kernel(d) | PatternFlaw::Density(d) => at(kernel, d.code, d.message), + PatternFlaw::Unit(d) => at(unit, d.code, d.message), + PatternFlaw::Score(d) => at(source, d.code, d.message), PatternFlaw::Budget(e) => budget_diagnostic(&e), - PatternFlaw::Lower(e) => lower_diagnostic(&e, spans), + PatternFlaw::Lower(e) => lower_diagnostic(&e, map), PatternFlaw::SilentExpansion => at( - spans.kernel, + kernel, "SWG0306", "the expansion produced no onsets — nothing to generate (change the kernel, \ depth, density, or rhythm seed)" .to_owned(), ), PatternFlaw::SilentWindow { used } => at( - spans.kernel, + kernel, "SWG0306", format!( "the first {used} template(s) the bars window rotates over are all silent — \ @@ -392,11 +424,13 @@ fn budget_diagnostic(e: &griff_pattern::PatternError) -> EvalDiagnostic { } /// A time-domain lowering flaw at the value that must change. -fn lower_diagnostic(e: &crate::LowerError, spans: &ProgramSpans) -> EvalDiagnostic { +fn lower_diagnostic(e: &crate::LowerError, map: &SourceMap) -> EvalDiagnostic { + let unit_span = released(map, AstId::MapRhythm(0), FieldKind::Unit); + let tail_span = released(map, AstId::MapRhythm(0), FieldKind::Tail); match e { crate::LowerError::UnitDoesNotDivideBar { bar_duration, unit } => EvalDiagnostic { code: "SWG0301", - location: DiagLocation::Span(spans.unit), + location: DiagLocation::Span(unit_span), message: format!( "unit {} does not divide the {}-tick bar exactly", unit.0, bar_duration.0 @@ -404,7 +438,7 @@ fn lower_diagnostic(e: &crate::LowerError, spans: &ProgramSpans) -> EvalDiagnost }, crate::LowerError::ZeroUnit => EvalDiagnostic { code: "SWG0301", - location: DiagLocation::Span(spans.unit), + location: DiagLocation::Span(unit_span), message: "the rhythm unit is zero ticks".to_owned(), }, crate::LowerError::IncompleteFinalBar { @@ -412,7 +446,7 @@ fn lower_diagnostic(e: &crate::LowerError, spans: &ProgramSpans) -> EvalDiagnost slots_per_bar, } => EvalDiagnostic { code: "SWG0302", - location: DiagLocation::Span(spans.tail), + location: DiagLocation::Span(tail_span), message: format!( "the final bar holds {have_slots} of {slots_per_bar} slots; a rest_pad tail \ pads it with timed rests" diff --git a/swang/src/syntax.rs b/swang/src/syntax.rs index e484479..ab6280b 100644 --- a/swang/src/syntax.rs +++ b/swang/src/syntax.rs @@ -59,6 +59,7 @@ mod format; mod header; mod lexer; mod parser; +mod source_map; mod span; mod token; @@ -69,7 +70,8 @@ pub use ast::v1::{ pub use diagnostic::Diagnostic; pub use format::v1::format; pub use header::{header_level, LANGUAGE_LEVEL}; -pub use parser::v1::{parse, parse_with_spans, ProgramSpans}; +pub use parser::v1::{parse, parse_with_source_map}; +pub use source_map::{AstId, FieldKind, FieldRef, Parsed, SourceMap}; pub use span::Span; #[cfg(test)] #[allow( diff --git a/swang/src/syntax/parser/v1.rs b/swang/src/syntax/parser/v1.rs index e1e987c..4e95e05 100644 --- a/swang/src/syntax/parser/v1.rs +++ b/swang/src/syntax/parser/v1.rs @@ -9,33 +9,19 @@ use crate::syntax::ast::v1::{ use crate::syntax::diagnostic::Diagnostic; use crate::syntax::header::{header_level, HEADER_WINDOW}; use crate::syntax::lexer::lex; +use crate::syntax::source_map::{AstId, FieldKind, Parsed, SourceMap}; use crate::syntax::span::{span_of, Span}; use crate::syntax::token::{Token, TokenKind}; use crate::TailPolicy; -/// Source locations of the program words an expansion frontend renders -/// diagnostics at (spec §3.5's CLI contract, §1.5's layers). +/// [`parse`], additionally returning the [`SourceMap`] side table. /// -/// This is a side table, not part of the AST: [`Program`] equality and the -/// `parse(format(ast)) == ast` law stay span-free. String-literal spans -/// include their quotes; value spans cover the value token alone. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct ProgramSpans { - /// The quoted `ascii` kernel literal. - pub kernel: Span, - /// The `unit` value (`1/16`). - pub unit: Span, - /// The `tail` value (`reject` / `rest_pad`). - pub tail: Span, - /// The quoted `source` literal. - pub source: Span, -} - -/// [`parse`], additionally returning the [`ProgramSpans`] side table. +/// One parser: [`parse`] is this function with the map dropped, so the two +/// cannot drift in what they accept or how they refuse. /// /// # Errors -/// Exactly [`parse`]'s errors — the two functions are one parser. -pub fn parse_with_spans(source: &str) -> Result<(Program, ProgramSpans), Vec> { +/// Exactly [`parse`]'s errors. +pub fn parse_with_source_map(source: &str) -> Result, Vec> { // `header_level` already enforced 1..=LANGUAGE_LEVEL; the map_err is // defense in depth, not a reachable path. let level = Level::new(header_level(source).map_err(|d| vec![d])?).map_err(|e| { @@ -56,9 +42,46 @@ pub fn parse_with_spans(source: &str) -> Result<(Program, ProgramSpans), Vec` plus one line break, and +/// [`header_level`] has already accepted it by the time this runs, so the +/// digits are the trailing run of the first line. +fn level_span(source: &str) -> Span { + let first_line_end = source + .as_bytes() + .iter() + .take(HEADER_WINDOW) + .position(|&b| b == b'\n') + .unwrap_or(source.len()); + let line = source.get(..first_line_end).unwrap_or(""); + let end = line.trim_end().len(); + let start = line + .get(..end) + .and_then(|head| head.rfind(|c: char| !c.is_ascii_digit())) + .map_or(0, |at| at.saturating_add(1)); + span_of(start, end) } /// Parses a Swang script into its [`Program`]. @@ -76,7 +99,7 @@ pub fn parse_with_spans(source: &str) -> Result<(Program, ProgramSpans), Vec Result> { - parse_with_spans(source).map(|(program, _)| program) + parse_with_source_map(source).map(|parsed| parsed.value) } // ── parsing ────────────────────────────────────────────────────────────── @@ -101,6 +124,7 @@ struct Parser { tokens: Vec, pos: usize, eof: Span, + map: SourceMap, } impl Parser { @@ -151,8 +175,8 @@ impl Parser { } /// `pattern { ascii "…" entries* }` and nothing after it. - fn parse_pattern(&mut self) -> Result<(PatternDef, ProgramSpans), Diagnostic> { - self.expect_word("pattern")?; + fn parse_pattern(&mut self) -> Result { + let keyword = self.expect_word("pattern")?; let name = self.expect_kind(TokenKind::Word, "a pattern name")?; self.expect_kind(TokenKind::OpenBrace, "`{`")?; @@ -174,36 +198,38 @@ impl Parser { else { return Err(self.unexpected_end()); }; - let fractalize = parse_fractalize(fractalize_entry)?; - let linearize = parse_linearize(linearize_entry)?; - let (map_rhythm, unit_span, tail_span) = parse_map_rhythm(map_rhythm_entry)?; - let (generate, source_span) = parse_generate(generate_entry)?; - let export = parse_export(export_entry)?; + let fractalize = parse_fractalize(fractalize_entry, &mut self.map)?; + let linearize = parse_linearize(linearize_entry, &mut self.map)?; + let map_rhythm = parse_map_rhythm(map_rhythm_entry, &mut self.map)?; + let generate = parse_generate(generate_entry, &mut self.map)?; + let export = parse_export(export_entry, &mut self.map)?; - let name = Ident::new(&name.text).map_err(|e| Diagnostic { + let name_ident = Ident::new(&name.text).map_err(|e| Diagnostic { // The lexer reads exactly the identifier charset; defensive. code: "SWG0401", span: name.span, message: e.to_string(), })?; - Ok(( - PatternDef { - name, - kernel: ascii, - fractalize, - linearize, - map_rhythm, - generate, - export, - }, - ProgramSpans { - kernel: kernel_span, - unit: unit_span, - tail: tail_span, - source: source_span, - }, - )) + // The block runs from its `pattern` keyword to its closing brace. + self.map.insert_node( + AstId::Pattern(0), + span_of(keyword.span.start as usize, close.span.end as usize), + ); + self.map + .insert_field(AstId::Pattern(0), FieldKind::Name, name.span); + self.map + .insert_field(AstId::Pattern(0), FieldKind::Kernel, kernel_span); + + Ok(PatternDef { + name: name_ident, + kernel: ascii, + fractalize, + linearize, + map_rhythm, + generate, + export, + }) } /// `ascii ""` — the block's first element. @@ -360,6 +386,15 @@ fn scan_pairs( Ok(pairs) } +/// A pipeline step's own span: its name word through its last argument. +fn entry_span(entry: &PipelineEntry) -> Span { + let end = entry + .args + .last() + .map_or(entry.name_span.end, |token| token.span.end); + span_of(entry.name_span.start as usize, end as usize) +} + /// A required word that never arrived: `SWG0403` at the construct's name. fn missing_word(construct: &str, word: &str, at: Span) -> Diagnostic { Diagnostic { @@ -369,7 +404,7 @@ fn missing_word(construct: &str, word: &str, at: Span) -> Diagnostic { } } -fn parse_fractalize(entry: &PipelineEntry) -> Result { +fn parse_fractalize(entry: &PipelineEntry, map: &mut SourceMap) -> Result { let pairs = scan_pairs( &entry.args, &["depth", "max_cells", "density", "seed"], @@ -379,12 +414,25 @@ fn parse_fractalize(entry: &PipelineEntry) -> Result { let mut max_cells = None; let mut density = None; let mut seed = None; + let mut located: Vec<(FieldKind, Span)> = Vec::new(); for (word, value) in &pairs { match word.text.as_str() { - "depth" => depth = Some(int_value::(value, "depth")?), - "max_cells" => max_cells = Some(int_value::(value, "max_cells")?), - "density" => density = Some((word.span, density_value(value)?)), - _ => seed = Some((word.span, int_value::(value, "seed")?)), + "depth" => { + depth = Some(int_value::(value, "depth")?); + located.push((FieldKind::Depth, value.span)); + } + "max_cells" => { + max_cells = Some(int_value::(value, "max_cells")?); + located.push((FieldKind::MaxCells, value.span)); + } + "density" => { + density = Some((word.span, density_value(value)?)); + located.push((FieldKind::Density, value.span)); + } + _ => { + seed = Some((word.span, int_value::(value, "seed")?)); + located.push((FieldKind::Seed, value.span)); + } } } let depth = depth.ok_or_else(|| missing_word("fractalize", "depth", entry.name_span))?; @@ -412,6 +460,10 @@ fn parse_fractalize(entry: &PipelineEntry) -> Result { } (None, None) => None, }; + map.insert_node(AstId::Fractalize(0), entry_span(entry)); + for (field, span) in located { + map.insert_field(AstId::Fractalize(0), field, span); + } Ok(Fractalize { depth, max_cells, @@ -419,19 +471,22 @@ fn parse_fractalize(entry: &PipelineEntry) -> Result { }) } -fn parse_linearize(entry: &PipelineEntry) -> Result { +fn parse_linearize(entry: &PipelineEntry, map: &mut SourceMap) -> Result { match entry.args.as_slice() { [] => Err(missing_word("linearize", "traversal", entry.name_span)), - [token] => Ok(Linearize { - traversal: closed_set( + [token] => { + let traversal = closed_set( token, &[ ("row_major", Traversal::RowMajor), ("snake", Traversal::Snake), ], "traversal", - )?, - }), + )?; + map.insert_node(AstId::Linearize(0), entry_span(entry)); + map.insert_field(AstId::Linearize(0), FieldKind::Traversal, token.span); + Ok(Linearize { traversal }) + } [_, extra, ..] => Err(Diagnostic { code: "SWG0401", span: extra.span, @@ -440,7 +495,7 @@ fn parse_linearize(entry: &PipelineEntry) -> Result { } } -fn parse_map_rhythm(entry: &PipelineEntry) -> Result<(MapRhythm, Span, Span), Diagnostic> { +fn parse_map_rhythm(entry: &PipelineEntry, map: &mut SourceMap) -> Result { let pairs = scan_pairs(&entry.args, &["unit", "tail"], "map_rhythm")?; let mut unit = None; let mut tail = None; @@ -465,10 +520,13 @@ fn parse_map_rhythm(entry: &PipelineEntry) -> Result<(MapRhythm, Span, Span), Di unit.ok_or_else(|| missing_word("map_rhythm", "unit", entry.name_span))?; let (tail, tail_span) = tail.ok_or_else(|| missing_word("map_rhythm", "tail", entry.name_span))?; - Ok((MapRhythm { unit, tail }, unit_span, tail_span)) + map.insert_node(AstId::MapRhythm(0), entry_span(entry)); + map.insert_field(AstId::MapRhythm(0), FieldKind::Unit, unit_span); + map.insert_field(AstId::MapRhythm(0), FieldKind::Tail, tail_span); + Ok(MapRhythm { unit, tail }) } -fn parse_generate(entry: &PipelineEntry) -> Result<(Generate, Span), Diagnostic> { +fn parse_generate(entry: &PipelineEntry, map: &mut SourceMap) -> Result { let block = match entry.args.as_slice() { [open, inner @ .., close] if open.kind == TokenKind::OpenBrace && close.kind == TokenKind::CloseBrace => @@ -494,43 +552,68 @@ fn parse_generate(entry: &PipelineEntry) -> Result<(Generate, Span), Diagnostic> let mut candidates = None; let mut strategy = None; let mut corpus = None; + let mut located: Vec<(FieldKind, Span)> = Vec::new(); for (word, value) in &pairs { match word.text.as_str() { - "source" => source = Some((string_value(value, "source")?, value.span)), - "bars" => bars = Some(int_value::(value, "bars")?), - "seed" => seed = Some(int_value::(value, "seed")?), - "candidates" => candidates = Some(int_value::(value, "candidates")?), - "strategy" => strategy = Some(strategy_value(value)?), - _ => corpus = Some(string_value(value, "corpus")?), + "source" => { + source = Some((string_value(value, "source")?, value.span)); + located.push((FieldKind::Source, value.span)); + } + "bars" => { + bars = Some(int_value::(value, "bars")?); + located.push((FieldKind::Bars, value.span)); + } + "seed" => { + seed = Some(int_value::(value, "seed")?); + located.push((FieldKind::Seed, value.span)); + } + "candidates" => { + candidates = Some(int_value::(value, "candidates")?); + located.push((FieldKind::Candidates, value.span)); + } + "strategy" => { + strategy = Some(strategy_value(value)?); + located.push((FieldKind::Strategy, value.span)); + } + _ => { + corpus = Some(string_value(value, "corpus")?); + located.push((FieldKind::Corpus, value.span)); + } } } - let (source, source_span) = - source.ok_or_else(|| missing_word("generate", "source", entry.name_span))?; - Ok(( - Generate { - source, - bars: bars.ok_or_else(|| missing_word("generate", "bars", entry.name_span))?, - seed: seed.ok_or_else(|| missing_word("generate", "seed", entry.name_span))?, - candidates: candidates - .ok_or_else(|| missing_word("generate", "candidates", entry.name_span))?, - strategy: strategy - .ok_or_else(|| missing_word("generate", "strategy", entry.name_span))?, - corpus, - }, - source_span, - )) + let (source, _) = source.ok_or_else(|| missing_word("generate", "source", entry.name_span))?; + let generate = Generate { + source, + bars: bars.ok_or_else(|| missing_word("generate", "bars", entry.name_span))?, + seed: seed.ok_or_else(|| missing_word("generate", "seed", entry.name_span))?, + candidates: candidates + .ok_or_else(|| missing_word("generate", "candidates", entry.name_span))?, + strategy: strategy.ok_or_else(|| missing_word("generate", "strategy", entry.name_span))?, + corpus, + }; + map.insert_node(AstId::Generate(0), entry_span(entry)); + for (field, span) in located { + map.insert_field(AstId::Generate(0), field, span); + } + Ok(generate) } -fn parse_export(entry: &PipelineEntry) -> Result { +fn parse_export(entry: &PipelineEntry, map: &mut SourceMap) -> Result { match entry.args.as_slice() { - [format_token, path] => Ok(Export { - format: closed_set( - format_token, - &[("midi", ExportFormat::Midi)], - "export format", - )?, - path: string_value(path, "export path")?, - }), + [format_token, path] => { + let export = Export { + format: closed_set( + format_token, + &[("midi", ExportFormat::Midi)], + "export format", + )?, + path: string_value(path, "export path")?, + }; + map.insert_node(AstId::Export(0), entry_span(entry)); + map.insert_field(AstId::Export(0), FieldKind::Format, format_token.span); + map.insert_field(AstId::Export(0), FieldKind::Path, path.span); + Ok(export) + } _ => Err(Diagnostic { code: "SWG0401", span: entry.name_span, diff --git a/swang/src/syntax/source_map.rs b/swang/src/syntax/source_map.rs new file mode 100644 index 0000000..54fdef7 --- /dev/null +++ b/swang/src/syntax/source_map.rs @@ -0,0 +1,207 @@ +//! Source locations as a side table (SWG-INF-04). +//! +//! The AST is span-free and stays that way: `Program` equality, the +//! `parse(format(ast)) == ast` law, and a lifter that builds programs with +//! no source text in sight all depend on it. Locations therefore live +//! beside the tree rather than inside it: +//! +//! ```text +//! source -> parse_with_source_map -> Parsed { value, source_map } +//! ``` +//! +//! # Prior art, and where this departs from it +//! +//! `rustc` keeps byte offsets and resolves line/column only when it renders; +//! this module does the same, so nothing here stores a line number as +//! semantic state. `rust-analyzer`'s `AstIdMap` separates *structural +//! identity* from *position*, which is the architecture borrowed here. +//! `rowan`'s `SyntaxNodePtr` is the same idea again, but a lossless CST is +//! not adopted — SWG-UI-07 owns that admission gate and has not opened it. +//! +//! The departure from `rust-analyzer` is the one worth stating out loud: its +//! `AstId`s are engineered to survive edits, because an IDE needs them to. +//! [`AstId`] here is **parse-local**. It is deterministic for a given AST +//! topology and blind to whitespace and legal word reordering, and that is +//! the whole guarantee. Inserting, deleting or reordering repeated nodes may +//! renumber every id after the change. Persistent identity across edits is +//! Phase 4C's problem, and pretending to solve it here would be pre-deciding +//! a question that task exists to answer. + +use std::collections::BTreeMap; + +use super::span::Span; + +/// A parse result and the locations of what it parsed. +/// +/// Generic because the level-2 parser will return one of these too, over a +/// different tree; the side-table architecture does not care what `T` is. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Parsed { + /// The tree. Span-free, exactly as `parse` would have returned it. + pub value: T, + /// Where each node and field of that tree came from. + pub source_map: SourceMap, +} + +/// Which construct a location belongs to. +/// +/// The integer is the occurrence ordinal of that kind in semantic AST +/// traversal order. Level 1 has exactly one of each construct, so every +/// ordinal is `0`; the field exists because level 2 has repeated nodes and +/// the identity model should not have to change when it arrives. +/// +/// Non-exhaustive: level 2 appends variants (score, master bar, track, …) +/// without breaking a caller that matches on today's set. +#[non_exhaustive] +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum AstId { + /// The whole program: header through the pattern block. + Program(u32), + /// A `pattern { … }` block. + Pattern(u32), + /// A `fractalize …` pipeline step. + Fractalize(u32), + /// A `linearize …` pipeline step. + Linearize(u32), + /// A `map_rhythm …` pipeline step. + MapRhythm(u32), + /// A `generate { … }` pipeline step. + Generate(u32), + /// An `export …` pipeline step. + Export(u32), +} + +/// Which value of a construct a location belongs to. +/// +/// A kind is not unique on its own: `seed` names both the pruning seed and +/// the generation seed, and the owning [`AstId`] in a [`FieldRef`] is what +/// tells them apart. That is deliberate — the alternative is a variant per +/// (construct, word) pair, which grows quadratically and says nothing the +/// pair does not already say. +/// +/// Non-exhaustive for the same reason as [`AstId`]. +#[non_exhaustive] +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum FieldKind { + /// The header's language level. + Level, + /// A pattern's name. + Name, + /// The quoted `ascii` kernel literal. + Kernel, + /// `fractalize depth`. + Depth, + /// `fractalize max_cells`. + MaxCells, + /// `fractalize density`, including its `bps` suffix. + Density, + /// A seed: pruning under `Fractalize`, generation under `Generate`. + Seed, + /// `linearize`'s traversal word. + Traversal, + /// `map_rhythm unit`, the whole `a/b` token. + Unit, + /// `map_rhythm tail`. + Tail, + /// The quoted `generate { source … }` literal. + Source, + /// `generate { bars … }`. + Bars, + /// `generate { candidates … }`. + Candidates, + /// `generate { strategy … }`. + Strategy, + /// The quoted `generate { corpus … }` literal. + Corpus, + /// `export`'s format word. + Format, + /// The quoted `export` path literal. + Path, +} + +/// One value's address: which construct, and which of its words. +/// +/// There is deliberately no `FieldRef::Node` variant. A construct's own +/// location is [`SourceMap::node_span`]'s business, and two ways to ask the +/// same question is one way too many. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct FieldRef { + node: AstId, + field: FieldKind, +} + +impl FieldRef { + /// Addresses `field` within `node`. + #[must_use] + pub const fn new(node: AstId, field: FieldKind) -> Self { + Self { node, field } + } + + /// The construct that owns the value. + #[must_use] + pub const fn node(self) -> AstId { + self.node + } + + /// Which of the construct's values. + #[must_use] + pub const fn field(self) -> FieldKind { + self.field + } +} + +/// Where every node and value of a parsed tree came from. +/// +/// A node span is the smallest contiguous range covering that construct's +/// syntax, without the whitespace around it. A field span covers the +/// **value**, not the word that names it: a diagnostic about `unit` wants to +/// underline `1/16`, not `unit`. String literals include their quotes, +/// because the quotes are part of the value's spelling. +/// +/// The maps are private. Reading is the whole contract; a caller that could +/// insert into a source map could make it disagree with the text it claims +/// to describe. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct SourceMap { + nodes: BTreeMap, + fields: BTreeMap, +} + +impl SourceMap { + /// Where `id`'s construct sits, if the tree has one. + #[must_use] + pub fn node_span(&self, id: AstId) -> Option { + self.nodes.get(&id).copied() + } + + /// Where `reference`'s value sits. + /// + /// `None` means the word was never written — an omitted `corpus` has no + /// location, as opposed to an empty one. + #[must_use] + pub fn field_span(&self, reference: FieldRef) -> Option { + self.fields.get(&reference).copied() + } + + /// Every located node, in key order. + pub fn nodes(&self) -> impl Iterator + '_ { + self.nodes.iter().map(|(&id, &span)| (id, span)) + } + + /// Every located value, in key order. + pub fn fields(&self) -> impl Iterator + '_ { + self.fields + .iter() + .map(|(&reference, &span)| (reference, span)) + } + + /// Records a construct's location. Crate-private: see the type's note. + pub(crate) fn insert_node(&mut self, id: AstId, span: Span) { + self.nodes.insert(id, span); + } + + /// Records a value's location. + pub(crate) fn insert_field(&mut self, node: AstId, field: FieldKind, span: Span) { + self.fields.insert(FieldRef::new(node, field), span); + } +} diff --git a/swang/src/syntax/tests.rs b/swang/src/syntax/tests.rs index 70ebc5d..9baf954 100644 --- a/swang/src/syntax/tests.rs +++ b/swang/src/syntax/tests.rs @@ -3,9 +3,10 @@ use griff_pattern::{DensityBps, Traversal}; use super::{ - format, header_level, parse, parse_with_spans, AstError, Diagnostic, Export, ExportFormat, - Fractalize, Generate, Ident, KernelLiteral, Level, Linearize, MapRhythm, PatternDef, Program, - Prune, StrategyName, StrategyPolicy, StringLiteral, Unit, LANGUAGE_LEVEL, + format, header_level, parse, parse_with_source_map, AstError, AstId, Diagnostic, Export, + ExportFormat, FieldKind, FieldRef, Fractalize, Generate, Ident, KernelLiteral, Level, + Linearize, MapRhythm, PatternDef, Program, Prune, StrategyName, StrategyPolicy, StringLiteral, + Unit, LANGUAGE_LEVEL, }; use crate::TailPolicy; @@ -445,29 +446,52 @@ fn a_leading_zero_is_swg0401_everywhere_not_only_in_the_header() { .expect("a zero density prunes everything but spells canonically"); } -// ── the span side table (the expand frontend's locations) ────────────── +// ── the source-location side table (the expand frontend's locations) ──── #[test] fn the_span_table_slices_the_source_to_the_owning_words() { - let (program, spans) = parse_with_spans(REFERENCE).expect("the reference parses"); - assert_eq!(program, reference_ast(), "one parser, two entry points"); + // Characterization, carried across SWG-INF-04's migration from + // `ProgramSpans` to `SourceMap` unchanged: these are the four locations + // §3.5 released, and the wider map does not move them. + let parsed = parse_with_source_map(REFERENCE).expect("the reference parses"); + assert_eq!( + parsed.value, + reference_ast(), + "one parser, two entry points" + ); let slice = |span: super::Span| &REFERENCE[span.start as usize..span.end as usize]; - assert_eq!(slice(spans.kernel), "\"X.X/XX./.XX\"", "quotes included"); - assert_eq!(slice(spans.unit), "1/16", "the value token alone"); - assert_eq!(slice(spans.tail), "rest_pad"); + let at = |node: AstId, field: FieldKind| { + slice( + parsed + .source_map + .field_span(FieldRef::new(node, field)) + .expect("a released location"), + ) + }; + assert_eq!( + at(AstId::Pattern(0), FieldKind::Kernel), + "\"X.X/XX./.XX\"", + "quotes included" + ); + assert_eq!( + at(AstId::MapRhythm(0), FieldKind::Unit), + "1/16", + "the value token alone" + ); + assert_eq!(at(AstId::MapRhythm(0), FieldKind::Tail), "rest_pad"); assert_eq!( - slice(spans.source), + at(AstId::Generate(0), FieldKind::Source), "\"corpus/Dance Gavin Dance - The Robot With Human Hair Part 2.gp5\"" ); } #[test] -fn parse_and_parse_with_spans_are_one_parser() { +fn parse_and_parse_with_source_map_are_one_parser() { // Same acceptance and same diagnostics on the same flawed source. let flawed = program_with("|> fractalize depth 1 max_cells 4096 density 9500bps"); assert_eq!( - parse_with_spans(&flawed).expect_err("seedless density")[0].code, + parse_with_source_map(&flawed).expect_err("seedless density")[0].code, first_error(&flawed).code ); } diff --git a/swang/tests/diagnostic_ownership.rs b/swang/tests/diagnostic_ownership.rs new file mode 100644 index 0000000..922df5c --- /dev/null +++ b/swang/tests/diagnostic_ownership.rs @@ -0,0 +1,224 @@ +//! The four diagnostic locations spec §3.5 released, checked end to end. +//! +//! `source_map_contract.rs` proves the **map** holds the right spans. That +//! is only half of the claim. Between the map and a rendered diagnostic sits +//! `eval::flaw_to_diagnostic`, which decides *which* span each flaw class +//! points at — and nothing in the map's own suite can see a mistake there. +//! Swap two arms of that match and every source-map test stays green while +//! a unit error starts underlining the seed path. +//! +//! So these witnesses drive the evaluator, take the `EvalDiagnostic` it +//! actually produces, and slice the source with the span it actually chose: +//! +//! ```text +//! kernel-related -> the quoted `ascii` literal +//! unit -> the `unit` value +//! tail -> the `tail` value +//! score-borne -> the quoted `source` value +//! ``` +//! +//! SWG-INF-04 widened what the parser can locate. It deliberately did not +//! widen what these four point at, and this file is where that stays true. + +// Reason: integration-test code. `unwrap`/`expect`/`panic` abort loudly with +// a clear message, which is exactly what a test harness wants. +#![allow( + clippy::expect_used, + clippy::unwrap_used, + clippy::panic, + clippy::indexing_slicing, + clippy::string_slice, + clippy::missing_assert_message +)] + +use griff_core::event::{Tempo, Ticks, TimeSignature}; +use griff_core::score::{LossReport, MasterBar, RepeatMarker, Score}; +use griff_core::slice::TickRange; +use griff_swang::eval::{compile_program, expand_program, DiagLocation}; + +// ── seed scores ───────────────────────────────────────────────────────────── + +fn bar(index: u64, start: u32, end: u32, numerator: u8, denominator: u8) -> MasterBar { + MasterBar { + index, + tick_range: TickRange::new(Ticks(start), Ticks(end)).expect("ordered range"), + time_signature: TimeSignature::new(numerator, denominator).expect("a meter"), + tempo: Tempo::from_bpm_integer(120).expect("a positive integer BPM"), + repeat: RepeatMarker::default(), + } +} + +fn score(ppqn: u16, master_bars: Vec) -> Score { + Score { + ticks_per_quarter: ppqn, + master_bars, + tracks: Vec::new(), + source_meta: None, + loss: LossReport::new(), + } +} + +/// Two 4/4 bars at PPQN 480 — a 1920-tick bar, which a 1/16 unit divides. +fn four_four() -> Score { + score(480, vec![bar(0, 0, 1920, 4, 4), bar(1, 1920, 3840, 4, 4)]) +} + +/// 7/8 at PPQN 480 is a 1680-tick bar, which a 1/4 unit (480) does not +/// divide — the shape the CLI suite already uses for `SWG0301`. +fn seven_eight() -> Score { + score(480, vec![bar(0, 0, 1680, 7, 8), bar(1, 1680, 3360, 7, 8)]) +} + +/// A meter change: score-borne, and the fix lives in the seed file rather +/// than anywhere in the program. +fn meter_change() -> Score { + score(480, vec![bar(0, 0, 1920, 4, 4), bar(1, 1920, 3600, 7, 8)]) +} + +// ── programs ──────────────────────────────────────────────────────────────── + +fn program(kernel: &str, fractalize: &str, unit: &str, tail: &str, bars: u64) -> String { + format!( + "swang 1\n\npattern p {{\n ascii \"{kernel}\"\n |> fractalize {fractalize}\n \ + |> linearize snake\n |> map_rhythm unit {unit} tail {tail}\n |> generate {{\n \ + source \"seed.gp5\"\n bars {bars}\n seed 42\n candidates 2\n \ + strategy auto\n }}\n |> export midi \"out.mid\"\n}}\n" + ) +} + +/// The one diagnostic an expansion refused with, and the source it slices. +fn refusal(source: &str, seed: &Score) -> (&'static str, String) { + let compiled = compile_program(source).expect("the program parses"); + let flaws = expand_program(&compiled, seed).expect_err("this expansion must refuse"); + let first = flaws.first().expect("a refusal carries a diagnostic"); + let DiagLocation::Span(span) = &first.location else { + panic!("expected a span location, got {:?}", first.location); + }; + ( + first.code, + source[span.start as usize..span.end as usize].to_owned(), + ) +} + +// ── the four released owners ──────────────────────────────────────────────── + +#[test] +fn a_score_borne_fact_points_at_the_quoted_source_value() { + // The meter changes inside the seed file. Nothing in the program is + // wrong; the location names the score that is. + let source = program( + "X.X/XX./.XX", + "depth 1 max_cells 4096", + "1/16", + "rest_pad", + 8, + ); + let (code, sliced) = refusal(&source, &meter_change()); + assert_eq!(code, "SWG0304"); + assert_eq!( + sliced, "\"seed.gp5\"", + "score-borne facts sit at the `source` value, quotes included" + ); +} + +#[test] +fn a_unit_that_does_not_divide_the_bar_points_at_the_unit_value() { + // 1/4 is 480 ticks; the 7/8 bar is 1680, which 480 does not divide. + let source = program( + "X.X/XX./.XX", + "depth 1 max_cells 4096", + "1/4", + "rest_pad", + 8, + ); + let (code, sliced) = refusal(&source, &seven_eight()); + assert_eq!(code, "SWG0301"); + assert_eq!(sliced, "1/4", "the unit value, not the `unit` word"); +} + +#[test] +fn an_incomplete_final_bar_points_at_the_tail_value() { + // Nine cells into sixteen-slot bars, with a tail that refuses to pad. + let source = program("X.X/XX./.XX", "depth 0 max_cells 32", "1/16", "reject", 8); + let (code, sliced) = refusal(&source, &four_four()); + assert_eq!(code, "SWG0302"); + assert_eq!( + sliced, "reject", + "the tail value is what the author must change" + ); +} + +#[test] +fn a_silent_expansion_points_at_the_kernel_literal() { + // One onset, seventeen cells, one bar: the window never reaches it. + let source = program( + "................X", + "depth 0 max_cells 32", + "1/16", + "rest_pad", + 1, + ); + let (code, sliced) = refusal(&source, &four_four()); + assert_eq!(code, "SWG0306"); + assert_eq!( + sliced, "\"................X\"", + "kernel-related facts sit at the quoted literal, quotes included" + ); +} + +// ── the ownership really is four *different* places ───────────────────────── + +#[test] +fn the_four_owners_are_four_distinct_locations() { + // The point of the file, stated as one assertion: swapping any two arms + // of the evaluator's match makes two of these collide. A witness that + // only checked "some span came back" would not notice. + let mut sliced = vec![ + refusal( + &program( + "X.X/XX./.XX", + "depth 1 max_cells 4096", + "1/16", + "rest_pad", + 8, + ), + &meter_change(), + ) + .1, + refusal( + &program( + "X.X/XX./.XX", + "depth 1 max_cells 4096", + "1/4", + "rest_pad", + 8, + ), + &seven_eight(), + ) + .1, + refusal( + &program("X.X/XX./.XX", "depth 0 max_cells 32", "1/16", "reject", 8), + &four_four(), + ) + .1, + refusal( + &program( + "................X", + "depth 0 max_cells 32", + "1/16", + "rest_pad", + 1, + ), + &four_four(), + ) + .1, + ]; + let before = sliced.len(); + sliced.sort(); + sliced.dedup(); + assert_eq!( + sliced.len(), + before, + "two flaw classes resolved to the same word: {sliced:?}" + ); +} diff --git a/swang/tests/source_map_contract.rs b/swang/tests/source_map_contract.rs new file mode 100644 index 0000000..29a7935 --- /dev/null +++ b/swang/tests/source_map_contract.rs @@ -0,0 +1,656 @@ +//! SWG-INF-04: the source map's contract, stated before it exists. +//! +//! `ProgramSpans` located four words because four were all the expansion +//! frontend needed. That is a special case wearing a struct: every other +//! value in a program is unlocatable, and each new one would have meant +//! another field. This suite states the general replacement: +//! +//! ```text +//! source -> parse_with_source_map -> Parsed { value, source_map } +//! ``` +//! +//! Three things it is emphatically not. It is not part of the AST — a +//! `Program` built in memory still formats and reparses with no source text +//! anywhere. It is not a licence to improve frozen level-1 diagnostics, +//! which keep the spans §3.5 already released. And its `AstId`s are not +//! persistent identities: they are parse-local handles, stable across +//! whitespace and legal word reordering, and Phase 4C still owns the +//! question of identity across edits. + +// Reason: integration-test code. `unwrap`/`expect`/`panic` abort loudly with +// a clear message, which is exactly what a test harness wants. +#![allow( + clippy::unwrap_used, + clippy::expect_used, + clippy::panic, + clippy::missing_assert_message, + clippy::indexing_slicing, + clippy::string_slice +)] + +use griff_pattern::{DensityBps, Traversal}; +use griff_swang::syntax::{ + format, parse, parse_with_source_map, AstId, Export, ExportFormat, FieldKind, FieldRef, + Fractalize, Generate, Ident, KernelLiteral, Level, Linearize, MapRhythm, Parsed, PatternDef, + Program, Prune, Span, StrategyName, StrategyPolicy, StringLiteral, Unit, +}; +use griff_swang::TailPolicy; + +/// The spec §3.1 reference program — every optional word present, so the +/// census sees the widest level-1 shape there is. +const REFERENCE: &str = r#"swang 1 + +pattern dgd_fractal { + ascii "X.X/XX./.XX" + |> fractalize depth 1 max_cells 4096 density 9500bps seed 4 + |> linearize snake + |> map_rhythm unit 1/16 tail rest_pad + |> generate { + source "corpus/Dance Gavin Dance - The Robot With Human Hair Part 2.gp5" + bars 8 + seed 42 + candidates 2 + strategy repeat_variation + corpus "corpus" + } + |> export midi "dgd_fractal_dense.mid" +} +"#; + +/// The same program with neither pruning nor a corpus — the narrowest shape. +const MINIMAL: &str = r#"swang 1 + +pattern p { + ascii "X.X/XX./.XX" + |> fractalize depth 1 max_cells 4096 + |> linearize snake + |> map_rhythm unit 1/16 tail rest_pad + |> generate { + source "seed.gp5" + bars 8 + seed 42 + candidates 2 + strategy auto + } + |> export midi "out.mid" +} +"#; + +fn slice(source: &str, span: Span) -> &str { + &source[span.start as usize..span.end as usize] +} + +// ── A. the full reference census ──────────────────────────────────────────── + +/// Every level-1 AST field, classified: it either carries a **field span** +/// of its own, or is a composite located by its child's **node span**. +/// +/// The destructuring is exhaustive and uses no `..` on purpose. Adding a +/// field to any level-1 AST struct stops this compiling, which forces the +/// choice to be made deliberately rather than defaulted into "unlocatable". +fn classify(program: &Program) -> (Vec, Vec) { + let mut fields = Vec::new(); + let mut nodes = vec![AstId::Program(0)]; + + let Program { level, pattern } = program; + let _: &Level = level; + fields.push(FieldRef::new(AstId::Program(0), FieldKind::Level)); + // `pattern` is composite: located by the Pattern node span. + + let PatternDef { + name, + kernel, + fractalize, + linearize, + map_rhythm, + generate, + export, + } = pattern; + let _: &Ident = name; + let _: &KernelLiteral = kernel; + nodes.push(AstId::Pattern(0)); + fields.push(FieldRef::new(AstId::Pattern(0), FieldKind::Name)); + fields.push(FieldRef::new(AstId::Pattern(0), FieldKind::Kernel)); + + let Fractalize { + depth, + max_cells, + prune, + } = fractalize; + let _: &u8 = depth; + let _: &u64 = max_cells; + nodes.push(AstId::Fractalize(0)); + fields.push(FieldRef::new(AstId::Fractalize(0), FieldKind::Depth)); + fields.push(FieldRef::new(AstId::Fractalize(0), FieldKind::MaxCells)); + // `clippy::unneeded_field_pattern` would have this be `Prune { .. }`. + // That is precisely what must not happen: naming every field is how a + // new one becomes a compile error here instead of silently unlocatable. + #[allow(clippy::unneeded_field_pattern)] + if let Some(Prune { + density: _, + seed: _, + }) = prune + { + fields.push(FieldRef::new(AstId::Fractalize(0), FieldKind::Density)); + fields.push(FieldRef::new(AstId::Fractalize(0), FieldKind::Seed)); + } + + let Linearize { traversal } = linearize; + let _ = traversal; + nodes.push(AstId::Linearize(0)); + fields.push(FieldRef::new(AstId::Linearize(0), FieldKind::Traversal)); + + let MapRhythm { unit, tail } = map_rhythm; + let _: &Unit = unit; + let _ = tail; + nodes.push(AstId::MapRhythm(0)); + fields.push(FieldRef::new(AstId::MapRhythm(0), FieldKind::Unit)); + fields.push(FieldRef::new(AstId::MapRhythm(0), FieldKind::Tail)); + + let Generate { + source, + bars, + seed, + candidates, + strategy, + corpus, + } = generate; + let _: &StringLiteral = source; + let _: &u64 = bars; + let _: &u64 = seed; + let _: &u64 = candidates; + let _: &StrategyPolicy = strategy; + nodes.push(AstId::Generate(0)); + fields.push(FieldRef::new(AstId::Generate(0), FieldKind::Source)); + fields.push(FieldRef::new(AstId::Generate(0), FieldKind::Bars)); + // The generation seed shares its `FieldKind` with the pruning seed; the + // owning `AstId` is what tells them apart. + fields.push(FieldRef::new(AstId::Generate(0), FieldKind::Seed)); + fields.push(FieldRef::new(AstId::Generate(0), FieldKind::Candidates)); + fields.push(FieldRef::new(AstId::Generate(0), FieldKind::Strategy)); + if corpus.is_some() { + fields.push(FieldRef::new(AstId::Generate(0), FieldKind::Corpus)); + } + + let Export { format: fmt, path } = export; + let _ = fmt; + let _: &StringLiteral = path; + nodes.push(AstId::Export(0)); + fields.push(FieldRef::new(AstId::Export(0), FieldKind::Format)); + fields.push(FieldRef::new(AstId::Export(0), FieldKind::Path)); + + (fields, nodes) +} + +#[test] +fn the_reference_program_maps_seven_nodes_and_eighteen_fields() { + let parsed = parse_with_source_map(REFERENCE).expect("the reference parses"); + let (expected_fields, expected_nodes) = classify(&parsed.value); + + assert_eq!(expected_nodes.len(), 7, "the level-1 node kinds"); + assert_eq!(expected_fields.len(), 18, "the level-1 value fields"); + + let mapped_nodes: Vec = parsed.source_map.nodes().map(|(id, _)| id).collect(); + let mapped_fields: Vec = parsed.source_map.fields().map(|(r, _)| r).collect(); + assert_eq!( + mapped_nodes.len(), + 7, + "every node is located: {mapped_nodes:?}" + ); + assert_eq!( + mapped_fields.len(), + 18, + "every value field is located: {mapped_fields:?}" + ); + + for id in expected_nodes { + assert!( + parsed.source_map.node_span(id).is_some(), + "no node span for {id:?}" + ); + } + for reference in expected_fields { + assert!( + parsed.source_map.field_span(reference).is_some(), + "no field span for {reference:?}" + ); + } +} + +#[test] +fn every_reference_field_slices_back_to_its_author_value() { + let parsed = parse_with_source_map(REFERENCE).expect("the reference parses"); + let at = |node: AstId, field: FieldKind| { + let span = parsed + .source_map + .field_span(FieldRef::new(node, field)) + .unwrap_or_else(|| panic!("no span for {node:?}/{field:?}")); + slice(REFERENCE, span) + }; + + assert_eq!(at(AstId::Program(0), FieldKind::Level), "1", "digits only"); + assert_eq!(at(AstId::Pattern(0), FieldKind::Name), "dgd_fractal"); + assert_eq!( + at(AstId::Pattern(0), FieldKind::Kernel), + "\"X.X/XX./.XX\"", + "quotes included" + ); + assert_eq!(at(AstId::Fractalize(0), FieldKind::Depth), "1"); + assert_eq!(at(AstId::Fractalize(0), FieldKind::MaxCells), "4096"); + assert_eq!( + at(AstId::Fractalize(0), FieldKind::Density), + "9500bps", + "the complete value token, suffix included" + ); + assert_eq!( + at(AstId::Fractalize(0), FieldKind::Seed), + "4", + "the pruning seed" + ); + assert_eq!(at(AstId::Linearize(0), FieldKind::Traversal), "snake"); + assert_eq!( + at(AstId::MapRhythm(0), FieldKind::Unit), + "1/16", + "the whole rational token" + ); + assert_eq!(at(AstId::MapRhythm(0), FieldKind::Tail), "rest_pad"); + assert_eq!( + at(AstId::Generate(0), FieldKind::Source), + "\"corpus/Dance Gavin Dance - The Robot With Human Hair Part 2.gp5\"" + ); + assert_eq!(at(AstId::Generate(0), FieldKind::Bars), "8"); + assert_eq!( + at(AstId::Generate(0), FieldKind::Seed), + "42", + "the generation seed, told apart by its owning node" + ); + assert_eq!(at(AstId::Generate(0), FieldKind::Candidates), "2"); + assert_eq!( + at(AstId::Generate(0), FieldKind::Strategy), + "repeat_variation" + ); + assert_eq!(at(AstId::Generate(0), FieldKind::Corpus), "\"corpus\""); + assert_eq!(at(AstId::Export(0), FieldKind::Format), "midi"); + assert_eq!( + at(AstId::Export(0), FieldKind::Path), + "\"dgd_fractal_dense.mid\"" + ); +} + +#[test] +fn a_node_span_contains_every_field_span_it_owns() { + // The weaker structural claim, but the one that catches a node span + // built from the wrong construct entirely. + let parsed = parse_with_source_map(REFERENCE).expect("the reference parses"); + for (reference, field_span) in parsed.source_map.fields() { + let node_span = parsed + .source_map + .node_span(reference.node()) + .expect("a located field's node is located"); + assert!( + node_span.start <= field_span.start && field_span.end <= node_span.end, + "{reference:?} at {field_span:?} escapes its node {node_span:?}" + ); + } +} + +#[test] +fn every_node_span_covers_exactly_its_construct() { + // Containment alone is far too weak: a node span of `0..source.len()` + // contains every field it owns and is wrong about all of them. The + // documented rule is the *smallest* contiguous range covering the + // construct's syntax, so the only honest witness is the slice itself. + let parsed = parse_with_source_map(REFERENCE).expect("the reference parses"); + let at = |id: AstId| { + slice( + REFERENCE, + parsed + .source_map + .node_span(id) + .unwrap_or_else(|| panic!("no node span for {id:?}")), + ) + }; + + assert_eq!( + at(AstId::Fractalize(0)), + "fractalize depth 1 max_cells 4096 density 9500bps seed 4", + "a pipeline step runs from its name word to its last argument" + ); + assert_eq!(at(AstId::Linearize(0)), "linearize snake"); + assert_eq!( + at(AstId::MapRhythm(0)), + "map_rhythm unit 1/16 tail rest_pad" + ); + assert_eq!( + at(AstId::Export(0)), + "export midi \"dgd_fractal_dense.mid\"", + "the path literal is the last argument, quotes included" + ); + + let generate = at(AstId::Generate(0)); + assert!( + generate.starts_with("generate {"), + "the generate step opens at its own word: {generate:?}" + ); + assert!( + generate.ends_with('}'), + "and closes at its block's brace, not at the last word inside it: {generate:?}" + ); + assert!( + generate.contains("corpus \"corpus\""), + "with the whole block between: {generate:?}" + ); + + let pattern = at(AstId::Pattern(0)); + assert!( + pattern.starts_with("pattern dgd_fractal {"), + "the block opens at its keyword: {pattern:?}" + ); + assert!(pattern.ends_with('}'), "and closes at its own brace"); + assert!( + !pattern.starts_with("swang"), + "the header is the program's, not the pattern's" + ); + + let program = at(AstId::Program(0)); + assert!( + program.starts_with("swang 1"), + "the program opens at its header: {program:?}" + ); + assert!( + program.ends_with('}'), + "and ends at the pattern's closing brace, not at the trailing newline" + ); + assert_eq!( + program.len(), + REFERENCE.trim_end().len(), + "the program covers the source but for the trailing newline" + ); +} + +// ── B. optionality ────────────────────────────────────────────────────────── + +#[test] +fn omitted_words_get_no_phantom_spans() { + let parsed = parse_with_source_map(MINIMAL).expect("the minimal program parses"); + let (expected_fields, _) = classify(&parsed.value); + assert_eq!( + expected_fields.len(), + 15, + "no prune and no corpus is three fields fewer" + ); + assert_eq!(parsed.source_map.fields().count(), 15); + + for absent in [ + FieldRef::new(AstId::Fractalize(0), FieldKind::Density), + FieldRef::new(AstId::Fractalize(0), FieldKind::Seed), + FieldRef::new(AstId::Generate(0), FieldKind::Corpus), + ] { + assert!( + parsed.source_map.field_span(absent).is_none(), + "{absent:?} was never written, so it has no location" + ); + } + // The generation seed is still there — it is a different owner. + assert!(parsed + .source_map + .field_span(FieldRef::new(AstId::Generate(0), FieldKind::Seed)) + .is_some()); +} + +// ── C. legal reordering ───────────────────────────────────────────────────── + +#[test] +fn reordered_words_keep_the_key_set_and_move_only_the_spans() { + // §3.2 lets the words of a construct arrive in any order; the canonical + // formatter decides the output order. Two such sources are the same + // program, so they must have the same handles — and different bytes. + let a = MINIMAL; + let b = r#"swang 1 + +pattern p { + ascii "X.X/XX./.XX" + |> fractalize max_cells 4096 depth 1 + |> linearize snake + |> map_rhythm tail rest_pad unit 1/16 + |> generate { + bars 8 + candidates 2 + source "seed.gp5" + strategy auto + seed 42 + } + |> export midi "out.mid" +} +"#; + + let pa = parse_with_source_map(a).expect("a parses"); + let pb = parse_with_source_map(b).expect("b parses"); + assert_eq!(pa.value, pb.value, "legal reordering is the same program"); + + let keys = |p: &Parsed| { + ( + p.source_map.nodes().map(|(id, _)| id).collect::>(), + p.source_map.fields().map(|(r, _)| r).collect::>(), + ) + }; + assert_eq!(keys(&pa), keys(&pb), "equal ASTs, equal handles"); + + // The deliberately moved words must slice to the same value from + // different bytes — a map built from canonical field order instead of + // the author's tokens would give identical spans here. + for (node, field) in [ + (AstId::Fractalize(0), FieldKind::Depth), + (AstId::MapRhythm(0), FieldKind::Unit), + (AstId::Generate(0), FieldKind::Source), + ] { + let sa = pa + .source_map + .field_span(FieldRef::new(node, field)) + .expect("present in a"); + let sb = pb + .source_map + .field_span(FieldRef::new(node, field)) + .expect("present in b"); + assert_ne!(sa, sb, "{node:?}/{field:?} did not move, but its word did"); + assert_eq!( + slice(a, sa), + slice(b, sb), + "{node:?}/{field:?} must still name the same value" + ); + } +} + +// ── D. UTF-8 boundaries ───────────────────────────────────────────────────── + +#[test] +fn every_span_lands_on_a_char_boundary_in_a_multibyte_source() { + // Multibyte text inside the string literals, which is where level 1 + // lets arbitrary UTF-8 live. A map computed in `chars` rather than + // bytes slices mid-character or out of range here. + let source = "swang 1\n\npattern p {\n ascii \"X.X/XX./.XX\"\n \ + |> fractalize depth 1 max_cells 4096\n |> linearize snake\n \ + |> map_rhythm unit 1/16 tail rest_pad\n |> generate {\n \ + source \"корпус/Ω音楽 — café.gp5\"\n bars 8\n seed 42\n \ + candidates 2\n strategy auto\n corpus \"кор—пус\"\n }\n \ + |> export midi \"вы—ход.mid\"\n}\n"; + let parsed = parse_with_source_map(source).expect("multibyte literals parse"); + + let mut checked = 0_usize; + for (id, span) in parsed.source_map.nodes() { + assert!(span.start <= span.end, "{id:?} is inverted"); + assert!( + span.end as usize <= source.len(), + "{id:?} runs past the source" + ); + assert!(source.is_char_boundary(span.start as usize), "{id:?} start"); + assert!(source.is_char_boundary(span.end as usize), "{id:?} end"); + checked += 1; + } + for (reference, span) in parsed.source_map.fields() { + assert!(span.start <= span.end, "{reference:?} is inverted"); + assert!( + span.end as usize <= source.len(), + "{reference:?} runs past the source" + ); + assert!( + source.is_char_boundary(span.start as usize), + "{reference:?} start" + ); + assert!( + source.is_char_boundary(span.end as usize), + "{reference:?} end" + ); + checked += 1; + } + assert!(checked >= 20, "only {checked} spans examined"); + + assert_eq!( + parsed + .source_map + .field_span(FieldRef::new(AstId::Generate(0), FieldKind::Source)) + .map(|s| slice(source, s)), + Some("\"корпус/Ω音楽 — café.gp5\""), + "the multibyte literal slices whole, quotes included" + ); +} + +// ── E. one parser ─────────────────────────────────────────────────────────── + +#[test] +fn the_two_entry_points_accept_the_same_programs() { + for source in [REFERENCE, MINIMAL] { + let plain = parse(source).expect("parses"); + let mapped = parse_with_source_map(source).expect("parses").value; + assert_eq!(plain, mapped, "one parser, two entry points"); + } +} + +#[test] +fn the_two_entry_points_refuse_identically() { + // Not just "both fail": the same codes, messages, spans, and order. + let flawed = [ + "swang 1\n\npattern p {\n ascii \"X.X/XX./.XX\"\n \ + |> fractalize depth 1 max_cells 4096 density 9500bps\n |> linearize snake\n \ + |> map_rhythm unit 1/16 tail rest_pad\n |> generate {\n \ + source \"s.gp5\"\n bars 8\n seed 42\n candidates 2\n \ + strategy auto\n }\n |> export midi \"o.mid\"\n}\n", + "swang 9\n", + "\u{feff}swang 1\n", + "swang 1\n\npattern p {\n}\n", + "not a header at all\n", + ]; + for source in flawed { + let a = parse(source).expect_err("refused"); + let b = parse_with_source_map(source).expect_err("refused identically"); + assert_eq!( + a.len(), + b.len(), + "same number of diagnostics for {source:?}" + ); + for (x, y) in a.iter().zip(b.iter()) { + assert_eq!(x.code, y.code); + assert_eq!(x.message, y.message); + assert_eq!(x.span, y.span); + } + } +} + +// ── F. the formatter needs no map ─────────────────────────────────────────── + +#[test] +fn an_ast_built_in_memory_still_formats_and_reparses() { + // No source text, no source map, no parse: a lifter constructs programs + // directly. If the formatter ever needed a map, this would not compile. + let program = Program { + level: Level::new(1).expect("this build's level"), + pattern: PatternDef { + name: Ident::new("built_by_hand").expect("a name"), + kernel: KernelLiteral::new("X.X/XX./.XX").expect("the spec kernel"), + fractalize: Fractalize { + depth: 1, + max_cells: 4096, + prune: Some(Prune { + density: DensityBps::new(9500).expect("in scale"), + seed: 4, + }), + }, + linearize: Linearize { + traversal: Traversal::Snake, + }, + map_rhythm: MapRhythm { + unit: Unit::new(1, 16).expect("a note value"), + tail: TailPolicy::RestPad, + }, + generate: Generate { + source: StringLiteral::new("seed.gp5").expect("a path"), + bars: 8, + seed: 42, + candidates: 2, + strategy: StrategyPolicy::Named(StrategyName::RepeatVariation), + corpus: None, + }, + export: Export { + format: ExportFormat::Midi, + path: StringLiteral::new("out.mid").expect("a path"), + }, + }, + }; + + let text = format(&program); + assert_eq!( + parse(&text).expect("the formatter's output parses"), + program, + "parse(format(ast)) == ast, with no source map in sight" + ); + assert_eq!( + format(&parse(&text).expect("reparses")), + text, + "and it is a fixed point" + ); +} + +// ── G. frozen diagnostic ownership ────────────────────────────────────────── + +#[test] +fn the_four_released_diagnostic_locations_are_unchanged() { + // §3.5 already released these four locations. The map now knows `bars`, + // `candidates`, `strategy` and the rest too — that is editor capability, + // not permission to move a diagnostic somebody's tooling already parses. + let parsed = parse_with_source_map(REFERENCE).expect("the reference parses"); + let at = |node: AstId, field: FieldKind| { + slice( + REFERENCE, + parsed + .source_map + .field_span(FieldRef::new(node, field)) + .expect("released location"), + ) + }; + assert_eq!(at(AstId::Pattern(0), FieldKind::Kernel), "\"X.X/XX./.XX\""); + assert_eq!(at(AstId::MapRhythm(0), FieldKind::Unit), "1/16"); + assert_eq!(at(AstId::MapRhythm(0), FieldKind::Tail), "rest_pad"); + assert_eq!( + at(AstId::Generate(0), FieldKind::Source), + "\"corpus/Dance Gavin Dance - The Robot With Human Hair Part 2.gp5\"" + ); +} + +// ── determinism of the iteration order ────────────────────────────────────── + +#[test] +fn iteration_order_is_deterministic_and_sorted() { + let parsed = parse_with_source_map(REFERENCE).expect("the reference parses"); + let once: Vec = parsed.source_map.nodes().map(|(id, _)| id).collect(); + let twice: Vec = parsed.source_map.nodes().map(|(id, _)| id).collect(); + assert_eq!(once, twice, "two walks of one map agree"); + + let mut sorted = once.clone(); + sorted.sort(); + assert_eq!(once, sorted, "nodes come out in key order"); + + let fields: Vec = parsed.source_map.fields().map(|(r, _)| r).collect(); + let mut sorted_fields = fields.clone(); + sorted_fields.sort(); + assert_eq!(fields, sorted_fields, "fields come out in key order"); +}