diff --git a/ROADMAP.md b/ROADMAP.md index 35ba6d4..49270ea 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,236 +1,116 @@ -# RustQ Roadmap — Next Generation - -Strategic plan to take RustQ from v0.5.x ("Rusty Elixir MVP") to a polished, -powerful typed-macro metalanguage for Rust codegen. - -State of this document reflects review on 2026-06-19 against `defrust-meta-mvp` -and the skia dogfood (`../skia_ex`). - -## Where we are - -- **Wave 1 is complete.** Semantic helpers lower through real RustQ AST, - native AST rendering is mandatory, `strict_native_ast` is gone, and Skia has - been validated against `semantic-escapes-ast` after the namespace cleanup. -- **Wave 2 is mostly complete.** Schema/native decoder/render coherence is - tested from `AST.Schema.nodes()`, rendering/build/lowering failures now raise - structured diagnostics, and tests mirror the source layout. The remaining Wave - 2 gap is a full golden lowering corpus. -- **Module structure has been cleaned up beyond the original roadmap.** - `NativeCodegen` is now `Codegen`, external Rust metadata lives under - `RustQ.Native.*`, Rustler helper modules are concept-oriented, `RustQ.Meta` is - split into focused submodules, config state lives in `RustQ.Config.State`, and - every module has real documentation instead of `@moduledoc false`. -- **Skia proves the dogfood.** `Skia.Codegen.Rusty.*` author drawing semantics - in `defrust` + ordinary `quote`/`unquote` with **zero** `expr!`/`pat!`/`stmt!`/ - `arm!`/`raw_*` escapes in semantic bodies. ~3.3k of 4k native lines are - generated; CI gates on `mix rustq.gen --check`. The ownership boundary (Skia - owns semantics, RustQ owns lowering) holds in practice. -- **Discipline is codified** in `AGENTS.md`; architecture enforcement belongs in - Reach/ExDNA/Credo/custom linting or schema-driven behavioral tests, not ad hoc - ExUnit source-grep tests. - -## Gaps (evidence → fix) - -| # | Gap | Evidence | Addressed in | -|---|-----|----------|--------------| -| A | `semantic_expr`/`semantic_pat` build Rust source strings that round-trip through `parse_syn`, even for shapes the mainstream `lower_expr` already emits as AST nodes. Parallel path; violates AGENTS #7. | `lib/rustq/meta/lower.ex:564–700`; skia never uses `expr!`, so this path is undertested | Wave 1 | -| B | Silent render fallback unless `strict_native_ast: true`. A render gap can emit subtly wrong Rust in real builds. | `lib/rustq/rust/ast/render.ex:106–117` | Wave 1 | -| C | No coherence guarantee between `defnode` ↔ Elixir renderer ↔ native decoder ↔ `Meta.Lower`. A new node must be wired in ~4 places with no check. Biggest scaling risk. | `ast/schema.ex` + `decoders/*.ex` | Wave 2 | -| D | No `RustQ.Syn` → Rusty-Elixir binding adapter. Wrapping a crate means hand-authoring every `defrust` shell. | `skia/codegen/skia_safe.ex` + `commands/*` | Wave 3 | -| E | Lowering completeness is anecdotal. `lower.ex` raises `unsupported` in several spots; no golden corpus. | grep `unsupported` in `lower.ex` | Wave 2 | -| F | Mutability/lifetime inference is heuristic; lifetimes come only from `R.lifetime(:a)` in specs, never inferred in bodies. | `lower.ex:790–845` | Wave 3 | -| G | No crate-level manifest. `rustq.exs` is file-list-oriented; skia hand-rolls 19 targets via `generated_targets/0`. | `skia/codegen.ex:11–60` | Wave 4 | -| H | Diagnostics are bare `ArgumentError`s. No spans, no offending-node reporting. | `lower.ex` raises | Wave 2 | -| I | No native-render dogfood gate across RustQ + skia proving every lowered body renders through native AST without fallback. | `RustQ.Native.render_ast/1` is mandatory in RustQ; skia still needs an explicit cross-project gate | Wave 1 / skia follow-up | -| **J** | **`unwrap!` noise.** ~190 uses in skia; the macro name is semantically wrong (`unwrap`/`unwrap()` means *panic* in both Rust and Elixir, but it generates `?` = *propagate*), and the `?` it spells is fully derivable from the called function's `@spec` return type vs the expected type at the call position. Author is forced to restate type information the compiler already has. | `grep -r unwrap! skia`; every decode helper has a `Result`/`Option` `@spec` | **Wave 3** | - -## The headline insight: type-driven propagation inference - -`unwrap!(expr) → expr?` is **syntax-to-syntax** lowering. Rust's `?` is -*mandatory* on the Rust surface because the compiler rejects -`let x = f()` when `f: -> Result`. But that mandate does **not** exist at the -Elixir/meta layer: RustQ has the called function's return type (from `@spec`, -confirmed for every skia decode helper) and the expected type at the call -position. Whether `?` is needed is fully determined by those types. Forcing the -author to spell it is making them transcribe a Rust-language requirement that -RustQ could derive. - -Lowering rule (target): - -> Insert `?` at a call site iff the call's return type is -> `Result<_>`/`Option<_>`/`NifResult<_>` **and** the expected type at that -> position is the inner type. - -Examples: - -```elixir -# today (skia, ~166 of 190 sites) -matrix = unwrap!(optional_matrix_from_term(matrix_term)) -tile_mode = unwrap!(GeneratedEnums.decode_tile_mode(tile_mode)) - -# after: identical generated Rust, no explicit operator -matrix = optional_matrix_from_term(matrix_term) -tile_mode = GeneratedEnums.decode_tile_mode(tile_mode) -``` - -```elixir -# stays silent — scrutinee expected as Result -case f(a) do - {:ok, v} -> ... - {:error, _} -> ... -end -``` - -This is *not* hairy type inference. It is the same check Rust's `?` desugaring -performs, moved to lowering time. It needs two pieces of plumbing, both of which -are Wave 3 prerequisites anyway: - -1. **Called-function return types at lowering time.** For `defrust` helpers: - available now via `@spec`. For **external** Rust calls - (`skia_safe::Canvas::draw_rect`, `effect.find_uniform`): requires `RustQ.Syn` - metadata wired into the lowering context — Wave 3 item 1. -2. **Expected-type propagation at lowering positions.** Extends the existing - `infer_expr_type` (`lower.ex`) into a small bidirectional check — the same - machinery as the light borrow/type model (Wave 3 item 3). - -This is why propagation inference is **Wave 3**, not Wave 1: it depends on the -`Syn`-type plumbing. It is also the deliverable that makes Rusty Elixir read as -clean as the Rust it emits — which is the whole point of the metalanguage. - -Honest readability tradeoff: Rust deliberately makes propagation visible. But in -the decode-block pattern that dominates skia, *every* line propagates, so -visibility carries no information — it is pure noise. The right boundary is: -infer by default; require an explicit `case` when you want to handle. That -matches how Elixir authors already think (`with`/`case` at boundaries, -straight-line elsewhere). - -Metric: target **~0 explicit propagation operators in skia** after migration. - ---- - -## Wave 1 — Consolidation & self-hosting (de-risk what exists) - -Goal: remove the ways the "AST-native" story is quietly undermined by string -round-trips and silent fallbacks. Cheap, high-confidence. - -1. **Collapse `semantic_expr`/`semantic_pat` into direct AST lowering.** ✅ Done. - `expr!`/`pat!`/`stmt!`/`arm!` reuse direct lowering; only `raw_*` remains as a - token-level escape hatch. Closes gap A. -2. **Require native AST rendering.** ✅ Done. No `strict_native_ast` mode and no - silent fallback: AST item rendering goes through `RustQ.Native.render_ast/1`. - Closes gap B. -3. **Native-render dogfood gate in skia.** ✅ Done via cross-project validation - on `rustq-semantic-escapes-validation`; Skia has been migrated to the new - namespaces and validated against latest `semantic-escapes-ast`. Closes gap I. - -> Note: the originally-proposed "rename `unwrap!` → `propagate!`" and "`let!` -> lowering" items are **superseded** by Wave 3 type-driven propagation -> inference, which treats the root cause (gap J) rather than the symptom. - -## Wave 2 — Schema coherence & diagnostics (scaling safety) - -Goal: make adding a node safe by construction; make failures legible. - -4. **Schema ↔ renderer ↔ decoder ↔ lowering coherence test.** ✅ Mostly done. - Coherence is derived from `AST.Schema.nodes()` for samples, schema fields, - native dispatch coverage, and decoder coverage. Closes the practical half-wired - node risk in gap C. -5. **Lowering golden corpus.** ❌ Remaining. `test/corpus/*.exs` → expected `.rs`, - covering every documented construct; CI diffs rendered output. Turns anecdotal - completeness into a measured surface. Closes gap E. -6. **Structured diagnostics.** ✅ Done for lowerer unsupported forms, defrust - build failures, and native render failures with `RustQ.Diagnostic`. Closes - gap H for the current failure paths. - -## Wave 3 — `Syn`-driven bindings & type-driven lowering (the new power) - -Goal: RustQ becomes compelling for *wrapping* crates, not just *authoring* new -ones; and Rusty Elixir reads as clean as the Rust it emits. - -7. **`RustQ.Binding` adapter layer.** 🟡 Started. `RustQ.Binding.Callable` - normalizes `RustQ.Syn.Function`, `RustQ.Syn.Method`, and - `RustQ.Native.Descriptor` into callable metadata with `RustQ.Meta.Type` - args/returns. `RustQ.Binding.Index` now provides local/target-qualified - return-type lookup, and `RustQ.Meta.Lower` accepts callable metadata through - its context. Remaining work: generate Rusty-Elixir-shaped lowering targets - and wrappers from these callables. Closes gap D. -8. **Round-trip `Syn` ↔ `defrust` type specs.** 🟡 Started. `RustQ.Meta.Type.from_syn/1` - and `RustQ.Spec.from_syn/1` now map structured `Syn.Type` metadata into - `RustQ.Meta.Type` for common path/ref/option/result/tuple/slice/array/raw - shapes. Remaining work: reverse mapping, richer lifetime/generic fidelity, - and method-signature-level spec derivation. Prerequisite for item 9; closes - gap D fully. -9. **Type-driven propagation inference (headline).** 🟡 Started. The lowerer now - infers Rust `?` in return position, typed assignment/let RHS positions, and - callable argument positions for local, remote, and typed receiver method calls - when callable metadata says the nested call returns - `Result`/`NifResult`/`Option` and the expected type is `T`. - Remaining work: broader local type propagation, project-wide/raw Rust helper - callable metadata, external native method/function callable metadata, and Skia - migration. Skia dogfood showed that raw helpers such as `stroke_paint` and - external calls such as `Paint::set_*`/`ImageFilters::*` still require explicit - `unwrap!` until their argument signatures are available to lowering. Closes - gap J. - Metric: ~0 explicit propagation operators in skia post-migration. -10. **Light borrow/`mut` model.** Carry lifetime/`mut` intent from `Syn` arg - types into lowered bindings instead of pure heuristics. Not full borrowck — - just "this arg is `&mut` so the binding is `mut_ref`". Closes gap F. - -### Post-0.9.3 downstream adapter audit - -Skia, Figler, and Kiwi Codec now dogfood released RustQ `~> 0.9.3` instead of -local path dependencies. A sweep of remaining `unwrap!`, `ref(...)`, -`mut_ref(...)`, `.as_ref()`, `.as_slice()`, and `.as_deref()` sites found that -most remaining occurrences are deliberate boundaries rather than easy cleanup: - -- **Type metadata:** `R.ref(...)` and `R.mut_ref(...)` in specs/types remain the - correct way to express Rust references in Elixir typespecs. -- **Stored/lifetime borrows:** Skia `SaveLayerRec` needs `bounds.as_ref()` - because `SaveLayerRec::bounds(&'a Rect)` stores the reference; borrowing an - owned `Rect` inside a match arm is invalid. -- **Indexed storage borrows:** Figler `ref(index(scene.nodes, node_index))` - avoids moving out of indexed scene storage. This should stay explicit unless - RustQ learns a borrow-preserving indexed-access model. -- **Slice/option adapter APIs:** Skia gradient colors, positions, - path-effect intervals, shader matrices/tile rects, and similar Rust APIs - require exact slice/option-reference adapter shapes. -- **Term map arrays:** Figler/Kiwi Codec `ref(keys)` / `ref(values)` around - generated temporary arrays are API-bound helper calls. Future cleanup would - need richer helper callable metadata, not blind removal. -- **Entrypoint propagation leftovers:** Skia `CommandDomain` still has macro- - generated `unwrap!(decode_args/1)`, `unwrap!(decode_opts/1)`, and generated - opts decoder calls. These are plausible future cleanup candidates because they - are local helper calls, but they should be covered by a focused corpus fixture - before migration. - -Future inference work should target one of the explicit "future candidate" -classes above with corpus coverage first. Do not chase the lifetime/API-required -adapter sites as cosmetic noise. - -## Wave 4 — Crate-level generation & ecosystem (polish & reach) - -Goal: RustQ graduates from "a file generator" to "a crate generator" and proves -generality. - -11. **`rustq.exs` crate manifest.** First-class `crate` blocks: multiple `.rs` - outputs, `mod` declaration generation, `Cargo.toml` `[features]`/deps sync, - content-hash incremental cache. Replaces skia's hand-rolled - `generated_targets/0`. Closes gap G. -12. **`defrustmod` composition primitives.** `pub use` re-exports, `impl` - grouping across modules, trait impl emission from specs. -13. **Second dogfood target beyond skia.** A smaller, different crate (e.g. a - serde-like or wgpu wrapper) to pressure-test that the design generalizes - past the drawing domain and that `Syn` adapters (Wave 3) actually pay off. -14. **Docs restructure.** Split README into a Rusty-Elixir guide (strategic - surface) vs reference for templates/builders (legacy-but-supported). Publish - the lowering grammar (constructs → Rust) generated from the Wave 2 corpus. - -## Sequencing rationale - -Waves 1–2 are cheap and remove the biggest risk: that the "AST-native" story is -quietly undermined by string round-trips and silent fallbacks, and that adding a -node can silently ship half-wired. Wave 3 is the differentiator — `Syn`-driven -binding adapters plus type-driven propagation inference are what make RustQ -compelling for *wrapping* crates and what make the authoring surface read as -clean as the Rust it emits. Wave 4 turns RustQ into a crate generator and proves -generality. +# RustQ roadmap + +Last reviewed 2026-09-30, at 1.0.0-rc.11. The [changelog](CHANGELOG.md) records +what shipped; this file covers where RustQ stands and what comes next. + +## Where things stand + +RustQ has been on the 1.0.0 release-candidate line since July. The +[compatibility policy](guides/compatibility.md) defines what 1.x will promise, +and the RC line exists to test that contract before it is frozen. + +The core pieces are in place: + +- Generated Rust is RustQ AST, rendered natively. There is no string fallback, + and coherence between the schema, the renderer, and the native decoder is + tested from `AST.Schema.nodes()`. +- `defrust`, `defrustp`, and `defnif` lower Elixir-shaped code to Rust. A golden + corpus under `test/corpus` covers the lowering, and `mix ci` checks it with + `rustq.corpus`. +- `RustQ.Native` builds, loads, and stubs a complete NIF crate from Elixir, with + no checked-in Rust. Crates that keep their own Cargo and release setup can + still use it with `build: false, load: false`. +- Lowering infers `?`, `&`, and `&mut` from specs and from callable metadata + read out of Rust sources and Cargo packages. In Skia, explicit `unwrap!` calls + went from about 190 in June to 9. +- The Rustler helpers generate NIF wrappers and stubs from Rust source, atom + registries, term codecs, options, and resources. From rc.11, they also build + term encoders for types read from another crate's source. +- Failures are reported as structured `RustQ.Diagnostic`s, and Reach checks + architecture and authoring smells. + +## Consumers + +| Project | RustQ | What it exercises | +| --- | --- | --- | +| oxc_ex | rc.10 | NIF wrappers and stubs from source, term decoders, lint boundary maps with `RustQ.Native` | +| vize_ex | rc.3, moving to rc.11 | NIF wrappers and stubs, `@type` result codecs, encoders generated from `vize_atelier_vapor`'s source | +| skia_ex | rc.3 | `defrust` drawing commands with generated targets | +| folio | rc.6 | Rustler generation | +| figler | rc.3 | `defrust` helpers over scene storage | +| kiwi_codec | rc.3 | `defrust` helpers and Rustler term helpers | + +Only oxc_ex is within one release of the current RC. + +## Road to 1.0.0 + +1.0.0 should change nothing but the version number. Cut it when: + +1. The consumers above run the latest RC without needing API changes. +2. `Term.encoders_from_source`, new in rc.11, has a second real consumer. Its + options were shaped by a single project, vize_ex; oxc_ex's lint types are the + obvious candidate. +3. A few releases in a row contain only fixes. +4. The compatibility policy decides whether it needs an experimental tier. + Today every documented API is covered in full, so anything shipped in 1.0.0 + is frozen for 1.x. +5. The identifier and typespec matrices below pass. + +## Before 1.0.0 + +**Identifier and typespec matrices.** rc.11 fixed three bugs in surfaces that +had simply never been exercised. Rust keywords failed to render as field +accesses, as struct fields, and as macro arguments, and a literal atom in a +typespec produced a Rust type named after the atom. Instead of waiting for the +next one, add two generated tests: one that renders every identifier-bearing AST +position with every Rust keyword, and one over the typespec forms that the +codecs accept. + +**Consumer upgrades.** Move the consumers to rc.11 and note any API friction +here. + +## 1.x + +**Remaining propagation inference.** Skia's last nine explicit `unwrap!` calls +fall into two groups. Six are the macro-generated `CommandDomain` entrypoint +decodes (`decode_args`, `decode_opts`, and the generated options decoders). The +other three are in text commands, where the call is a setter argument or a +`case` arm (`decode_text_decoration_mode`, `paragraph_paint_y`). Each needs a +corpus fixture before the inference changes. Under the compatibility policy, +new inference may only apply where RustQ currently requires the explicit form; +programs that already compile must keep producing equivalent Rust. + +**Borrow and `mut` intent.** Carry `&mut` from `Syn` argument types into lowered +bindings instead of relying on heuristics. This is not a borrow checker, only +"this argument is `&mut`, so the binding is a mutable reference". + +**Bindings from `Syn`.** `RustQ.Binding.Callable` already normalizes Rust +functions and methods for lowering. What remains is generating Rusty-Elixir +wrappers and lowering targets from those callables, the reverse mapping from +specs to `Syn` types, and better fidelity for generics and lifetimes. + +**Codecs from Rust source.** Decoders to mirror `Term.encoders_from_source`, +then move oxc_ex's lint types and vize_ex's remaining handwritten shapes onto +them. + +**Multi-file generation for existing crates.** `RustQ.Native` covers crates +RustQ owns. Crates that keep Cargo ownership still hand-roll their target lists, +as Skia does with `generated_targets/0`. `rustq.exs` should be able to declare +several outputs and the `mod` declarations that tie them together. + +**Module composition.** `defrustimpl` covers inherent and trait impls. `pub use` +re-exports and grouping impls across modules are still open. + +## Deliberate limits + +Some explicit borrows and adapters are required by the Rust API rather than +missing inference, and should not be "cleaned up": + +- Stored references, such as Skia's `SaveLayerRec::bounds(&'a Rect)`, need a + borrow that outlives the call. +- Indexed storage, such as Figler's `ref(index(scene.nodes, node_index))`, is + borrowed to avoid moving out of the collection. +- Slice and option-reference parameters in Skia's gradient, path effect, and + shader APIs need exact adapter shapes. + +Inference work should target the open cases above, with corpus coverage first.