From 28783a504259fd1fea25d5297b005e53c8021a29 Mon Sep 17 00:00:00 2001 From: Danila Poyarkov Date: Wed, 30 Sep 2026 10:17:48 +0400 Subject: [PATCH] Rewrite the roadmap for the 1.0 release line The roadmap still described RustQ at 0.5 and tracked most finished work as open: the lowering corpus, the second dogfood target, the docs split, and most of propagation inference. It also never mentioned the 1.0 release line, RustQ.Native, or source-derived codecs. Replace it with where things stand at rc.11, the consumers and the RustQ versions they pin, criteria for cutting 1.0.0, and the remaining work split between before 1.0.0 and 1.x. Skia's explicit unwrap! count is measured again against its current master: 9, down from about 190. --- ROADMAP.md | 352 ++++++++++++++++++----------------------------------- 1 file changed, 116 insertions(+), 236 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 35ba6d4..49270ea 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,236 +1,116 @@ -# RustQ Roadmap — Next Generation - -Strategic plan to take RustQ from v0.5.x ("Rusty Elixir MVP") to a polished, -powerful typed-macro metalanguage for Rust codegen. - -State of this document reflects review on 2026-06-19 against `defrust-meta-mvp` -and the skia dogfood (`../skia_ex`). - -## Where we are - -- **Wave 1 is complete.** Semantic helpers lower through real RustQ AST, - native AST rendering is mandatory, `strict_native_ast` is gone, and Skia has - been validated against `semantic-escapes-ast` after the namespace cleanup. -- **Wave 2 is mostly complete.** Schema/native decoder/render coherence is - tested from `AST.Schema.nodes()`, rendering/build/lowering failures now raise - structured diagnostics, and tests mirror the source layout. The remaining Wave - 2 gap is a full golden lowering corpus. -- **Module structure has been cleaned up beyond the original roadmap.** - `NativeCodegen` is now `Codegen`, external Rust metadata lives under - `RustQ.Native.*`, Rustler helper modules are concept-oriented, `RustQ.Meta` is - split into focused submodules, config state lives in `RustQ.Config.State`, and - every module has real documentation instead of `@moduledoc false`. -- **Skia proves the dogfood.** `Skia.Codegen.Rusty.*` author drawing semantics - in `defrust` + ordinary `quote`/`unquote` with **zero** `expr!`/`pat!`/`stmt!`/ - `arm!`/`raw_*` escapes in semantic bodies. ~3.3k of 4k native lines are - generated; CI gates on `mix rustq.gen --check`. The ownership boundary (Skia - owns semantics, RustQ owns lowering) holds in practice. -- **Discipline is codified** in `AGENTS.md`; architecture enforcement belongs in - Reach/ExDNA/Credo/custom linting or schema-driven behavioral tests, not ad hoc - ExUnit source-grep tests. - -## Gaps (evidence → fix) - -| # | Gap | Evidence | Addressed in | -|---|-----|----------|--------------| -| A | `semantic_expr`/`semantic_pat` build Rust source strings that round-trip through `parse_syn`, even for shapes the mainstream `lower_expr` already emits as AST nodes. Parallel path; violates AGENTS #7. | `lib/rustq/meta/lower.ex:564–700`; skia never uses `expr!`, so this path is undertested | Wave 1 | -| B | Silent render fallback unless `strict_native_ast: true`. A render gap can emit subtly wrong Rust in real builds. | `lib/rustq/rust/ast/render.ex:106–117` | Wave 1 | -| C | No coherence guarantee between `defnode` ↔ Elixir renderer ↔ native decoder ↔ `Meta.Lower`. A new node must be wired in ~4 places with no check. Biggest scaling risk. | `ast/schema.ex` + `decoders/*.ex` | Wave 2 | -| D | No `RustQ.Syn` → Rusty-Elixir binding adapter. Wrapping a crate means hand-authoring every `defrust` shell. | `skia/codegen/skia_safe.ex` + `commands/*` | Wave 3 | -| E | Lowering completeness is anecdotal. `lower.ex` raises `unsupported` in several spots; no golden corpus. | grep `unsupported` in `lower.ex` | Wave 2 | -| F | Mutability/lifetime inference is heuristic; lifetimes come only from `R.lifetime(:a)` in specs, never inferred in bodies. | `lower.ex:790–845` | Wave 3 | -| G | No crate-level manifest. `rustq.exs` is file-list-oriented; skia hand-rolls 19 targets via `generated_targets/0`. | `skia/codegen.ex:11–60` | Wave 4 | -| H | Diagnostics are bare `ArgumentError`s. No spans, no offending-node reporting. | `lower.ex` raises | Wave 2 | -| I | No native-render dogfood gate across RustQ + skia proving every lowered body renders through native AST without fallback. | `RustQ.Native.render_ast/1` is mandatory in RustQ; skia still needs an explicit cross-project gate | Wave 1 / skia follow-up | -| **J** | **`unwrap!` noise.** ~190 uses in skia; the macro name is semantically wrong (`unwrap`/`unwrap()` means *panic* in both Rust and Elixir, but it generates `?` = *propagate*), and the `?` it spells is fully derivable from the called function's `@spec` return type vs the expected type at the call position. Author is forced to restate type information the compiler already has. | `grep -r unwrap! skia`; every decode helper has a `Result`/`Option` `@spec` | **Wave 3** | - -## The headline insight: type-driven propagation inference - -`unwrap!(expr) → expr?` is **syntax-to-syntax** lowering. Rust's `?` is -*mandatory* on the Rust surface because the compiler rejects -`let x = f()` when `f: -> Result`. But that mandate does **not** exist at the -Elixir/meta layer: RustQ has the called function's return type (from `@spec`, -confirmed for every skia decode helper) and the expected type at the call -position. Whether `?` is needed is fully determined by those types. Forcing the -author to spell it is making them transcribe a Rust-language requirement that -RustQ could derive. - -Lowering rule (target): - -> Insert `?` at a call site iff the call's return type is -> `Result<_>`/`Option<_>`/`NifResult<_>` **and** the expected type at that -> position is the inner type. - -Examples: - -```elixir -# today (skia, ~166 of 190 sites) -matrix = unwrap!(optional_matrix_from_term(matrix_term)) -tile_mode = unwrap!(GeneratedEnums.decode_tile_mode(tile_mode)) - -# after: identical generated Rust, no explicit operator -matrix = optional_matrix_from_term(matrix_term) -tile_mode = GeneratedEnums.decode_tile_mode(tile_mode) -``` - -```elixir -# stays silent — scrutinee expected as Result -case f(a) do - {:ok, v} -> ... - {:error, _} -> ... -end -``` - -This is *not* hairy type inference. It is the same check Rust's `?` desugaring -performs, moved to lowering time. It needs two pieces of plumbing, both of which -are Wave 3 prerequisites anyway: - -1. **Called-function return types at lowering time.** For `defrust` helpers: - available now via `@spec`. For **external** Rust calls - (`skia_safe::Canvas::draw_rect`, `effect.find_uniform`): requires `RustQ.Syn` - metadata wired into the lowering context — Wave 3 item 1. -2. **Expected-type propagation at lowering positions.** Extends the existing - `infer_expr_type` (`lower.ex`) into a small bidirectional check — the same - machinery as the light borrow/type model (Wave 3 item 3). - -This is why propagation inference is **Wave 3**, not Wave 1: it depends on the -`Syn`-type plumbing. It is also the deliverable that makes Rusty Elixir read as -clean as the Rust it emits — which is the whole point of the metalanguage. - -Honest readability tradeoff: Rust deliberately makes propagation visible. But in -the decode-block pattern that dominates skia, *every* line propagates, so -visibility carries no information — it is pure noise. The right boundary is: -infer by default; require an explicit `case` when you want to handle. That -matches how Elixir authors already think (`with`/`case` at boundaries, -straight-line elsewhere). - -Metric: target **~0 explicit propagation operators in skia** after migration. - ---- - -## Wave 1 — Consolidation & self-hosting (de-risk what exists) - -Goal: remove the ways the "AST-native" story is quietly undermined by string -round-trips and silent fallbacks. Cheap, high-confidence. - -1. **Collapse `semantic_expr`/`semantic_pat` into direct AST lowering.** ✅ Done. - `expr!`/`pat!`/`stmt!`/`arm!` reuse direct lowering; only `raw_*` remains as a - token-level escape hatch. Closes gap A. -2. **Require native AST rendering.** ✅ Done. No `strict_native_ast` mode and no - silent fallback: AST item rendering goes through `RustQ.Native.render_ast/1`. - Closes gap B. -3. **Native-render dogfood gate in skia.** ✅ Done via cross-project validation - on `rustq-semantic-escapes-validation`; Skia has been migrated to the new - namespaces and validated against latest `semantic-escapes-ast`. Closes gap I. - -> Note: the originally-proposed "rename `unwrap!` → `propagate!`" and "`let!` -> lowering" items are **superseded** by Wave 3 type-driven propagation -> inference, which treats the root cause (gap J) rather than the symptom. - -## Wave 2 — Schema coherence & diagnostics (scaling safety) - -Goal: make adding a node safe by construction; make failures legible. - -4. **Schema ↔ renderer ↔ decoder ↔ lowering coherence test.** ✅ Mostly done. - Coherence is derived from `AST.Schema.nodes()` for samples, schema fields, - native dispatch coverage, and decoder coverage. Closes the practical half-wired - node risk in gap C. -5. **Lowering golden corpus.** ❌ Remaining. `test/corpus/*.exs` → expected `.rs`, - covering every documented construct; CI diffs rendered output. Turns anecdotal - completeness into a measured surface. Closes gap E. -6. **Structured diagnostics.** ✅ Done for lowerer unsupported forms, defrust - build failures, and native render failures with `RustQ.Diagnostic`. Closes - gap H for the current failure paths. - -## Wave 3 — `Syn`-driven bindings & type-driven lowering (the new power) - -Goal: RustQ becomes compelling for *wrapping* crates, not just *authoring* new -ones; and Rusty Elixir reads as clean as the Rust it emits. - -7. **`RustQ.Binding` adapter layer.** 🟡 Started. `RustQ.Binding.Callable` - normalizes `RustQ.Syn.Function`, `RustQ.Syn.Method`, and - `RustQ.Native.Descriptor` into callable metadata with `RustQ.Meta.Type` - args/returns. `RustQ.Binding.Index` now provides local/target-qualified - return-type lookup, and `RustQ.Meta.Lower` accepts callable metadata through - its context. Remaining work: generate Rusty-Elixir-shaped lowering targets - and wrappers from these callables. Closes gap D. -8. **Round-trip `Syn` ↔ `defrust` type specs.** 🟡 Started. `RustQ.Meta.Type.from_syn/1` - and `RustQ.Spec.from_syn/1` now map structured `Syn.Type` metadata into - `RustQ.Meta.Type` for common path/ref/option/result/tuple/slice/array/raw - shapes. Remaining work: reverse mapping, richer lifetime/generic fidelity, - and method-signature-level spec derivation. Prerequisite for item 9; closes - gap D fully. -9. **Type-driven propagation inference (headline).** 🟡 Started. The lowerer now - infers Rust `?` in return position, typed assignment/let RHS positions, and - callable argument positions for local, remote, and typed receiver method calls - when callable metadata says the nested call returns - `Result`/`NifResult`/`Option` and the expected type is `T`. - Remaining work: broader local type propagation, project-wide/raw Rust helper - callable metadata, external native method/function callable metadata, and Skia - migration. Skia dogfood showed that raw helpers such as `stroke_paint` and - external calls such as `Paint::set_*`/`ImageFilters::*` still require explicit - `unwrap!` until their argument signatures are available to lowering. Closes - gap J. - Metric: ~0 explicit propagation operators in skia post-migration. -10. **Light borrow/`mut` model.** Carry lifetime/`mut` intent from `Syn` arg - types into lowered bindings instead of pure heuristics. Not full borrowck — - just "this arg is `&mut` so the binding is `mut_ref`". Closes gap F. - -### Post-0.9.3 downstream adapter audit - -Skia, Figler, and Kiwi Codec now dogfood released RustQ `~> 0.9.3` instead of -local path dependencies. A sweep of remaining `unwrap!`, `ref(...)`, -`mut_ref(...)`, `.as_ref()`, `.as_slice()`, and `.as_deref()` sites found that -most remaining occurrences are deliberate boundaries rather than easy cleanup: - -- **Type metadata:** `R.ref(...)` and `R.mut_ref(...)` in specs/types remain the - correct way to express Rust references in Elixir typespecs. -- **Stored/lifetime borrows:** Skia `SaveLayerRec` needs `bounds.as_ref()` - because `SaveLayerRec::bounds(&'a Rect)` stores the reference; borrowing an - owned `Rect` inside a match arm is invalid. -- **Indexed storage borrows:** Figler `ref(index(scene.nodes, node_index))` - avoids moving out of indexed scene storage. This should stay explicit unless - RustQ learns a borrow-preserving indexed-access model. -- **Slice/option adapter APIs:** Skia gradient colors, positions, - path-effect intervals, shader matrices/tile rects, and similar Rust APIs - require exact slice/option-reference adapter shapes. -- **Term map arrays:** Figler/Kiwi Codec `ref(keys)` / `ref(values)` around - generated temporary arrays are API-bound helper calls. Future cleanup would - need richer helper callable metadata, not blind removal. -- **Entrypoint propagation leftovers:** Skia `CommandDomain` still has macro- - generated `unwrap!(decode_args/1)`, `unwrap!(decode_opts/1)`, and generated - opts decoder calls. These are plausible future cleanup candidates because they - are local helper calls, but they should be covered by a focused corpus fixture - before migration. - -Future inference work should target one of the explicit "future candidate" -classes above with corpus coverage first. Do not chase the lifetime/API-required -adapter sites as cosmetic noise. - -## Wave 4 — Crate-level generation & ecosystem (polish & reach) - -Goal: RustQ graduates from "a file generator" to "a crate generator" and proves -generality. - -11. **`rustq.exs` crate manifest.** First-class `crate` blocks: multiple `.rs` - outputs, `mod` declaration generation, `Cargo.toml` `[features]`/deps sync, - content-hash incremental cache. Replaces skia's hand-rolled - `generated_targets/0`. Closes gap G. -12. **`defrustmod` composition primitives.** `pub use` re-exports, `impl` - grouping across modules, trait impl emission from specs. -13. **Second dogfood target beyond skia.** A smaller, different crate (e.g. a - serde-like or wgpu wrapper) to pressure-test that the design generalizes - past the drawing domain and that `Syn` adapters (Wave 3) actually pay off. -14. **Docs restructure.** Split README into a Rusty-Elixir guide (strategic - surface) vs reference for templates/builders (legacy-but-supported). Publish - the lowering grammar (constructs → Rust) generated from the Wave 2 corpus. - -## Sequencing rationale - -Waves 1–2 are cheap and remove the biggest risk: that the "AST-native" story is -quietly undermined by string round-trips and silent fallbacks, and that adding a -node can silently ship half-wired. Wave 3 is the differentiator — `Syn`-driven -binding adapters plus type-driven propagation inference are what make RustQ -compelling for *wrapping* crates and what make the authoring surface read as -clean as the Rust it emits. Wave 4 turns RustQ into a crate generator and proves -generality. +# RustQ roadmap + +Last reviewed 2026-09-30, at 1.0.0-rc.11. The [changelog](CHANGELOG.md) records +what shipped; this file covers where RustQ stands and what comes next. + +## Where things stand + +RustQ has been on the 1.0.0 release-candidate line since July. The +[compatibility policy](guides/compatibility.md) defines what 1.x will promise, +and the RC line exists to test that contract before it is frozen. + +The core pieces are in place: + +- Generated Rust is RustQ AST, rendered natively. There is no string fallback, + and coherence between the schema, the renderer, and the native decoder is + tested from `AST.Schema.nodes()`. +- `defrust`, `defrustp`, and `defnif` lower Elixir-shaped code to Rust. A golden + corpus under `test/corpus` covers the lowering, and `mix ci` checks it with + `rustq.corpus`. +- `RustQ.Native` builds, loads, and stubs a complete NIF crate from Elixir, with + no checked-in Rust. Crates that keep their own Cargo and release setup can + still use it with `build: false, load: false`. +- Lowering infers `?`, `&`, and `&mut` from specs and from callable metadata + read out of Rust sources and Cargo packages. In Skia, explicit `unwrap!` calls + went from about 190 in June to 9. +- The Rustler helpers generate NIF wrappers and stubs from Rust source, atom + registries, term codecs, options, and resources. From rc.11, they also build + term encoders for types read from another crate's source. +- Failures are reported as structured `RustQ.Diagnostic`s, and Reach checks + architecture and authoring smells. + +## Consumers + +| Project | RustQ | What it exercises | +| --- | --- | --- | +| oxc_ex | rc.10 | NIF wrappers and stubs from source, term decoders, lint boundary maps with `RustQ.Native` | +| vize_ex | rc.3, moving to rc.11 | NIF wrappers and stubs, `@type` result codecs, encoders generated from `vize_atelier_vapor`'s source | +| skia_ex | rc.3 | `defrust` drawing commands with generated targets | +| folio | rc.6 | Rustler generation | +| figler | rc.3 | `defrust` helpers over scene storage | +| kiwi_codec | rc.3 | `defrust` helpers and Rustler term helpers | + +Only oxc_ex is within one release of the current RC. + +## Road to 1.0.0 + +1.0.0 should change nothing but the version number. Cut it when: + +1. The consumers above run the latest RC without needing API changes. +2. `Term.encoders_from_source`, new in rc.11, has a second real consumer. Its + options were shaped by a single project, vize_ex; oxc_ex's lint types are the + obvious candidate. +3. A few releases in a row contain only fixes. +4. The compatibility policy decides whether it needs an experimental tier. + Today every documented API is covered in full, so anything shipped in 1.0.0 + is frozen for 1.x. +5. The identifier and typespec matrices below pass. + +## Before 1.0.0 + +**Identifier and typespec matrices.** rc.11 fixed three bugs in surfaces that +had simply never been exercised. Rust keywords failed to render as field +accesses, as struct fields, and as macro arguments, and a literal atom in a +typespec produced a Rust type named after the atom. Instead of waiting for the +next one, add two generated tests: one that renders every identifier-bearing AST +position with every Rust keyword, and one over the typespec forms that the +codecs accept. + +**Consumer upgrades.** Move the consumers to rc.11 and note any API friction +here. + +## 1.x + +**Remaining propagation inference.** Skia's last nine explicit `unwrap!` calls +fall into two groups. Six are the macro-generated `CommandDomain` entrypoint +decodes (`decode_args`, `decode_opts`, and the generated options decoders). The +other three are in text commands, where the call is a setter argument or a +`case` arm (`decode_text_decoration_mode`, `paragraph_paint_y`). Each needs a +corpus fixture before the inference changes. Under the compatibility policy, +new inference may only apply where RustQ currently requires the explicit form; +programs that already compile must keep producing equivalent Rust. + +**Borrow and `mut` intent.** Carry `&mut` from `Syn` argument types into lowered +bindings instead of relying on heuristics. This is not a borrow checker, only +"this argument is `&mut`, so the binding is a mutable reference". + +**Bindings from `Syn`.** `RustQ.Binding.Callable` already normalizes Rust +functions and methods for lowering. What remains is generating Rusty-Elixir +wrappers and lowering targets from those callables, the reverse mapping from +specs to `Syn` types, and better fidelity for generics and lifetimes. + +**Codecs from Rust source.** Decoders to mirror `Term.encoders_from_source`, +then move oxc_ex's lint types and vize_ex's remaining handwritten shapes onto +them. + +**Multi-file generation for existing crates.** `RustQ.Native` covers crates +RustQ owns. Crates that keep Cargo ownership still hand-roll their target lists, +as Skia does with `generated_targets/0`. `rustq.exs` should be able to declare +several outputs and the `mod` declarations that tie them together. + +**Module composition.** `defrustimpl` covers inherent and trait impls. `pub use` +re-exports and grouping impls across modules are still open. + +## Deliberate limits + +Some explicit borrows and adapters are required by the Rust API rather than +missing inference, and should not be "cleaned up": + +- Stored references, such as Skia's `SaveLayerRec::bounds(&'a Rect)`, need a + borrow that outlives the call. +- Indexed storage, such as Figler's `ref(index(scene.nodes, node_index))`, is + borrowed to avoid moving out of the collection. +- Slice and option-reference parameters in Skia's gradient, path effect, and + shader APIs need exact adapter shapes. + +Inference work should target the open cases above, with corpus coverage first.