diff --git a/.github/workflows/ts-test-suite.yml b/.github/workflows/ts-test-suite.yml index 1c947012..cbc1d086 100644 --- a/.github/workflows/ts-test-suite.yml +++ b/.github/workflows/ts-test-suite.yml @@ -128,8 +128,8 @@ jobs: cd hub-client npm run build:all - # - name: Run hub-client tests - # shell: bash - # run: | - # cd hub-client - # npm run test:ci + - name: Run hub-client tests + shell: bash + run: | + cd hub-client + npm run test:ci diff --git a/claude-notes/plans/2026-03-16-extensions-master-plan.md b/claude-notes/plans/2026-03-16-extensions-grand-plan.md similarity index 82% rename from claude-notes/plans/2026-03-16-extensions-master-plan.md rename to claude-notes/plans/2026-03-16-extensions-grand-plan.md index 453baa20..7cd7f4ec 100644 --- a/claude-notes/plans/2026-03-16-extensions-master-plan.md +++ b/claude-notes/plans/2026-03-16-extensions-grand-plan.md @@ -1,9 +1,12 @@ -# Quarto Extensions Master Plan +# Quarto Extensions Grand Plan **Created**: 2026-03-16 -**Status**: In Progress (Phases 1 + 4 complete) -**Worktree**: `worktree-extensions-phase1` branch, at `.claude/worktrees/extensions-phase1` -**Sub-plans**: Phase 1 detail at `claude-notes/plans/2026-03-16-extensions-phase1-yml-and-metadata.md` +**Status**: In Progress (Phases 1, 2, 3, 4 complete) +**Sub-plans**: +- Phase 1: `claude-notes/plans/2026-03-16-extensions-phase1-yml-and-metadata.md` +- Phase 2: `claude-notes/plans/2026-03-17-extensions-phase2-filter-resolution.md` +- Phase 3: `claude-notes/plans/2026-03-20-extensions-phase3-shortcode-resolution.md` +- Phase 4: `claude-notes/plans/2026-03-16-extensions-phase4-templates-partials.md` ## Codebase Context for New Agents @@ -321,55 +324,55 @@ before file discovery. **Goal**: Parse `_extension.yml`, discover extensions, and merge extension metadata into the rendering pipeline. No filter/shortcode execution yet — just metadata. -**Current status** (as of 2026-03-16): +**Status**: Complete (merged as `68420002`) -| Sub-phase | Status | Summary | -|-----------|--------|---------| -| 1.1 Data model | Done | `ExtensionId`, `Extension`, `Contributes`, `ExtensionFilter` in `extension/types.rs` | -| 1.2 YAML parser | Done | `read_extension()` in `extension/read.rs` with "common" key merging | -| 1.3 Discovery | Done | `discover_extensions()` in `extension/discover.rs`, walks `_extensions/` dirs | -| 1.4 Format parsing | Done | `parse_format_descriptor()` splits "acm-html" → extension "acm" + base "html" | -| 1.4b Format string preservation | **TODO** | `Format` struct needs `target_format`, `extension_name`, `display_name` fields | -| 1.5 Metadata merge | Done | `build_extension_metadata_layer()` in metadata_merge.rs, but **blocked by 1.4b** | -| 1.6 format-resources | Deferred | Will be a separate PR | -| 1.7 Smoke tests | TODO | Depends on 1.4b | -| 1.8 Workspace verify | Partial | Build + tests pass; WASM verify still needed | - -**Key blocker**: Phase 1.4b. The `Format` struct currently has no field to preserve -the original format string (e.g., "acm-html"). `format_from_name()` in `render_to_file.rs` -silently falls back to `Format::html()`, losing the extension name. The metadata -merge layer receives "html" instead of "acm-html" and never finds an extension match. - -See the Phase 1 sub-plan for full details on 1.4b implementation. +All sub-phases done. `Format` struct has `target_format`, `extension_name`, +`display_name` fields. Extension metadata merges correctly into the pipeline. +`format-resources` deferred to a later phase. See sub-plan for details. --- -### Phase 2: Extension Filter Resolution +### Phase 2: Extension Filter Resolution ✅ **Goal**: Resolve filter names that reference extensions (e.g., `filters: [lightbox]` where `lightbox` is an extension contributing filters). -- [ ] Update `filter_resolve.rs` to accept extension context -- [ ] When a filter name doesn't resolve to a file path, look it up in extensions -- [ ] If found, substitute the extension's contributed filter paths -- [ ] Handle per-format filters from format extensions -- [ ] Tests: extension filter resolution, missing extension → error +- [x] Update `filter_resolve.rs` to accept extension context +- [x] When a filter name doesn't resolve to a file path, look it up in extensions +- [x] If found, substitute the extension's contributed filter paths +- [x] Handle per-format filters from format extensions +- [x] Tests: extension filter resolution, missing extension → error + +**Status**: Complete (merged as `cffc2e6c`) + +Two mechanisms implemented: +1. **Per-format filters**: `mark_path_valued_keys()` converts filter paths in + extension format metadata to `ConfigValueKind::Path`, which gets rebased by + `adjust_paths_to_document_dir()` during metadata merge. +2. **Name-based resolution**: `resolve_filters()` accepts `&[Extension]` and + `&dyn SystemRuntime`. Uses file-first resolution (matching TS Quarto): check + if path exists on disk, only try extension lookup if it doesn't. Supports + both string and map forms with `at` propagation. -**Depends on**: Phase 1 (extension discovery) + current user-filters work +**Detail plan**: `claude-notes/plans/2026-03-17-extensions-phase2-filter-resolution.md` -### Phase 3: Extension Shortcode Resolution +### Phase 3: Extension Shortcode Resolution ✅ **Goal**: Resolve shortcode references from extensions and wire them into the -shortcode processing pipeline. +shortcode processing pipeline. Includes block-level shortcode support and a new +`LuaShortcodeEngine` in pampa for loading and dispatching Lua shortcode handlers. -- [ ] Discover shortcodes contributed by active extensions -- [ ] Add extension shortcode paths to shortcode resolution -- [ ] Handle per-format shortcodes from format extensions -- [ ] Tests +- [x] Mark per-format shortcode paths as `ConfigValueKind::Path` in `mark_path_valued_keys()` +- [x] Create `LuaShortcodeEngine` in pampa (single Lua state, load scripts, dispatch by name) +- [x] Add block-level shortcode support (`ShortcodeResult::Blocks`, two-pass resolution) +- [x] Wire extensions and Lua engine into `ShortcodeResolveTransform` +- [x] Name-based extension lookup for unknown shortcode names +- [x] `quarto.shortcode` Lua API (`read_arg`, `error_output`) +- [x] Integration and smoke tests -**Open questions**: -- How do shortcodes currently work in q2? (Need to check `ShortcodeResolveTransform`) -- Are shortcodes already resolved from file paths, or is there a registry? +**Detail plan**: `claude-notes/plans/2026-03-20-extensions-phase3-shortcode-resolution.md` + +**Depends on**: Phase 1 (extension discovery), Phase 2 (filter resolution pattern) ### Phase 4: Template and Partial Support ✅ @@ -492,6 +495,27 @@ shortcode processing pipeline. - Which built-in extensions are essential for initial release? - Can we bundle TS Quarto's built-in extensions directly, or do they need porting? +### Phase 12: Semver Validation for `quarto-required` + +**Goal**: Validate the `quarto-required` field in `_extension.yml` against the +running Quarto version, warning or erroring when an extension requires a version +the user doesn't have. + +- [ ] Add `semver` crate dependency (dtolnay's — the de facto Rust standard) +- [ ] Parse `quarto-required` as `VersionReq` during `read_extension()` +- [ ] Check against `quarto_util::version::cli_version()` during extension discovery +- [ ] Emit a diagnostic when version doesn't satisfy the requirement +- [ ] Optionally parse and store `version` field as `semver::Version` +- [ ] Tests + +**Notes**: +- `cli_version()` returns `"99.9.9-dev"` during development. Under strict semver, + prereleases don't satisfy range constraints like `>=1.4.0`. Either strip the + `-dev` suffix before checking or use `99.9.9` as the dev version. +- TS Quarto warns (not hard error) on version mismatch. +- The extension's own `version` field is currently stored as a plain string. + Could optionally be parsed as `semver::Version` for consistency. + --- ## Architecture Decisions @@ -543,9 +567,7 @@ and `!concat` tag support. No new merge logic needed. ## Open Questions (Cross-cutting) -1. **Semver validation**: Should we validate `version` and `quarto-required` fields - during parsing, or just store them as strings? TS Quarto uses the `semver` npm - package for range checking. +1. ~~**Semver validation**~~: Moved to Phase 12. 2. **Extension caching**: TS Quarto caches extensions per-context. In q2, should we cache per-render, per-project, or globally? (Probably per-project is sufficient.) @@ -563,16 +585,18 @@ and `!concat` tag support. No new merge logic needed. infrastructure exists in `quarto-yaml` — we should preserve it through extension loading. -6. **Format name mapping**: ~~When a user writes `format: acm-pdf`, how do we - determine that `acm` is the extension name and `pdf` is the base format?~~ - **RESOLVED**: `parse_format_descriptor()` in `extension/discover.rs` splits on - the last hyphen matching a known base format. Matches TS Quarto's - `parseFormatString()`. The `Format` struct needs `target_format` and - `extension_name` fields to carry this through the pipeline (Phase 1.4b). +## Design Decisions (Resolved) + +6. **Format name mapping**: `parse_format_descriptor()` in `extension/discover.rs` + splits on the last hyphen matching a known base format (e.g., `"acm-pdf"` → + extension `"acm"`, base `"pdf"`). The `Format` struct carries `target_format`, + `extension_name`, and `display_name` fields. Matches TS Quarto's + `parseFormatString()`. (Resolved in Phase 1.4b.) -7. **Extension ordering**: When multiple extensions contribute to the same format, - what's the merge order? TS Quarto uses discovery order (built-in first, then - project hierarchy). We should match this. +7. **Extension ordering**: Multiple extensions contributing to the same format are + merged in discovery order: built-in first, then project hierarchy (closest to + project root first, closest to document last). This matches TS Quarto's behavior. + Implemented in `discover_extensions()`. (Resolved in Phase 1.3.) ## TS Quarto ↔ Rust Quarto Vocabulary @@ -600,4 +624,5 @@ Confirmed via DeepWiki research on `quarto-dev/quarto-cli`: - q2 metadata merge: `crates/quarto-core/src/stage/stages/metadata_merge.rs` - q2 filter resolve: `crates/quarto-core/src/filter_resolve.rs` - q2 user filters plan: `claude-notes/plans/2026-03-16-user-filters-pipeline.md` +- Metadata pipeline / Lua detection: `claude-notes/plans/2026-03-18-metadata-pipeline-lua-detection.md` - q2 config merging design: `claude-notes/plans/2025-12-07-config-merging-design.md` diff --git a/claude-notes/plans/2026-03-16-extensions-phase1-yml-and-metadata.md b/claude-notes/plans/2026-03-16-extensions-phase1-yml-and-metadata.md index bb2c926f..556cb6ac 100644 --- a/claude-notes/plans/2026-03-16-extensions-phase1-yml-and-metadata.md +++ b/claude-notes/plans/2026-03-16-extensions-phase1-yml-and-metadata.md @@ -1,12 +1,12 @@ # Extensions Phase 1: _extension.yml Parsing and Metadata Contributions **Created**: 2026-03-16 -**Status**: Complete (1.1-1.5, 1.4b, 1.7-1.8 done; 1.6 deferred) -**Parent Plan**: `claude-notes/plans/2026-03-16-extensions-master-plan.md` +**Status**: Complete (all phases done; 1.6 format-resources deferred) +**Parent Plan**: `claude-notes/plans/2026-03-16-extensions-grand-plan.md` ## Codebase Context for New Agents -**READ FIRST**: The master plan (`claude-notes/plans/2026-03-16-extensions-master-plan.md`) +**READ FIRST**: The grand plan (`claude-notes/plans/2026-03-16-extensions-grand-plan.md`) has a "Codebase Context" section with the full crate map, type reference, and testing patterns. Read that before starting this work. @@ -873,14 +873,14 @@ use the `NativeRuntime` or similar concrete implementation — grep for | `crates/quarto-core/src/lib.rs` | Modify | Done | Register `extension` module | | `crates/quarto-core/src/stage/context.rs` | Modify | Done | Add `extensions` field to `StageContext` | | `crates/quarto-core/src/stage/stages/metadata_merge.rs` | Modify | Done | Insert extension layer + `build_extension_metadata_layer()` | -| `crates/quarto-core/src/format.rs` | Modify | TODO | Add `target_format`, `extension_name`, `display_name` to `Format`; add `from_format_string()` | -| `crates/quarto-core/src/render_to_file.rs` | Modify | TODO | Update `format_from_name()` to use `Format::from_format_string()` | -| `crates/quarto-test/src/runner.rs` | Modify | TODO | Update format construction if needed | -| `crates/quarto/tests/smoke-all/extensions/` | Create | TODO | Smoke test fixtures | +| `crates/quarto-core/src/format.rs` | Modify | Done | Add `target_format`, `extension_name`, `display_name` to `Format`; add `from_format_string()` | +| `crates/quarto-core/src/render_to_file.rs` | Modify | Done | Update `format_from_name()` to use `Format::from_format_string()` | +| `crates/quarto-test/src/runner.rs` | Modify | Done | Update format construction if needed | +| `crates/quarto/tests/smoke-all/extensions/` | Create | Done | Smoke test fixtures | ## References -- Master plan: `claude-notes/plans/2026-03-16-extensions-master-plan.md` +- Grand plan: `claude-notes/plans/2026-03-16-extensions-grand-plan.md` - TS Quarto format descriptor: `src/core/pandoc/pandoc-formats.ts` (`parseFormatString()`) - TS Quarto format identifier: `src/config/types.ts` (`FormatIdentifier` interface) - TS Quarto extension reading: `src/extension/extension.ts` diff --git a/claude-notes/plans/2026-03-16-extensions-phase4-templates-partials.md b/claude-notes/plans/2026-03-16-extensions-phase4-templates-partials.md index eb4b6b57..2b16fdc0 100644 --- a/claude-notes/plans/2026-03-16-extensions-phase4-templates-partials.md +++ b/claude-notes/plans/2026-03-16-extensions-phase4-templates-partials.md @@ -2,7 +2,7 @@ **Created**: 2026-03-16 **Status**: Complete -**Parent Plan**: `claude-notes/plans/2026-03-16-extensions-master-plan.md` +**Parent Plan**: `claude-notes/plans/2026-03-16-extensions-grand-plan.md` **Depends on**: Phase 1 (complete) ## Overview @@ -361,9 +361,9 @@ on `quarto-system-runtime`), the `RuntimeResolver` must live in `quarto-core` - [x] **4.7.1** `cargo build --workspace` — clean build - [x] **4.7.2** `cargo nextest run --workspace` — 6693 tests pass, 0 failures -### Phase 4.8: Update master plan +### Phase 4.8: Update grand plan -- [x] **4.8.1** Updated `claude-notes/plans/2026-03-16-extensions-master-plan.md` +- [x] **4.8.1** Updated `claude-notes/plans/2026-03-16-extensions-grand-plan.md` to mark Phase 4 complete with summary of changes. --- diff --git a/claude-notes/plans/2026-03-17-extensions-phase2-filter-resolution.md b/claude-notes/plans/2026-03-17-extensions-phase2-filter-resolution.md index 4ad5abb4..797d7c5f 100644 --- a/claude-notes/plans/2026-03-17-extensions-phase2-filter-resolution.md +++ b/claude-notes/plans/2026-03-17-extensions-phase2-filter-resolution.md @@ -1,8 +1,8 @@ # Extensions Phase 2: Extension Filter Resolution **Created**: 2026-03-17 -**Status**: Not Started -**Parent Plan**: `claude-notes/plans/2026-03-16-extensions-master-plan.md` +**Status**: Complete (merged as `cffc2e6c`) +**Parent Plan**: `claude-notes/plans/2026-03-16-extensions-grand-plan.md` **Depends on**: Phase 1 (complete), Lua filter support (complete, rebased) ## What Phase 1 Built (already on this branch) @@ -485,7 +485,7 @@ extension filter smoke tests below follow the same pattern. - [x] **2.5.1** `cargo build --workspace` — clean build - [x] **2.5.2** `cargo nextest run --workspace` — all tests pass -- [x] **2.5.3** Update master plan to mark Phase 2 complete +- [x] **2.5.3** Update grand plan to mark Phase 2 complete --- diff --git a/claude-notes/plans/2026-03-20-extensions-phase3-shortcode-resolution.md b/claude-notes/plans/2026-03-20-extensions-phase3-shortcode-resolution.md new file mode 100644 index 00000000..4fec256b --- /dev/null +++ b/claude-notes/plans/2026-03-20-extensions-phase3-shortcode-resolution.md @@ -0,0 +1,808 @@ +# Extensions Phase 3: Shortcode Resolution + +**Created**: 2026-03-20 +**Status**: COMPLETE +**Branch**: `feature/shortcode-extensions` +**Parent Plan**: `claude-notes/plans/2026-03-16-extensions-grand-plan.md` +**Depends on**: Phase 1 (complete), Phase 2 (complete), Lua filter support (complete) + +## HANDOFF STATUS (read this first if resuming) + +Phases 3.1, 3.2, 3.3 are **complete and committed** (commits `19c926b2` and `d3570f4d`). + +Phase 3.4.1-3.4.5 are **complete** (uncommitted). All 6912 workspace tests pass. + +Remaining work: +- **3.5**: Integration tests (metadata merge + end-to-end) +- **3.6**: Smoke tests (real extension shortcode rendering) +- **3.7**: Workspace verification (`cargo xtask verify`) and commit + +## What Already Exists + +### q2 shortcode infrastructure + +- **Parser**: Tree-sitter grammar parses `{{< name args >}}` into `Inline::Shortcode` nodes. + Shortcodes are always inline — there is no `Block::Shortcode` variant. +- **AST type**: `Shortcode` struct in `quarto-pandoc-types/src/shortcode.rs` with `is_escaped`, + `name`, `positional_args`, `keyword_args`, `source_info`. +- **Transform**: `ShortcodeResolveTransform` in `quarto-core/src/transforms/shortcode_resolve.rs` + walks the AST and dispatches to `ShortcodeHandler` trait implementations. +- **Built-in handlers**: Only `MetaShortcodeHandler` (`{{< meta key >}}`). +- **Pipeline position**: Runs in `AstTransformsStage`, after callout resolution, before metadata + normalization. The transform pipeline is built statically by `build_transform_pipeline()`. +- **Result type**: `ShortcodeResult` has `Inlines(Vec)`, `Error(ShortcodeError)`, `Preserve`. + No block-level result variant. +- **Context limitation**: `ShortcodeResolveTransform` receives `RenderContext` which has no access + to extensions, runtime, or target format. `AstTransformsStage` has `StageContext` (with extensions + and runtime) but doesn't pass them to transforms. + +### Extension shortcode storage (Phase 1) + +- **Top-level**: `Contributes.shortcodes: Vec` — absolute paths, parsed by + `parse_shortcodes()` in `extension/read.rs`. +- **Per-format**: `Contributes.formats: HashMap` — format metadata may + contain a `shortcodes` key with relative paths as plain strings. NOT currently marked as + `ConfigValueKind::Path` by `mark_path_valued_keys()`. +- **Discovery**: `ctx.extensions: Vec` on `StageContext`, populated during context + creation. + +### Lua engine (pampa) + +- `apply_lua_filter()` in `pampa/src/lua/filter.rs` creates a fresh `Lua` state per filter + invocation. Calls `register_pandoc_namespace()` to set up `pandoc.*`, `quarto.*` globals. +- `register_pandoc_namespace(lua, runtime, mediabag)` in `pampa/src/lua/constructors.rs` + registers inline/block constructors, utils, text, JSON, path namespaces. Requires a + `SharedMediaBag` argument (created via `create_shared_mediabag()` from + `pampa/src/lua/mediabag.rs`). +- Lua state setup handles WASM (restricted StdLib) vs native (full). +- No shortcode-specific Lua API exists yet. + +## How TS Quarto Does It + +### Shortcode handler loading + +`initShortcodeHandlers()` in `quarto-pre/shortcodes-handlers.lua` loads all shortcode Lua +scripts into a shared `handlers` table. Each script can register handlers two ways: + +1. **Return a table**: `return { hello = function(args, kwargs, meta, raw_args, context) ... end }` +2. **Define in environment**: Functions defined in the script's environment are harvested. + +Built-in handlers (`meta`, `env`, `var`, `pagebreak`, `brand`, `contents`) are registered +AFTER user/extension handlers, so built-ins override same-named user handlers. + +### Shortcode calling convention + +`callShortcodeHandler()` calls: `handler.handle(args, kwargs, meta, raw_args, context)` where: +- `args` — `pandoc.List` of `{value: string}` or `{name: key, value: string}` tables +- `kwargs` — table keyed by name, with metatable defaulting missing keys to empty Inlines +- `meta` — metatable proxy that reads document metadata via `readMetadata()` +- `raw_args` — flat list of raw string values +- `context` — `"block"`, `"inline"`, or `"text"` + +### Block vs inline context (two-pass) + +`shortcodes_filter()` in `customnodes/shortcodes.lua` uses two passes: + +1. **First pass** (block context): Walks `Para` and `Plain` nodes. If the node contains a + single shortcode, calls handler with `context = "block"`. Result is converted via + `shortcodeResultAsBlocks()` — handler can return Blocks, Inlines (wrapped in Para), + or a string (wrapped in Para). The block result replaces the original Para/Plain. + +2. **Second pass** (inline context): Walks remaining `Shortcode` nodes. Calls handler with + `context = "inline"`. Result is converted via `shortcodeResultAsInlines()` — Blocks + are flattened to inlines, strings become `Str`. + +A third context `"text"` is used for shortcodes found inside code blocks and attributes +(parsed by LPEG, resolved to plain strings). + +### Extension shortcode collection + +Two independent paths (same pattern as filters): + +- **Top-level** (`contributes.shortcodes`): `extensionShortcodes()` in `filters.ts` iterates + all extensions, collects `contributes.shortcodes` paths. These are always active. +- **Per-format** (`contributes.formats.html.shortcodes`): Flow through format metadata + resolution in `readExtensionFormat()`. Paths resolved to absolute. Active only for + matching format. + +Both sets are passed as the `kShortcodes` parameter to the Lua filter pipeline. + +### `quarto.shortcode` Lua API + +Two helper functions available to shortcode handlers: +- `quarto.shortcode.read_arg(args, n)` — reads nth argument, handles Inlines-to-string +- `quarto.shortcode.error_output(shortcode, message_or_args, context)` — formatted error + output as Blocks, Inlines, or text depending on context + +## Design Decisions + +### Threading extensions and runtime into the transform + +`ShortcodeResolveTransform` needs access to extension shortcode paths, `SystemRuntime` +(to read Lua files via VFS), and the target format string. Rather than expanding +`RenderContext`, the transform is constructed with these as **owned data** at setup time. + +**Key constraint**: `AstTransform: Send + Sync`, but mlua's `Lua` is `!Send + !Sync` +(the `send` feature is not enabled). Therefore the `LuaShortcodeEngine` (which holds a +`Lua` state) **cannot be stored as a field** on `ShortcodeResolveTransform`. Instead: + +- `ShortcodeResolveTransform::new()` stores only `Send + Sync` data: + `Vec`, `Vec`, `Arc`, `String`. + The format string stored is `ctx.format.identifier.as_str()` (the base format like + `"html"`, not the full `"acm-html"` descriptor). This is passed to + `LuaShortcodeEngine::new()` and set as the Lua `FORMAT` global. + Note: `Format.identifier` is a `FormatIdentifier` enum; `.as_str()` returns the + base format string. `Format.target_format` is the full descriptor including + extension prefix (e.g., `"acm-html"`). Use `identifier.as_str()` for the Lua + `FORMAT` global, matching `UserFiltersStage` at `user_filters.rs:135`. +- `ShortcodeResolveTransform::transform()` creates a `LuaShortcodeEngine` **on the stack**, + loads scripts, resolves all shortcodes, and lets the engine drop at the end. +- This gives one Lua state per render, reused across all shortcodes in the document — + same as described in Performance Notes. + +**Pipeline construction timing**: Currently `AstTransformsStage::new()` builds the +transform pipeline at construction time via `build_transform_pipeline()`. But the +extensions, runtime, and format are only available at run time via `StageContext`. +Therefore: + +- `AstTransformsStage` stores `Option` instead of `TransformPipeline`. +- `new()` stores `None`. `run()` calls `build_transform_pipeline(...)` with `StageContext` + data to build the pipeline just-in-time. +- `with_pipeline(p)` stores `Some(p)` — used as-is in `run()`, preserving the existing + test pattern (`AstTransformsStage::with_pipeline(TransformPipeline::new())`). + +`build_transform_pipeline()` gains parameters for the shortcode transform. Other transforms +in the pipeline are unaffected — they continue to take no construction parameters. + +**Verified**: `SystemRuntime: Send + Sync` (trait bound in `quarto_system_runtime`). +`Extension` derives `Clone` with all `Send + Sync` fields. `Arc` is +`Send + Sync`. pampa re-exports `quarto_system_runtime::SystemRuntime` — they are the +same trait. + +### Per-format shortcode paths through metadata merge + +Per-format shortcodes (`contributes.formats.html.shortcodes`) go through metadata merge +(Option A from discussion). Add `"shortcodes"` to `mark_path_valued_keys()` — simpler +than filters since shortcode entries are always plain string paths (no map form, no +reserved names like `citeproc`/`quarto`). + +After merge, `meta["shortcodes"]` contains rebased paths from both extensions and user +frontmatter. The transform collects these paths and loads them into the Lua state. + +Top-level shortcodes (`contributes.shortcodes`) are collected by name-based resolution +when the transform encounters an unknown shortcode name — same pattern as Phase 2's +filter name resolution. + +### Block-level shortcode support + +Add `ShortcodeResult::Blocks(Vec)` variant. The resolution logic uses two passes +matching TS Quarto: + +1. In `resolve_blocks()`, detect `Para`/`Plain` containing exactly one `Inline::Shortcode`. + Call handler with block context. If result is `Blocks`, splice them in place of the + `Para`/`Plain`. If `Inlines`, replace the shortcode inline as usual. + +2. Remaining inline shortcodes resolved as before. + +This means Lua handlers (e.g., a `pagebreak` extension) can return `pandoc.RawBlock()` +and it will replace the paragraph. Rust built-in handlers can also return +`ShortcodeResult::Blocks` if needed in the future. + +### Lua shortcode engine in pampa + +New module `pampa/src/lua/shortcode.rs` with a `LuaShortcodeEngine` struct. + +**Important**: `LuaShortcodeEngine` is `!Send + !Sync` because it holds a `Lua` state. +It is **never stored as a struct field** on any `Send + Sync` type. It is only created +as a local variable inside `ShortcodeResolveTransform::transform()`, used for the +duration of that call, and dropped afterward. + +```rust +pub struct LuaShortcodeEngine { + lua: Lua, // !Send + !Sync + handler_names: Vec, // registered handler names for diagnostics + runtime: Arc, // for on-demand script loading +} + +impl LuaShortcodeEngine { + /// Create engine, set up Lua state with pandoc/quarto globals. + /// Internally creates a SharedMediaBag (via create_shared_mediabag()) and calls + /// register_pandoc_namespace(lua, runtime, mediabag) — same as apply_lua_filter(). + pub fn new(target_format: &str, runtime: Arc) -> Result; + + /// Load a shortcode Lua script. Registers all handlers it defines. + /// Supports both return-table and environment-function conventions. + pub fn load_script(&mut self, script_path: &Path) -> Result<()>; + + /// Call a named shortcode handler. + /// Returns None if no handler is registered for the name. + pub fn call( + &self, + name: &str, + shortcode: &Shortcode, + metadata: &ConfigValue, + context: ShortcodeCallContext, + ) -> Option; + + /// Check if a handler is registered for the given name. + pub fn has_handler(&self, name: &str) -> bool; +} + +/// Context in which a shortcode is being resolved (block, inline, or text). +/// Named `ShortcodeCallContext` to avoid conflict with the existing +/// `ShortcodeContext` struct in `shortcode_resolve.rs` (which holds +/// metadata + source_info for resolution). +pub enum ShortcodeCallContext { Block, Inline, Text } + +pub enum LuaShortcodeResult { + Inlines(Vec), + Blocks(Vec), + Text(String), + Error(String), +} +``` + +This is distinct from `apply_lua_filter` — no AST traversal, just function dispatch. +A single `LuaShortcodeEngine` is created per render (inside `transform()`) and reused +for all shortcode invocations in the document. Cross-render caching (reusing the engine +between keystrokes) is a future optimization that would benefit both filters and shortcodes. + +### Handler name collision priority + +1. **Built-in Rust handlers** (e.g., `MetaShortcodeHandler`) — always win +2. **Lua handlers loaded later** override earlier ones (within the Lua engine) +3. Extension shortcodes are loaded before user-specified shortcodes, so user wins + +The **outcome** matches TS Quarto (built-ins always win, user overrides extensions), +but the **mechanism** differs: TS Quarto registers built-ins last so they overwrite +earlier entries. In q2, the Rust handlers are checked first (before Lua dispatch), +achieving the same effect without registering built-ins in Lua. + +### `quarto.shortcode` Lua API + +Register `quarto.shortcode.read_arg(args, n)` and `quarto.shortcode.error_output(name, args, context)` +in the Lua state during `LuaShortcodeEngine::new()`. These are compatibility APIs needed +by existing TS Quarto extensions. + +### Handler calling convention + +Convert q2's `Shortcode` struct to the TS Quarto Lua calling convention: +- `positional_args` → `args` (pandoc.List of `{value = string}`) +- `keyword_args` → `kwargs` (table keyed by name) +- `metadata` → `meta` (metatable proxy reading from `ConfigValue`) +- positional args as strings → `raw_args` +- block/inline/text → `context` + +The result conversion does **not** reuse `handle_inline_return`/`handle_block_return` +from `filter.rs`. Those functions have filter-specific semantics that are wrong for +shortcodes: `nil` → clone original (filters mean "no change"; shortcodes mean "handler +failed"), catch-all `_` → clone original (swallows `Value::String` which is the most +common shortcode return type), and they require an `&Inline`/`&Block` "original" +parameter that doesn't exist in shortcode context (shortcodes call freestanding Lua +functions, not element callbacks). + +Instead, `LuaShortcodeEngine::call()` has its own top-level return dispatch matching +TS Quarto's `shortcodeResultAsInlines`/`shortcodeResultAsBlocks`: + +- `Value::Nil` → `LuaShortcodeResult::Error("Shortcode '...' returned nil")` +- `Value::String(s)` → `LuaShortcodeResult::Text(s)` +- `Value::UserData` → try `LuaInline` first, then `LuaBlock`; produce `Inlines` + or `Blocks` accordingly +- `Value::Table` → iterate elements, try each as `LuaInline` or `LuaBlock`, + classify the collection (all inlines → `Inlines`, any blocks → `Blocks`) +- Other → `LuaShortcodeResult::Error(...)` + +To avoid duplicating the low-level userdata extraction, new `pub(crate)` helper +functions are created in `filter.rs` wrapping the borrow pattern (the existing code +inlines `ud.borrow::()?.0.clone()` directly in match arms): +- `extract_lua_inline(ud: &UserData) -> Result` +- `extract_lua_block(ud: &UserData) -> Result` +- `extract_lua_inlines_from_table(table: &Table) -> Result>` +- `extract_lua_blocks_from_table(table: &Table) -> Result>` + +The shortcode engine composes these primitives with its own nil/string/error handling. + +--- + +## Work Items + +### Phase 3.1: Mark per-format shortcode paths as `!path` + +Add `"shortcodes"` to `PATH_VALUED_KEYS` in `extension/read.rs`. Simpler than filters — +entries are always plain string paths, no map form, no reserved names. The existing +generic array handling (lines 276-284) already converts each `Scalar(String)` element +to `ConfigValueKind::Path`, so no custom code is needed. + +- [x] **3.1.1** Add `"shortcodes"` to the `PATH_VALUED_KEYS` constant (line 219). + +- [x] **3.1.2** Tests in `read.rs`: + - `test_format_shortcode_paths_marked`: `shortcodes: [handler.lua]` → `ConfigValueKind::Path` + - `test_format_shortcode_multiple_paths_marked`: array with multiple entries, all marked + - `test_shortcode_marking_doesnt_affect_other_keys`: `toc`, `theme` etc unchanged + +### Phase 3.2: `LuaShortcodeEngine` in pampa + +New module for loading and dispatching Lua shortcode handlers. + +- [x] **3.2.1** Create `pampa/src/lua/shortcode.rs` with `LuaShortcodeEngine` struct, + `ShortcodeCallContext` enum, `LuaShortcodeResult` enum. + +- [x] **3.2.2** Implement `LuaShortcodeEngine::new()`: create Lua state (WASM-aware), + create a `SharedMediaBag` via `create_shared_mediabag()` (required third argument to + `register_pandoc_namespace(lua, runtime, mediabag)`), call + `register_pandoc_namespace()`, set `FORMAT` global, register `quarto.shortcode` + sub-namespace with `read_arg` and `error_output`. + +- [x] **3.2.3** Implement `LuaShortcodeEngine::load_script()`: read script via runtime, + execute in sandboxed environment. Scan for handlers via both conventions: + - If script returns a table, iterate keys as handler names + - Otherwise, scan environment for callable values + Register all found handlers in an internal `HashMap`. + +- [x] **3.2.4** Implement `LuaShortcodeEngine::call()`: look up handler by name, convert + `Shortcode` args to Lua tables matching TS Quarto convention, call handler, convert + result back to Rust types using shortcode-specific dispatch: + - `Value::Nil` → `LuaShortcodeResult::Error` (handler produced no output) + - `Value::String` → `LuaShortcodeResult::Text` + - `Value::UserData` → try `extract_lua_inline`, then `extract_lua_block` + - `Value::Table` → iterate with `extract_lua_inlines_from_table` / + `extract_lua_blocks_from_table`, classify collection + - Other → `LuaShortcodeResult::Error` + +- [x] **3.2.4a** Create new `pub(crate)` helper functions in `pampa/src/lua/filter.rs` + for reuse by the shortcode engine. These are **new functions**, not extractions of + existing ones — the current code inlines the borrow pattern directly in match arms + (e.g., `ud.borrow::()?.0.clone()` at `filter.rs:318-319`). The helpers + wrap this pattern: + - `extract_lua_inline(ud: &UserData) -> Result` — `ud.borrow::()?.0.clone()` + - `extract_lua_block(ud: &UserData) -> Result` — `ud.borrow::()?.0.clone()` + - `extract_lua_inlines_from_table(table: &Table) -> Result>` — iterate + table entries, call `extract_lua_inline` on each UserData + - `extract_lua_blocks_from_table(table: &Table) -> Result>` — iterate + table entries, call `extract_lua_block` on each UserData + The top-level `handle_inline_return` / `handle_block_return` are NOT reused (their + nil/fallback semantics are filter-specific and wrong for shortcodes). + Optionally, refactor `handle_inline_return`/`handle_block_return` to call these + new helpers, but this is not required — the existing code works fine. + +- [x] **3.2.5** Register module in `pampa/src/lua/mod.rs`. + +- [x] **3.2.6** Tests: + - `test_load_script_return_table`: script returns `{hello = function() ... end}` → + handler registered + - `test_load_script_env_function`: script defines `function hello() ... end` → + handler registered + - `test_call_returns_inlines`: handler returns `pandoc.Inlines{pandoc.Str("hi")}` → + `LuaShortcodeResult::Inlines` + - `test_call_returns_blocks`: handler returns `pandoc.RawBlock("html", "
")` → + `LuaShortcodeResult::Blocks` + - `test_call_returns_string`: handler returns `"hello"` → + `LuaShortcodeResult::Text` + - `test_call_returns_nil`: handler returns `nil` → + `LuaShortcodeResult::Error` + - `test_call_unknown_handler`: no handler for name → returns `None` + - `test_handler_receives_args`: handler that echoes first arg → correct value + - `test_handler_receives_kwargs`: handler that reads named arg → correct value + - `test_handler_receives_meta`: handler reads `meta.title` → correct value + - `test_handler_receives_context`: handler returns context string → matches + - `test_later_script_overrides_earlier`: two scripts defining same name → last wins + - `test_read_arg_helper`: Lua code using `quarto.shortcode.read_arg()` works + - `test_wasm_lua_state`: (cfg wasm32) engine creates successfully with restricted libs + +### Phase 3.3: Block-level shortcode support + +Add `ShortcodeResult::Blocks` variant and two-pass resolution logic. + +- [x] **3.3.1** Add `Blocks(Vec)` variant to `ShortcodeResult` enum. + +- [x] **3.3.2** Add `ResolutionContext` enum to `shortcode_resolve.rs`: `Block`, `Inline`. + This is distinct from the existing `ShortcodeContext` struct (which holds metadata + + source_info) and from pampa's `ShortcodeCallContext` (which also includes `Text`). + Pass `ResolutionContext` to `ShortcodeHandler::resolve()` (signature change). + Known affected sites (compiler will find any missed): + - Trait def (line 86), `resolve_shortcode()` (line 237) — add parameter + - `MetaShortcodeHandler::resolve()` (line 102) — add param, ignore it + - `resolve_inlines()` (line 442) — pass `ResolutionContext::Inline` + - 5 test functions (lines 744, 776, 797, 823, 838) — add `ResolutionContext::Inline` + Run `cargo check -p quarto-core` after the signature change to find all sites. + +- [x] **3.3.3** Update `resolve_blocks()`: change from `for block in blocks.iter_mut()` + to **index-based iteration** (like `resolve_inlines()` already does at line 434), + because splicing block results requires replacing one element with multiple. + + The current structure delegates per-block work to `resolve_block()`. The new logic + adds a **block-context shortcode check** at the `resolve_blocks()` level, before + falling through to `resolve_block()` for the general case: + + ```rust + fn resolve_blocks(blocks: &mut Vec, transform: &..., metadata: &..., diagnostics: &mut ...) { + let mut i = 0; + while i < blocks.len() { + // Check for block-context shortcode: Para/Plain with exactly one Shortcode + if let Some(shortcode) = single_shortcode_in_para_or_plain(&blocks[i]) { + let ctx = ShortcodeContext { metadata, source_info: &shortcode.source_info }; + match transform.resolve_shortcode(shortcode, &ctx, ResolutionContext::Block) { + ShortcodeResult::Blocks(new_blocks) => { + let n = new_blocks.len(); + blocks.splice(i..=i, new_blocks); + i += n.max(1); // advance past spliced blocks + continue; + } + ShortcodeResult::Inlines(inlines) => { + // Replace the shortcode inline within the Para/Plain + replace_shortcode_in_block(&mut blocks[i], inlines); + i += 1; + continue; + } + ShortcodeResult::Error(err) => { /* same as inline error handling */ } + ShortcodeResult::Preserve => { /* convert to literal */ } + } + } + // General case: recurse into block (handles mixed-content Para, Div, lists, etc.) + resolve_block(&mut blocks[i], transform, metadata, diagnostics); + i += 1; + } + } + ``` + + Helper `single_shortcode_in_para_or_plain(block: &Block) -> Option<&Shortcode>`: + returns `Some` if the block is `Para`/`Plain` with `content.len() == 1` and + `content[0]` is `Inline::Shortcode` (and not escaped). Returns `None` otherwise. + + Helper `replace_shortcode_in_block(block: &mut Block, inlines: Vec)`: + replaces the single `Inline::Shortcode` in the Para/Plain content with the inlines. + +- [x] **3.3.4** Handle `ShortcodeResult::Blocks` in `resolve_inlines()` (graceful + degradation). When a shortcode in inline context returns `Blocks`, flatten them + to inlines using the existing `flatten_blocks_to_inlines()` helper (line 200). + Add a new match arm in `resolve_inlines()` after the `Inlines` arm: + ```rust + ShortcodeResult::Blocks(blocks) => { + let replacement = flatten_blocks_to_inlines(&blocks); + let replacement_len = replacement.len(); + inlines.splice(i..=i, replacement); + i += replacement_len.max(1); + } + ``` + +- [x] **3.3.5** Update `MetaShortcodeHandler` to accept context parameter (always returns + Inlines regardless of context — no behavior change). + +- [x] **3.3.6** Tests: + - `test_block_shortcode_replaces_para`: Para with single shortcode returning Blocks → + Para replaced by those Blocks + - `test_inline_shortcode_in_para_stays_inline`: Para with text + shortcode → resolved + as inline (not block context) + - `test_block_result_in_inline_context`: shortcode in inline context returns Blocks → + flattened to Inlines via `flatten_blocks_to_inlines` (graceful degradation) + - `test_escaped_shortcode_block_context`: escaped shortcode alone in Para → preserved + as literal text + +### Phase 3.4: Wire extensions and Lua into `ShortcodeResolveTransform` + +Connect the Lua engine to the transform, collecting shortcode scripts from both metadata +and extension lookup. + +- [x] **3.4.1** Add parameterized constructor to `ShortcodeResolveTransform`: + ```rust + pub fn with_lua_support( + lua_shortcode_paths: Vec, // from merged metadata + extensions: Vec, // owned, for name-based lookup + runtime: Arc, + target_format: String, + ) -> Self + ``` + Constructor stores all parameters as owned fields (`Send + Sync`). Does **not** create + `LuaShortcodeEngine` — that happens in `transform()`. Keep existing `new()` (no args) + for backward compatibility in tests that only need built-in handlers. + +- [x] **3.4.2** Update `transform()` to create `LuaShortcodeEngine` on the stack: + ```rust + fn transform(&self, ast: &mut Pandoc, ctx: &mut RenderContext) -> Result<()> { + // Create Lua engine if we have paths or extensions + let mut engine = if !self.lua_shortcode_paths.is_empty() + || !self.extensions.is_empty() { + let mut e = LuaShortcodeEngine::new(&self.target_format, self.runtime.clone())?; + for path in &self.lua_shortcode_paths { + e.load_script(path)?; + } + Some(e) + } else { + None + }; + // Pass engine.as_mut() to resolution functions alongside &self.handlers + // ... + // engine dropped here + } + ``` + Resolution functions (`resolve_blocks`, `resolve_inlines`, etc.) gain an + `Option<&mut LuaShortcodeEngine>` parameter alongside the existing + `&ShortcodeResolveTransform` (or just `&[Box]`). + +- [x] **3.4.3** Update `build_transform_pipeline()` to accept parameters needed by the + shortcode transform. Other transforms continue to take no parameters. + The `target_format` is `ctx.format.identifier.as_str()` (base format like `"html"`). + ```rust + pub fn build_transform_pipeline( + shortcode_paths: Vec, // from extract_shortcode_paths() + extensions: Vec, // from ctx.extensions.clone() + runtime: Arc, // from ctx.runtime.clone() + target_format: String, // from ctx.format.identifier.as_str().to_string() + ) -> TransformPipeline + ``` + **Only one call site** exists: `AstTransformsStage::new()` at `ast_transforms.rs:64`. + Pipeline builders (`build_html_pipeline_stages`, etc.) call `AstTransformsStage::new()`, + not `build_transform_pipeline()` directly. Since `new()` is changing to store `None` + (pipeline built lazily in `run()`), the no-arg `build_transform_pipeline()` becomes + unused by `new()`. Change its signature to accept the shortcode params directly. + No other call sites need updating. + +- [x] **3.4.4** Update `AstTransformsStage` to build the pipeline in `run()`: + - Change field from `pipeline: TransformPipeline` to + `custom_pipeline: Option`. + - `new()` stores `None`. + - `with_pipeline(p)` stores `Some(p)` — preserves existing test pattern. + - In `run()`: if `custom_pipeline` is `Some`, use it. Otherwise, extract data from + `StageContext` and call `build_transform_pipeline(...)`: + - `ctx.extensions.clone()` — `Vec`, all discovered extensions + - `ctx.runtime.clone()` — `Arc` + - `ctx.format.identifier.as_str().to_string()` — base format (e.g., `"html"`) + - Shortcode paths from merged metadata: extract `doc.ast.meta["shortcodes"]` as + `Vec`. After metadata merge + `adjust_paths_to_document_dir()`, the + array contains `ConfigValueKind::Path(s)` entries where `s` is a path relative + to the document directory. Convert via: + ```rust + fn extract_shortcode_paths(meta: &ConfigValue, document_dir: &Path) -> Vec { + let Some(sc_val) = meta.get("shortcodes") else { return vec![] }; + let Some(items) = sc_val.as_array() else { return vec![] }; + items.iter().filter_map(|item| { + match &item.value { + ConfigValueKind::Path(s) => Some(document_dir.join(s)), + ConfigValueKind::Scalar(_) => item.as_str().map(|s| document_dir.join(s)), + _ => None, + } + }).collect() + } + ``` + The `document_dir` is `ctx.document.input.parent()` (same pattern as + `UserFiltersStage` at `user_filters.rs:101-105`). The `Scalar` fallback + handles user-frontmatter shortcodes that weren't marked as Path (user + paths aren't processed by `mark_path_valued_keys`, which only runs on + extension format metadata). + +- [x] **3.4.5** Update resolution logic: when a shortcode name doesn't match a built-in + Rust handler AND isn't in the Lua engine's loaded handlers, try name-based extension + lookup via `find_extension(name, &self.extensions)` (import from + `crate::extension::discover::find_extension`, same as `filter_resolve.rs:15` and + `metadata_merge.rs:31`). If the extension contributes shortcodes + (`ext.contributes.shortcodes`), load them into the engine on demand + (`engine.load_script()`) and retry dispatch. The engine is `&mut` so on-demand + loading works. + +- [x] **3.4.6** Tests: + - `test_lua_shortcode_from_metadata_paths`: Lua script path in metadata → handler works + - `test_lua_shortcode_by_extension_name`: `{{< my-ext >}}` with matching extension → + extension's shortcode scripts loaded and handler called + - `test_rust_handler_overrides_lua`: both Rust `meta` and Lua `meta` handler → + Rust handler wins + - `test_unknown_shortcode_error`: no handler anywhere → diagnostic error + - `test_extension_shortcode_block_context`: extension shortcode returning Blocks → works + +### Phase 3.5: Integration tests + +- [x] **3.5.1** Test in `metadata_merge.rs`: + `test_extension_format_shortcode_paths_rebased_through_merge`: extension contributes + `formats.html.shortcodes: [handler.lua]` → after merge, path resolves to extension dir. + +- [x] **3.5.2** Test in `shortcode_resolve.rs`: + `test_full_transform_with_lua_shortcode`: end-to-end test with Lua script file, extension + discovery, shortcode in AST → resolved content appears. + +- [x] **3.5.3** Test in `shortcode_resolve.rs`: + `test_full_transform_block_shortcode`: Lua handler returns `pandoc.RawBlock(...)`, + shortcode alone in Para → Para replaced with RawBlock. + +### Phase 3.6: Smoke tests + +- [x] **3.6.1** Create `crates/quarto/tests/smoke-all/extensions/shortcode-extension/`: + Extension with `contributes.shortcodes: [hello.lua]`. Document uses `{{< hello >}}`. + `hello.lua` returns `pandoc.Inlines{pandoc.Str("HELLO-SHORTCODE-ACTIVE")}`. + Assert: `ensureFileRegexMatches: [["HELLO-SHORTCODE-ACTIVE"]]`. + +- [x] **3.6.2** Create `crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/`: + Format extension with `contributes.formats.html.shortcodes: [greeting.lua]`. + Document uses test key `myext-html` and `{{< greeting >}}`. + Assert: shortcode output appears in HTML. + +- [x] **3.6.3** Create smoke test for block-level shortcode: + Extension shortcode that returns `pandoc.RawBlock("html", "
")`. + Document has `{{< break >}}` alone on a line. + Assert: `ensureHtmlElements: [["hr.ext-break"]]`. + +- ~~**3.6.4**~~ Removed — lipsum is a future built-in shortcode (see Future Work section), + not part of the infrastructure delivered by Phase 3. + +### Phase 3.7: Workspace verification + +- [x] **3.7.1** `cargo build --workspace` — clean build +- [x] **3.7.2** `cargo nextest run --workspace` — all 6919 tests pass +- [x] **3.7.3** `cargo xtask verify` — lint, format, build with `-D warnings` all pass + (tree-sitter CLI not installed on this machine — pre-existing) +- [x] **3.7.4** Update grand plan to mark Phase 3 complete + +--- + +## Files to Create/Modify + +| File | Action | Description | +|------|--------|-------------| +| `crates/pampa/src/lua/shortcode.rs` | Create | (3.2) `LuaShortcodeEngine`, `ShortcodeCallContext`, `LuaShortcodeResult` | +| `crates/pampa/src/lua/filter.rs` | Modify | (3.2.4a) Extract userdata helpers as `pub(crate)` | +| `crates/pampa/src/lua/mod.rs` | Modify | (3.2) Register shortcode module | +| `crates/quarto-core/src/extension/read.rs` | Modify | (3.1) Add `"shortcodes"` to `mark_path_valued_keys()` | +| `crates/quarto-core/src/transforms/shortcode_resolve.rs` | Modify | (3.3-3.4) Block support, Lua dispatch, extension lookup. Also remove unused `use quarto_analysis::AnalysisContext` import (line 46). | +| `crates/quarto-core/src/pipeline.rs` | Modify | (3.4) `build_transform_pipeline()` accepts shortcode params | +| `crates/quarto-core/src/stage/stages/ast_transforms.rs` | Modify | (3.4) Pass StageContext data to transform pipeline | +| `crates/quarto/tests/smoke-all/extensions/shortcode-extension/` | Create | (3.6) Smoke test | +| `crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/` | Create | (3.6) Smoke test | + +--- + +## Performance Notes + +The `LuaShortcodeEngine` creates one Lua state per render (inside `transform()`) and +reuses it for all shortcode invocations in the document. This is important because a +document may contain dozens of shortcodes — creating a Lua state per invocation would +be expensive. + +The transform pipeline is also rebuilt per render (in `AstTransformsStage::run()`). +This is negligible — it allocates a `Vec` of ~11 small structs. The shortcode transform's +`with_lua_support()` constructor stores only owned data; no Lua initialization happens +until `transform()` is called. + +The Lua state is NOT cached across renders. In the hub-client, the entire pipeline +(including `AstTransformsStage`) is rebuilt per keystroke. Cross-render caching of the +Lua state is a future optimization opportunity that would benefit both shortcodes and +filters — it's not specific to this phase. The key insight is that extension Lua files +don't change between keystrokes; only document content does. + +Lua filters have the same per-render cost today (`apply_lua_filter` creates a fresh +`Lua::new()` per filter per render). A caching layer would be pipeline-level +infrastructure, tracked separately. + +--- + +## Risks and Open Questions + +All risks have been resolved through research. See "Resolved Design Questions" below. + +## Resolved Design Questions + +These were identified during plan review and are now resolved: + +- **`Lua` is `!Send + !Sync`**: Resolved by creating the `LuaShortcodeEngine` inside + `transform()` as a stack-local variable, never storing it in the transform struct. + Same pattern as `apply_lua_filter()` which creates a fresh `Lua` per call. + +- **Pipeline construction timing**: Resolved by moving pipeline construction from + `AstTransformsStage::new()` to `run()`, using `Option` to preserve + the `with_pipeline()` test pattern. + +- **`SystemRuntime` trait identity**: Confirmed that pampa re-exports + `quarto_system_runtime::SystemRuntime` — they are the same trait. `Arc` + is `Send + Sync` because the trait has `Send + Sync` supertraits. + +- **`ShortcodeContext` naming conflict**: The existing `ShortcodeContext` struct (metadata + + source_info) is kept. The new block/inline enum in `shortcode_resolve.rs` is named + `ResolutionContext`. The pampa-side enum is named `ShortcodeCallContext`. + +- **Ownership for extensions**: `ShortcodeResolveTransform` stores `Vec` (owned, + cloned from `StageContext`). `Extension` derives `Clone` with all `Send + Sync` fields. + +- **Userdata unwrapping for shortcode results**: The filter engine's + `handle_inline_return` / `handle_block_return` (`filter.rs:313,343`) are **not** + reused — their semantics (nil → keep original, unknown → keep original, requires + `&Inline`/`&Block` original param) are filter-specific and wrong for shortcodes. + Instead, **new** `pub(crate)` helper functions are created in `filter.rs` wrapping + the borrow pattern (the existing code inlines `ud.borrow::()?.0.clone()` + directly in match arms — there are no existing standalone functions to extract). + The shortcode engine builds its own top-level dispatch using these helpers: + nil → error, string → text, userdata/table → use helpers, other → error. + This matches TS Quarto's `shortcodeResultAsInlines`/`shortcodeResultAsBlocks`. + +- **`quarto.shortcode.error_output`**: Implementable as pure Lua registered during + `LuaShortcodeEngine::new()`. Calls existing pandoc constructors (`pandoc.Para`, + `pandoc.Strong`, `pandoc.Str`) from `register_pandoc_namespace()`. No Rust needed. + +- **`build_transform_pipeline` signature change**: Only one call site exists + (`AstTransformsStage::new()` → moves to `run()`). `with_pipeline()` bypasses it + entirely. Tests using `ShortcodeResolveTransform::new()` (no args) are unaffected. + +- **`Format` field for target format string**: `Format` has three relevant fields: + `identifier: FormatIdentifier` (enum, use `.as_str()` for `"html"`), + `target_format: String` (full string like `"acm-html"`), + `extension_name: Option` (e.g., `Some("acm")`). + Use `identifier.as_str()` for the Lua `FORMAT` global — this gives the base format + like `"html"`, matching `UserFiltersStage` at `user_filters.rs:135`. + +- **Extracting shortcode paths from merged metadata**: After metadata merge, + `doc.ast.meta["shortcodes"]` contains an array of `ConfigValueKind::Path` entries + (rebased relative to document dir by `adjust_paths_to_document_dir()`). Extract + with `document_dir.join(s)` for each `Path(s)` entry. Also handle `Scalar` fallback + for user-frontmatter shortcodes not processed by `mark_path_valued_keys`. + +- **`ShortcodeResult::Blocks` in inline context**: When `resolve_inlines()` encounters + a `Blocks` result, it calls the existing `flatten_blocks_to_inlines()` (line 200) + for graceful degradation. This matches TS Quarto's `shortcodeResultAsInlines()`. + +- **`ShortcodeHandler::resolve()` signature change**: Mechanical update. Affected sites + in `shortcode_resolve.rs` (all add `ResolutionContext::Inline`): + - Trait def (line 86), `resolve_shortcode()` (line 237) — add parameter + - `MetaShortcodeHandler::resolve()` (line 102) — add param, ignore it + - `resolve_inlines()` (line 442) — pass `ResolutionContext::Inline` + - 5 test functions (lines 744, 776, 797, 823, 838) — add `ResolutionContext::Inline` + +--- + +## Future Work: Built-in Shortcodes + +Phase 3 delivers the shortcode resolution **infrastructure** (Lua engine, block/inline +dispatch, extension loading). The following built-in shortcodes from TS Quarto are not +yet implemented in q2 and will need separate work items. This list is exhaustive as of +Quarto 1.x. + +### Already implemented in q2 (Rust) + +| Shortcode | Status | Notes | +|-----------|--------|-------| +| `meta` | Done | `MetaShortcodeHandler` in `shortcode_resolve.rs` | + +### Core Lua handlers (from `shortcodes-handlers.lua`) + +These are registered as built-in handlers in TS Quarto. In q2, they could be +implemented as either Rust `ShortcodeHandler` impls or bundled Lua scripts. + +| Shortcode | Args | Returns | Notes | +|-----------|------|---------|-------| +| `var` | key (dot notation) | Inlines | Reads from `_variables.yml` file. Requires variables file support. | +| `env` | name, optional default | Inlines (`pandoc.Str`) | Reads `os.getenv()`. Straightforward Lua or Rust impl. | +| `pagebreak` | none | RawBlock (format-specific) | Returns `\newpage{}` (LaTeX), `
` (HTML), OpenXML for DOCX, etc. Context-insensitive (always block). | +| `brand` | subcommand (color/logo), name, optional mode | Inlines or Blocks | Reads `_brand.yml`. `brand color primary` → color string. `brand logo main` → Image element(s) with light/dark classes. Requires brand config support. | +| `contents` | optional ID | RawInline (JSON) | Generates TOC marker. Internal use for callout TOC integration. | + +### Built-in extension shortcodes (from `resources/extensions/quarto/`) + +These ship as built-in Quarto extensions. In q2, they would be bundled Lua scripts +loaded via the extension mechanism. + +| Shortcode | Args | Returns | Notes | +|-----------|------|---------|-------| +| `lipsum` | range (e.g., `1-3`), count, or none (default `1-5`); `random=true` | Blocks (list of Para) | Lorem ipsum placeholder text. Good first candidate — no dependencies, exercises block-level return. | +| `kbd` | default shortcut; `mac=`, `win=`, `linux=` named args; `mode=plain` | RawInline (HTML ``) or Inlines | Keyboard shortcut display with OS-specific variants. | +| `video` | src URL; `width=`, `height=`, `title=`, `start=`, `aspect-ratio=` | RawBlock (HTML) or Link | Embeds YouTube, Vimeo, Brightcove, or local video. Format-aware. | +| `placeholder` | width, height; `format=svg\|png` | Image (data URI) | Generates colored placeholder images. Returns differently for text vs visual contexts. | +| `version` | none | string | Returns Quarto version. Trivial impl. | + +### Pre-engine shortcodes (TypeScript in TS Quarto) + +These are handled before engine execution in TS Quarto and operate on raw markdown +text, not the Pandoc AST. They may need a different mechanism in q2. + +| Shortcode | Args | Returns | Notes | +|-----------|------|---------|-------| +| `include` | filename | Raw markdown content | Inserts content from another file. Handled pre-parse in TS Quarto. May need a pre-parse stage or tree-sitter integration in q2. | +| `embed` | notebook filename; `echo=`, `outputs=` | Raw markdown content | Embeds Jupyter notebook cells. Post-engine in TS Quarto. Requires notebook support. | + +### Recommended implementation order + +1. **`env`** — trivial, no dependencies, good smoke test for the Lua engine +2. **`lipsum`** — no dependencies, exercises block-level returns, useful for testing +3. **`pagebreak`** — format-specific RawBlock, exercises format dispatch +4. **`version`** — trivial string return +5. **`kbd`** — exercises named args, OS detection, HTML generation +6. **`var`** — requires `_variables.yml` support (separate feature) +7. **`video`** — complex HTML generation, URL parsing +8. **`placeholder`** — image generation, data URIs +9. **`brand`** — requires `_brand.yml` support (separate feature) +10. **`contents`** — internal, depends on callout/TOC infrastructure +11. **`include`** / **`embed`** — pre-engine, needs separate architecture diff --git a/claude-notes/plans/2026-03-20-quarto-lua-api.md b/claude-notes/plans/2026-03-20-quarto-lua-api.md new file mode 100644 index 00000000..37828c8d --- /dev/null +++ b/claude-notes/plans/2026-03-20-quarto-lua-api.md @@ -0,0 +1,595 @@ +# Quarto Lua API: `quarto.*` Namespace Implementation + +**Created**: 2026-03-20 +**Status**: IN PROGRESS +**Branch**: `feature/shortcode-extensions` +**Triggered by**: lipsum extension fails because `quarto.utils` is nil + +## READ THIS FIRST (Zero-Knowledge Bootstrap) + +This plan is for the **pampa** crate (`crates/pampa/`), which is the core Quarto +Markdown engine. It has a Lua subsystem in `crates/pampa/src/lua/` that provides +Pandoc-compatible Lua APIs to extension authors. + +**What is this about?** Quarto extensions (written in Lua) expect a `quarto.*` global +namespace with utility functions. We have the `pandoc.*` namespace fully implemented, +but the `quarto.*` namespace is nearly empty. This causes real extensions to crash. + +**Reproducing the bug:** +```bash +cd ~/docs/lipsum +cargo run --manifest-path /Users/gordon/src/q2/Cargo.toml --bin q2 -- render index.qmd +``` +Error: `attempt to index a nil value (field 'utils')` — because `quarto.utils` doesn't exist. + +**Key files in `crates/pampa/src/lua/`:** + +| File | Purpose | +|------|---------| +| `constructors.rs` | `register_pandoc_namespace()` — sets up `pandoc.*` globals, then calls `register_quarto_namespace()` at the end (line 262) | +| `diagnostics.rs` | `register_quarto_namespace()` — creates the `quarto` global table with ONLY `warn`, `error`, `_diagnostics` | +| `shortcode.rs` | `LuaShortcodeEngine` — loads/dispatches shortcode Lua scripts. `register_shortcode_api()` adds `quarto.shortcode.*` | +| `filter.rs` | `apply_lua_filter()` — runs Lua filters on ASTs. Sets up Lua state, calls `register_pandoc_namespace()` | +| `json.rs` | `register_pandoc_json()` — implements `pandoc.json.decode/encode/null` | +| `utils.rs` | `register_pandoc_utils()` — implements `pandoc.utils.stringify/type/sha1/etc` | +| `path.rs` | `register_pandoc_path()` — implements `pandoc.path.directory/join/normalize/etc` | +| `system.rs` | `register_pandoc_system()` — implements `pandoc.system.get_working_directory/etc` | +| `types.rs` | `LuaInline`, `LuaBlock` wrapper types for passing AST nodes to/from Lua | +| `mod.rs` | Module declarations and public re-exports | + +**Lua state initialization sequence** (both shortcode and filter engines): + +``` +1. Lua::new() // Create Lua VM +2. register_pandoc_namespace(lua, ...) // Sets up pandoc.* (constructors.rs) + ├── pandoc.Str, pandoc.Para, ... // AST constructors + ├── pandoc.utils.* // stringify, type, sha1, etc. + ├── pandoc.json.* // decode, encode, null + ├── pandoc.path.* // directory, join, normalize, etc. + ├── pandoc.system.* // get_working_directory, etc. + └── register_quarto_namespace(lua) // Creates quarto table (diagnostics.rs) + ├── quarto.warn(msg, elem?) + ├── quarto.error(msg, elem?) + └── quarto._diagnostics // internal storage +3. lua.globals().set("FORMAT", ...) // Set target format global +4. [shortcode only] register_shortcode_api(lua) // Adds quarto.shortcode.* +``` + +**What's missing** (needed by real extensions): +- `quarto.utils` sub-table (especially `resolve_path`) +- `quarto.json` sub-table (should alias `pandoc.json`) +- `quarto.log` sub-table (logging to stderr) + +## Problem Statement + +The `quarto` Lua global currently only exposes `warn`, `error`, `_diagnostics`, and +`shortcode.*`. Real-world Quarto extensions (including built-in ones like lipsum, kbd, +video, placeholder) expect a much richer `quarto.*` API surface. The lipsum extension +specifically needs `quarto.utils.resolve_path()`, `quarto.json.decode()`, and +`quarto.log.error()`. + +**The test extension** (`~/docs/lipsum/`) contains: +- `index.qmd` — uses `{{< lipsum 3 >}}` +- `_extensions/lipsum/_extension.yml` — declares `contributes.shortcodes: [lipsum.lua]` +- `_extensions/lipsum/lipsum.lua` — the shortcode handler (100 lines) +- `_extensions/lipsum/lipsum.json` — lorem ipsum paragraph data + +The lipsum handler does three things that fail: +1. Line 20: `quarto.utils.resolve_path("lipsum.json")` — resolve path relative to script +2. Line 23: `quarto.json.decode(fileContents)` — parse JSON +3. Line 26: `quarto.log.error("Unable to read lipsum data file.")` — log error + +It also uses `io.open()` (native Lua file I/O), `math.randomseed(os.time())`, and +`pandoc.utils.stringify()` / `pandoc.Para()` — these all already work. + +## TS Quarto Reference + +In TypeScript Quarto (`~/src/quarto-cli/`), the `quarto` namespace is defined in: +`src/resources/pandoc/datadir/init.lua` (lines 812-1047) + +Key implementation details from TS Quarto: + +### `quarto.utils.resolve_path(path)` (line 970) +```lua +-- resolve_path = resolvePathExt (line 970) +local function resolvePathExt(path) -- line 322 + if isRelativeRef(path) then + return resolvePath(pandoc.path.join({scriptDir(), pandoc.path.normalize(path)})) + else + return path + end +end +``` +Where `scriptDir()` returns the directory of the currently-executing script file. +`scriptDir()` (line 180) reads from a `scriptFile` stack that tracks which Lua file +is being loaded. `resolvePath` (line 313) joins with the working directory if relative. + +### `quarto.json` (line 993) +```lua +local json = require '_json' -- line 151 +-- ... +json = json, -- line 993 +``` +It's a separate JSON library (`_json.lua`), but functionally equivalent to `pandoc.json`. + +### `quarto.log` (line 995) +```lua +local logging = require 'logging' -- line 153 +-- ... +log = logging, -- line 995 +``` +Defined in `logging.lua`. Pure Lua module that writes to `io.stderr`. Key functions: +- `output(...)` — write stringified args to stderr +- `error(...)` — prefix `(E)`, only if loglevel >= -1 +- `warning(...)` — prefix `(W)`, only if loglevel >= 0 +- `info(...)` — prefix `(I)`, only if loglevel >= 1 +- `debug(...)` — prefix `(D)`, only if loglevel >= 2 +- `trace(...)` — prefix `(T)`, only if loglevel >= 3 +- `setloglevel(level)` — set level, return old +- `dump(value, maxlen)` — pretty-print value + +Default loglevel is 0 (warnings and errors shown). + +Stringify logic: uses `pandoc.utils.type()` to detect types, dumps tables recursively, +calls `tostring()` on primitives. + +### Built-in extension API usage + +| Extension | `quarto.*` APIs used | +|-----------|---------------------| +| **lipsum** | `utils.resolve_path`, `json.decode`, `log.error` | +| **kbd** | `shortcode.read_arg`, `shortcode.error_output`, `doc.is_format`, `doc.add_html_dependency`, `log.warning` | +| **video** | `doc.add_html_dependency`, `doc.include_text`, `doc.is_format`, `doc.has_bootstrap`, `utils.as_inlines` | +| **placeholder** | `utils.resolve_path`, `base64.encode`, `format.is_typst_output`, `shortcode.error_output` | +| **version** | `quarto.version` | + +## What Already Exists Under `pandoc.*` (Can Be Aliased) + +| API | Rust source | Notes | +|-----|-------------|-------| +| `pandoc.json.decode(str)` | `json.rs:48` | Full JSON decode impl | +| `pandoc.json.encode(value)` | `json.rs:28` | Full JSON encode impl | +| `pandoc.json.null` | `json.rs:22` | LightUserData sentinel | +| `pandoc.utils.stringify(elem)` | `utils.rs:24` | Full impl for all AST types | +| `pandoc.utils.type(value)` | `utils.rs:267` | Returns Pandoc-aware type name | +| `pandoc.path.directory(path)` | `path.rs` | Returns parent directory | +| `pandoc.path.join(parts)` | `path.rs` | Joins path components | +| `pandoc.path.normalize(path)` | `path.rs` | Normalizes path separators | +| `pandoc.path.is_absolute(path)` | `path.rs` | Check if absolute | +| `pandoc.path.is_relative(path)` | `path.rs` | Check if relative | +| `pandoc.system.get_working_directory()` | `system.rs` | Returns cwd | + +## Design + +### Architecture: `register_quarto_api()` + +Create a new module `pampa/src/lua/quarto_api.rs` that extends the existing `quarto` +table (already created by `diagnostics.rs`) with additional sub-namespaces. + +```rust +/// Extends the `quarto` global (already created by register_quarto_namespace) +/// with additional API sub-namespaces: quarto.json, quarto.log, quarto.utils. +/// +/// Must be called AFTER register_pandoc_namespace() (which creates both +/// `pandoc` and `quarto` globals). +pub fn register_quarto_api(lua: &Lua) -> Result<()> +``` + +**No options struct needed.** The `_quarto_script_dir` global is set separately +by the caller (shortcode engine or filter engine) before script evaluation. + +**Call sequence after this change:** + +``` +1. Lua::new() +2. register_pandoc_namespace(lua, ...) // pandoc.* + base quarto table +3. register_quarto_api(lua) // NEW: extends quarto with json/log/utils +4. lua.globals().set("FORMAT", ...) +5. [shortcode only] register_shortcode_api(lua) // quarto.shortcode.* +``` + +The function retrieves the existing `quarto` table and adds sub-tables: +```rust +let quarto: Table = lua.globals().get("quarto")?; +// Add quarto.json (alias pandoc.json) +// Add quarto.log (new impl) +// Add quarto.utils (new impl) +``` + +### `quarto.utils.resolve_path(path)` — Key Design Decision + +**How it works in TS Quarto:** Before each script is loaded, a `scriptFile` stack is +pushed with the script path. `scriptDir()` returns the directory of the top of the stack. +`resolve_path` joins the relative path with `scriptDir()`. + +**Our approach:** Use a Lua global `_quarto_script_dir` (string). The shortcode engine +sets this before each `chunk.eval()` in `load_script()`, and restores it after. + +```rust +// In LuaShortcodeEngine::load_script(), BEFORE chunk.eval(): +let script_dir = script_path.parent().unwrap_or(Path::new("")); +self.lua.globals().set( + "_quarto_script_dir", + script_dir.to_string_lossy().to_string() +)?; +// ... chunk.eval() ... +// Restore is optional since the next load_script will overwrite it +``` + +The Lua function reads this global: +```rust +// Registered as a Rust closure in register_quarto_api(): +lua.create_function(|lua, path: String| { + // Check if relative + let p = Path::new(&path); + if p.is_absolute() { + return Ok(path); + } + // Get script dir from Lua global + let script_dir: String = lua.globals() + .get::("_quarto_script_dir") + .unwrap_or_default(); + if script_dir.is_empty() { + return Ok(path); // No script dir set, return as-is + } + let resolved = PathBuf::from(&script_dir).join(&path); + // Normalize (collapse .. and .) + Ok(normalize_path(&resolved)) +})?; +``` + +**Why a global and not a closure capture?** The `quarto.utils.resolve_path` function is +registered once during `register_quarto_api()`, but `_quarto_script_dir` changes for +each script loaded. A Lua global is the simplest way to communicate this. + +**IMPORTANT:** lipsum's `readLipsum()` is called lazily (from the handler, not at +script load time), so the script dir must remain valid after `load_script()` returns. +Since the shortcode engine loads all scripts then calls handlers, the script dir will +be set to the LAST loaded script's directory. This is fine if there's only one script +per extension (the common case). For multiple scripts, the last one wins — matching +TS Quarto behavior since it also uses a stack that pops after load. + +Actually, looking more carefully: the lipsum handler calls `readLipsum()` which calls +`resolve_path("lipsum.json")`. This happens during `engine.call()`, not during +`load_script()`. At that point, `_quarto_script_dir` is set to whatever the last +`load_script()` set it to. If we load scripts per-extension, this is the extension's +script directory — correct for lipsum. + +But if we load scripts from multiple extensions, the script dir for earlier extensions +would be wrong when their handlers are called later. **Fix:** Store the script directory +per handler name when loading, and set `_quarto_script_dir` before each `call()`. + +```rust +// In LuaShortcodeEngine: +struct LuaShortcodeEngine { + lua: Lua, + handlers: HashMap, + handler_script_dirs: HashMap, // NEW: name -> script dir + runtime: Arc, +} + +// In load_script(): record script dir for each handler registered +// In call(): set _quarto_script_dir before calling the handler +``` + +### `quarto.json` — Simple alias + +Just alias the already-registered `pandoc.json` table: +```rust +let pandoc: Table = lua.globals().get("pandoc")?; +let pandoc_json: Table = pandoc.get("json")?; +quarto.set("json", pandoc_json)?; +``` + +This gives `quarto.json.decode`, `quarto.json.encode`, and `quarto.json.null` for free. + +### `quarto.log` — Rust-backed stderr logging + +Implement as Rust closures that write to stderr via `eprintln!`. This works on both +native and WASM (where `io.stderr` doesn't exist). + +The stringify logic for each argument: +- `string` / `number` / `boolean` → `tostring()` +- `table` / `userdata` → use `pandoc.utils.stringify()` (already registered) +- `nil` → `"nil"` + +Log level is stored as a Lua number in `quarto.log.loglevel` (default 0). + +```rust +fn register_quarto_log(lua: &Lua, quarto: &Table) -> Result<()> { + let log = lua.create_table()?; + log.set("loglevel", 0)?; // Default: warnings + errors + + // quarto.log.output(...) — always writes + log.set("output", lua.create_function(|lua, args: MultiValue| { + let text = stringify_log_args(lua, &args)?; + eprintln!("{}", text); + Ok(()) + })?)?; + + // quarto.log.error(...) — writes if loglevel >= -1 + // quarto.log.warning(...) — writes if loglevel >= 0 + // etc. + + quarto.set("log", log)?; + Ok(()) +} +``` + +### Test patterns + +Existing shortcode tests use this pattern (from `shortcode.rs` tests): +```rust +use tempfile::TempDir; + +fn make_runtime() -> Arc { + Arc::new(NativeRuntime::new()) +} + +fn write_script(dir: &Path, name: &str, content: &str) -> PathBuf { + let path = dir.join(name); + std::fs::write(&path, content).unwrap(); + path +} + +#[test] +fn test_example() { + let tmp = TempDir::new().unwrap(); + let script = write_script(tmp.path(), "hello.lua", r#" + return { hello = function(args, kwargs, meta, raw_args, context) + return pandoc.Str("hello-world") + end } + "#); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine.call("hello", &make_empty_args(), ShortcodeCallContext::Inline).unwrap(); + // ... assert on result ... +} +``` + +For `quarto_api.rs` unit tests, use the same pattern as `utils.rs` tests: +```rust +fn create_test_lua() -> Lua { + let lua = Lua::new(); + let runtime = Arc::new(NativeRuntime::new()); + register_pandoc_namespace(&lua, runtime, create_shared_mediabag()).unwrap(); + register_quarto_api(&lua).unwrap(); // NEW + lua +} + +#[test] +fn test_quarto_json_decode() { + let lua = create_test_lua(); + let result: String = lua.load(r#"return quarto.json.decode('{"a":1}').a"#).eval().unwrap(); + // Note: pandoc.json.decode returns Lua tables, so field access works +} +``` + +## Work Items + +### Phase 1: Core infrastructure (`quarto_api.rs`) + +- [x] **1.1** Create `pampa/src/lua/quarto_api.rs` with `pub fn register_quarto_api(lua: &Lua) -> Result<()>`. + This function retrieves the existing `quarto` global table and extends it. + +- [x] **1.2** Implement `quarto.json` as alias of `pandoc.json`. + Three lines: get `pandoc.json` table, set as `quarto.json`. + +- [x] **1.3** Implement `quarto.log` namespace with these functions: + - `quarto.log.output(...)` — stringify args, write to stderr via `eprintln!` + - `quarto.log.error(...)` — `(E)` prefix, only if `quarto.log.loglevel >= -1` + - `quarto.log.warning(...)` — `(W)` prefix, only if `quarto.log.loglevel >= 0` + - `quarto.log.info(...)` — `(I)` prefix, only if `quarto.log.loglevel >= 1` + - `quarto.log.debug(...)` — `(D)` prefix, only if `quarto.log.loglevel >= 2` + - `quarto.log.trace(...)` — `(T)` prefix, only if `quarto.log.loglevel >= 3` + - `quarto.log.setloglevel(level)` — set `loglevel`, return old value + - `quarto.log.loglevel` — numeric field, default 0 + - Use `eprintln!` in Rust for output (works on native and WASM) + - Stringify: for each arg, use Lua `tostring()` for primitives (string, number, boolean, + nil). For tables, do recursive key=value dumping. For userdata, try `tostring()` which + will invoke the `__tostring` metamethod if present (our LuaInline/LuaBlock have this). + Do NOT call `pandoc.utils.stringify()` — it expects specific AST types and would error + on arbitrary tables. This matches TS Quarto's `logging.lua` approach. + +- [x] **1.4** Implement `quarto.utils` sub-namespace: + - `quarto.utils.resolve_path(path)` — if relative, join with `_quarto_script_dir` global; + if absolute, return as-is. Uses `std::path::Path` in Rust. Needs a `normalize_path()` + helper that collapses `.` and `..` without touching the filesystem (since + `std::path::Path::canonicalize()` requires the path to exist on disk). + - `quarto.utils.type(value)` — alias `pandoc.utils.type` + - ~~`quarto.utils.resolve_path_relative_to_document(path)`~~ — DEFERRED to Tier 2. + Needs the document path plumbed through, which neither shortcode nor filter engine + currently provides. Not needed by lipsum or other Tier 1 extensions. + +- [x] **1.5** Add `mod quarto_api;` to `pampa/src/lua/mod.rs`. Add `pub use`. + +- [x] **1.6** Tests in `quarto_api.rs`: + - `test_quarto_json_decode` — `quarto.json.decode('{"a":1}')` returns table + - `test_quarto_json_encode` — `quarto.json.encode({a=1})` returns string + - `test_quarto_log_error_runs` — `quarto.log.error("test")` doesn't panic + - `test_quarto_log_respects_level` — `quarto.log.info("test")` is silent at level 0 + - `test_quarto_log_setloglevel` — returns old level, changes behavior + - `test_quarto_utils_resolve_path_absolute` — `/abs/path` returned as-is + - `test_quarto_utils_resolve_path_relative` — `foo.json` joined with script dir + - `test_quarto_utils_resolve_path_no_script_dir` — returns relative path as-is + - `test_quarto_utils_type` — `quarto.utils.type(pandoc.Str("x"))` returns `"Str"` + +### Phase 2: Wire into shortcode engine + +- [x] **2.1** In `LuaShortcodeEngine::new()` (shortcode.rs), call `register_quarto_api(&lua)` + after `register_pandoc_namespace()` (line 82) and before `register_shortcode_api()` (line 89). + Import: `use super::quarto_api::register_quarto_api;` + +- [x] **2.2** Add `handler_script_dirs: HashMap` field to `LuaShortcodeEngine`. + +- [x] **2.3** In `load_script()`: before `chunk.eval()` (line 138), set `_quarto_script_dir` + to `script_path.parent()`. After registering handlers (both conventions), store the + script dir in `handler_script_dirs` for each handler name. + +- [x] **2.4** In `call()`: before calling the handler function, set `_quarto_script_dir` + to the value from `handler_script_dirs[name]`. + +- [x] **2.5** Tests: + - `test_shortcode_resolve_path` — write `data.json` next to script, handler calls + `quarto.utils.resolve_path("data.json")`, verify it returns the correct absolute path + - `test_shortcode_quarto_json` — handler calls `quarto.json.decode('{"x":1}')`, + returns the value + - `test_shortcode_quarto_log` — handler calls `quarto.log.warning("test")`, no crash + +### Phase 3: Wire into filter engine + +- [x] **3.1** In `apply_lua_filter()` (filter.rs), call `register_quarto_api(&lua)` after + `register_pandoc_namespace()` (line 136) and before `lua.globals().set("FORMAT", ...)`. + +- [x] **3.2** Set `_quarto_script_dir` to `filter_path.parent()` before loading the + filter script (line 174). This is a one-shot setting since filters run one at a time. + +- [x] **3.3** Tests: + - `test_filter_quarto_json_available` — filter script can call `quarto.json.decode` + - `test_filter_quarto_log_available` — filter script can call `quarto.log.warning` + +### Phase 4: End-to-end validation + +- [~] **4.1** Test lipsum extension manually (PARTIAL — quarto.* API works, but lipsum + fails due to a separate issue: `pandoc.Para(string)` doesn't auto-convert strings to + inlines. This is a `pandoc.Para` constructor compatibility issue, not a quarto API issue): + ```bash + cd ~/docs/lipsum + cargo run --manifest-path /Users/gordon/src/q2/Cargo.toml --bin q2 -- render index.qmd + ``` + Should produce HTML with 3 paragraphs of lorem ipsum text. + +- [ ] **4.2** (DEFERRED) Create smoke test: `crates/quarto/tests/smoke-all/extensions/shortcode-resolve-path/` + with a shortcode extension that uses `quarto.utils.resolve_path()` to load a JSON + data file and `quarto.json.decode()` to parse it. + +- [x] **4.3** `cargo nextest run --workspace` — all 6975 tests pass. + +- [x] **4.4** `cargo build --workspace` — clean build. + +## Files to Create/Modify + +| File | Action | Description | +|------|--------|-------------| +| `crates/pampa/src/lua/quarto_api.rs` | **Create** | `register_quarto_api()` — quarto.json, quarto.log, quarto.utils | +| `crates/pampa/src/lua/mod.rs` | Modify | Add `mod quarto_api;` and `pub use` | +| `crates/pampa/src/lua/shortcode.rs` | Modify | Call `register_quarto_api()`, add `handler_script_dirs`, set `_quarto_script_dir` per handler | +| `crates/pampa/src/lua/filter.rs` | Modify | Call `register_quarto_api()`, set `_quarto_script_dir` | + +Files NOT modified: `constructors.rs`, `diagnostics.rs` — existing behavior preserved. + +## mlua Patterns You'll Need + +The Lua binding crate is `mlua`. Key patterns used throughout the codebase: + +```rust +use mlua::{Function, Lua, MultiValue, Result, Table, Value}; + +// Get existing global table +let quarto: Table = lua.globals().get("quarto")?; + +// Create sub-table +let utils = lua.create_table()?; +quarto.set("utils", utils)?; + +// Create Rust-backed Lua function +utils.set("resolve_path", lua.create_function(|lua, path: String| { + // ... Rust logic ... + Ok(result_string) +})?)?; + +// Variadic args (for quarto.log.*) +log.set("output", lua.create_function(|lua, args: MultiValue| { + for arg in args.iter() { + match arg { + Value::String(s) => { /* ... */ } + Value::Number(n) => { /* ... */ } + _ => { /* ... */ } + } + } + Ok(()) +})?)?; + +// Read Lua global from inside a closure +let script_dir: String = lua.globals() + .get::("_quarto_script_dir") + .unwrap_or_default(); + +// Read table field from inside a closure +let log_table: Table = lua.globals().get::("quarto")?.get::
("log")?; +let level: i32 = log_table.get("loglevel")?; +``` + +**WASM considerations:** On `#[cfg(target_arch = "wasm32")]`, the Lua state is created +with restricted StdLib (no `io`, `os`, `debug`, `package`). Our `eprintln!`-based +logging avoids the `io.stderr` dependency. `std::path::Path` works fine in WASM. + +## Risks + +- **`_quarto_script_dir` per-handler in shortcode engine**: We track script dir per + handler name and restore it before each `call()`. This handles the case where multiple + extensions are loaded. Without this, the last `load_script()` would set the dir for + ALL handlers. + +- **Log output in tests**: `eprintln!` writes to stderr. Tests can't easily capture + this, so just test that the functions don't error. Don't assert on output content. + +- **WASM log output**: `eprintln!` in WASM goes to the browser console (via + `console.error`). This is acceptable. + +## Future Work (Tier 2+) + +After this plan, the most impactful next APIs to implement would be: +1. `quarto.doc.is_format(name)` — format detection with aliases (needed by kbd, video) +2. `quarto.doc.add_html_dependency(dep)` — HTML dependency injection +3. `quarto.base64.encode/decode` — needed by placeholder +4. `quarto.utils.as_inlines/as_blocks` — AST conversion helpers +5. `quarto.utils.string_to_inlines/string_to_blocks` — markdown parsing from Lua + +## TS Quarto Full API Surface (Reference) + +For completeness, here is every `quarto.*` API in TS Quarto. Only Tier 1 (above) is +in scope for this plan. + +``` +quarto.warn(msg, elem?) ✅ Already implemented +quarto.error(msg, elem?) ✅ Already implemented +quarto.shortcode.read_arg ✅ Already implemented +quarto.shortcode.error_output ✅ Already implemented +quarto.json.decode 🔨 This plan (alias pandoc.json) +quarto.json.encode 🔨 This plan (alias pandoc.json) +quarto.log.output/error/warning/info/debug/trace 🔨 This plan +quarto.log.setloglevel/loglevel 🔨 This plan +quarto.utils.resolve_path 🔨 This plan +quarto.utils.type 🔨 This plan (alias pandoc.utils.type) +quarto.utils.resolve_path_relative_to_document ⏳ Tier 2 (deferred — needs doc path plumbing) +quarto.doc.is_format ⏳ Tier 2 +quarto.doc.add_html_dependency ⏳ Tier 2 +quarto.doc.include_text ⏳ Tier 2 +quarto.doc.has_bootstrap ⏳ Tier 2 +quarto.doc.input_file ⏳ Tier 3 +quarto.doc.output_file ⏳ Tier 3 +quarto.doc.use_latex_package ⏳ Tier 3 +quarto.doc.cite_method ⏳ Tier 3 +quarto.doc.pdf_engine ⏳ Tier 3 +quarto.base64.encode/decode ⏳ Tier 2 +quarto.utils.as_inlines ⏳ Tier 2 +quarto.utils.as_blocks ⏳ Tier 2 +quarto.utils.string_to_inlines ⏳ Tier 2 +quarto.utils.string_to_blocks ⏳ Tier 2 +quarto.utils.dump ⏳ Tier 3 +quarto.utils.match ⏳ Tier 3 +quarto.utils.is_empty_node ⏳ Tier 3 +quarto.project.* ⏳ Tier 3 +quarto.metadata.get ⏳ Tier 3 +quarto.variables.get ⏳ Tier 3 +quarto.config.version ⏳ Tier 3 +quarto.brand.* ⏳ Tier 3 +quarto.format.* ⏳ Tier 3 +quarto.version ⏳ Tier 3 +quarto.paths.* ⏳ Tier 3 +quarto.Callout/Tabset/etc ⏳ Tier 3 (custom AST constructors) +``` diff --git a/crates/pampa/src/lua/filter.rs b/crates/pampa/src/lua/filter.rs index daa0ba36..511b1120 100644 --- a/crates/pampa/src/lua/filter.rs +++ b/crates/pampa/src/lua/filter.rs @@ -19,6 +19,7 @@ use crate::pandoc::{Block, Inline, Pandoc}; use super::constructors::register_pandoc_namespace; use super::mediabag::create_shared_mediabag; +use super::quarto_api::register_quarto_api; use super::readwrite::{create_reader_options_table, create_writer_options_table}; use super::runtime::SystemRuntime; use super::types::{LuaBlock, LuaInline, blocks_to_lua_table, inlines_to_lua_table}; @@ -135,6 +136,18 @@ pub fn apply_lua_filter( // Register pandoc namespace with constructors (also registers quarto namespace) register_pandoc_namespace(&lua, runtime, mediabag)?; + // Register quarto.json, quarto.log, quarto.utils + register_quarto_api(&lua)?; + + // Set script dir for quarto.utils.resolve_path + let script_dir = filter_path + .parent() + .unwrap_or(Path::new("")) + .to_string_lossy() + .to_string(); + lua.globals() + .set("_quarto_script_dir", script_dir.as_str())?; + // Set global variables // FORMAT - the target output format (html, latex, etc.) lua.globals().set("FORMAT", target_format)?; @@ -309,6 +322,42 @@ fn get_filter_table(lua: &Lua) -> Result
{ Ok(filter_table) } +/// Extract an Inline from a Lua UserData value. +pub(crate) fn extract_lua_inline(ud: &mlua::AnyUserData) -> Result { + Ok(ud.borrow::()?.0.clone()) +} + +/// Extract a Block from a Lua UserData value. +pub(crate) fn extract_lua_block(ud: &mlua::AnyUserData) -> Result { + Ok(ud.borrow::()?.0.clone()) +} + +/// Extract a Vec from a Lua table of UserData values. +pub(crate) fn extract_lua_inlines_from_table(table: &mlua::Table) -> Result> { + let len = table.raw_len(); + let mut inlines = Vec::new(); + for i in 1..=len { + let value: Value = table.get(i)?; + if let Value::UserData(ud) = value { + inlines.push(extract_lua_inline(&ud)?); + } + } + Ok(inlines) +} + +/// Extract a Vec from a Lua table of UserData values. +pub(crate) fn extract_lua_blocks_from_table(table: &mlua::Table) -> Result> { + let len = table.raw_len(); + let mut blocks = Vec::new(); + for i in 1..=len { + let value: Value = table.get(i)?; + if let Value::UserData(ud) = value { + blocks.push(extract_lua_block(&ud)?); + } + } + Ok(blocks) +} + /// Handle return value from an inline filter fn handle_inline_return(ret: Value, original: &Inline) -> Result> { match ret { diff --git a/crates/pampa/src/lua/filter_tests.rs b/crates/pampa/src/lua/filter_tests.rs index c9cc5aac..30835863 100644 --- a/crates/pampa/src/lua/filter_tests.rs +++ b/crates/pampa/src/lua/filter_tests.rs @@ -5574,3 +5574,85 @@ end _ => panic!("Expected two Paragraphs"), } } + +// ========================================================================= +// quarto.* API availability in filters +// ========================================================================= + +#[test] +fn test_filter_quarto_json_available() { + let dir = TempDir::new().unwrap(); + let filter_path = dir.path().join("json_test.lua"); + fs::write( + &filter_path, + r#" +function Str(elem) + local t = quarto.json.decode('{"val":"decoded"}') + return pandoc.Str(t.val) +end +"#, + ) + .unwrap(); + + let pandoc = Pandoc { + meta: Default::default(), + blocks: vec![Block::Paragraph(crate::pandoc::Paragraph { + content: vec![Inline::Str(crate::pandoc::Str { + text: "input".to_string(), + source_info: quarto_source_map::SourceInfo::default(), + })], + source_info: quarto_source_map::SourceInfo::default(), + })], + }; + let context = ASTContext::new(); + let (result, _, _) = + apply_lua_filter(&pandoc, &context, &filter_path, "html", native_runtime()) + .expect("Filter should succeed"); + + match &result.blocks[0] { + Block::Paragraph(p) => match &p.content[0] { + Inline::Str(s) => assert_eq!(s.text, "decoded"), + other => panic!("Expected Str, got {:?}", other), + }, + other => panic!("Expected Paragraph, got {:?}", other), + } +} + +#[test] +fn test_filter_quarto_log_available() { + let dir = TempDir::new().unwrap(); + let filter_path = dir.path().join("log_test.lua"); + fs::write( + &filter_path, + r#" +function Str(elem) + quarto.log.warning("test warning from filter") + return pandoc.Str("logged") +end +"#, + ) + .unwrap(); + + let pandoc = Pandoc { + meta: Default::default(), + blocks: vec![Block::Paragraph(crate::pandoc::Paragraph { + content: vec![Inline::Str(crate::pandoc::Str { + text: "input".to_string(), + source_info: quarto_source_map::SourceInfo::default(), + })], + source_info: quarto_source_map::SourceInfo::default(), + })], + }; + let context = ASTContext::new(); + let (result, _, _) = + apply_lua_filter(&pandoc, &context, &filter_path, "html", native_runtime()) + .expect("Filter should succeed"); + + match &result.blocks[0] { + Block::Paragraph(p) => match &p.content[0] { + Inline::Str(s) => assert_eq!(s.text, "logged"), + other => panic!("Expected Str, got {:?}", other), + }, + other => panic!("Expected Paragraph, got {:?}", other), + } +} diff --git a/crates/pampa/src/lua/mod.rs b/crates/pampa/src/lua/mod.rs index 68712671..0e5bf25f 100644 --- a/crates/pampa/src/lua/mod.rs +++ b/crates/pampa/src/lua/mod.rs @@ -15,8 +15,10 @@ mod json; mod list; pub mod mediabag; mod path; +mod quarto_api; mod readwrite; pub mod runtime; +pub mod shortcode; mod system; mod text; mod types; @@ -28,3 +30,7 @@ pub use filter::{LuaFilterError, apply_lua_filters}; pub use runtime::NativeRuntime; #[allow(unused_imports)] pub use runtime::{RuntimeError, RuntimeResult, SystemRuntime}; +#[allow(unused_imports)] +pub use shortcode::{ + LuaShortcodeEngine, LuaShortcodeError, LuaShortcodeResult, ShortcodeArgs, ShortcodeCallContext, +}; diff --git a/crates/pampa/src/lua/quarto_api.rs b/crates/pampa/src/lua/quarto_api.rs new file mode 100644 index 00000000..67dbb03b --- /dev/null +++ b/crates/pampa/src/lua/quarto_api.rs @@ -0,0 +1,437 @@ +/* + * lua/quarto_api.rs + * Copyright (c) 2025 Posit, PBC + * + * Extends the `quarto` global table with additional API sub-namespaces: + * quarto.json, quarto.log, quarto.utils. + * + * Must be called AFTER register_pandoc_namespace() (which creates both + * `pandoc` and `quarto` globals). + */ + +use mlua::{Lua, MultiValue, Result, Table, Value}; +use std::path::{Component, Path, PathBuf}; + +/// Extends the `quarto` global (already created by register_quarto_namespace) +/// with additional API sub-namespaces: quarto.json, quarto.log, quarto.utils. +pub fn register_quarto_api(lua: &Lua) -> Result<()> { + let quarto: Table = lua.globals().get("quarto")?; + + register_quarto_json(lua, &quarto)?; + register_quarto_log(lua, &quarto)?; + register_quarto_utils(lua, &quarto)?; + + Ok(()) +} + +/// `quarto.json` — alias of `pandoc.json` +fn register_quarto_json(lua: &Lua, quarto: &Table) -> Result<()> { + let pandoc: Table = lua.globals().get("pandoc")?; + let pandoc_json: Table = pandoc.get("json")?; + quarto.set("json", pandoc_json)?; + Ok(()) +} + +/// `quarto.log` — Rust-backed stderr logging +fn register_quarto_log(lua: &Lua, quarto: &Table) -> Result<()> { + let log = lua.create_table()?; + log.set("loglevel", 0)?; // Default: warnings + errors + + // quarto.log.output(...) — always writes + log.set( + "output", + lua.create_function(|lua, args: MultiValue| { + let text = stringify_log_args(lua, &args)?; + eprintln!("{}", text); + Ok(()) + })?, + )?; + + // Helper macro-like approach: create level-gated log functions + // quarto.log.error(...) — writes if loglevel >= -1 + log.set("error", create_log_fn(lua, "(E)", -1)?)?; + // quarto.log.warning(...) — writes if loglevel >= 0 + log.set("warning", create_log_fn(lua, "(W)", 0)?)?; + // quarto.log.info(...) — writes if loglevel >= 1 + log.set("info", create_log_fn(lua, "(I)", 1)?)?; + // quarto.log.debug(...) — writes if loglevel >= 2 + log.set("debug", create_log_fn(lua, "(D)", 2)?)?; + // quarto.log.trace(...) — writes if loglevel >= 3 + log.set("trace", create_log_fn(lua, "(T)", 3)?)?; + + // quarto.log.setloglevel(level) — set level, return old + log.set( + "setloglevel", + lua.create_function(|lua, level: i32| { + let log_table: Table = lua.globals().get::
("quarto")?.get::
("log")?; + let old: i32 = log_table.get("loglevel")?; + log_table.set("loglevel", level)?; + Ok(old) + })?, + )?; + + quarto.set("log", log)?; + Ok(()) +} + +/// Create a level-gated log function with a prefix. +fn create_log_fn(lua: &Lua, prefix: &'static str, min_level: i32) -> Result { + lua.create_function(move |lua, args: MultiValue| { + let log_table: Table = lua.globals().get::
("quarto")?.get::
("log")?; + let level: i32 = log_table.get("loglevel")?; + if level >= min_level { + let text = stringify_log_args(lua, &args)?; + eprintln!("{} {}", prefix, text); + } + Ok(()) + }) +} + +/// Stringify log arguments. Each arg is converted to a string and joined with tabs. +fn stringify_log_args(lua: &Lua, args: &MultiValue) -> Result { + let mut parts = Vec::new(); + for arg in args.iter() { + parts.push(stringify_log_value(lua, arg, 0)?); + } + Ok(parts.join("\t")) +} + +/// Stringify a single value for logging. +fn stringify_log_value(lua: &Lua, value: &Value, depth: usize) -> Result { + if depth > 5 { + return Ok("
".to_string()); + } + match value { + Value::Nil => Ok("nil".to_string()), + Value::Boolean(b) => Ok(b.to_string()), + Value::Integer(n) => Ok(n.to_string()), + Value::Number(n) => Ok(n.to_string()), + Value::String(s) => Ok(s.to_str()?.to_string()), + Value::Table(t) => stringify_table(lua, t, depth), + Value::UserData(_) => { + // Use Lua tostring() which invokes __tostring metamethod + let tostring: mlua::Function = lua.globals().get("tostring")?; + let result: String = tostring.call(value.clone())?; + Ok(result) + } + Value::Function(_) => Ok("".to_string()), + _ => Ok(format!("<{}>", value.type_name())), + } +} + +/// Stringify a Lua table recursively for logging. +fn stringify_table(lua: &Lua, table: &Table, depth: usize) -> Result { + let mut parts = Vec::new(); + let len = table.raw_len(); + + // Check if it's a sequence (array-like) + if len > 0 { + for i in 1..=len { + let val: Value = table.get(i)?; + parts.push(stringify_log_value(lua, &val, depth + 1)?); + } + return Ok(format!("{{{}}}", parts.join(", "))); + } + + // Otherwise treat as key-value + for pair in table.pairs::() { + let (k, v) = pair?; + let ks = stringify_log_value(lua, &k, depth + 1)?; + let vs = stringify_log_value(lua, &v, depth + 1)?; + parts.push(format!("{}={}", ks, vs)); + } + Ok(format!("{{{}}}", parts.join(", "))) +} + +/// `quarto.utils` — utility functions +fn register_quarto_utils(lua: &Lua, quarto: &Table) -> Result<()> { + let utils = lua.create_table()?; + + // quarto.utils.resolve_path(path) — resolve relative to script dir + utils.set( + "resolve_path", + lua.create_function(|lua, path: String| { + let p = Path::new(&path); + if p.is_absolute() { + return Ok(path); + } + let script_dir: String = lua + .globals() + .get::("_quarto_script_dir") + .unwrap_or_default(); + if script_dir.is_empty() { + return Ok(path); + } + let resolved = PathBuf::from(&script_dir).join(&path); + Ok(normalize_path(&resolved)) + })?, + )?; + + // quarto.utils.type(value) — alias pandoc.utils.type + let pandoc: Table = lua.globals().get("pandoc")?; + let pandoc_utils: Table = pandoc.get("utils")?; + let type_fn: mlua::Function = pandoc_utils.get("type")?; + utils.set("type", type_fn)?; + + quarto.set("utils", utils)?; + Ok(()) +} + +/// Normalize a path by collapsing `.` and `..` without touching the filesystem. +fn normalize_path(path: &Path) -> String { + let mut components = Vec::new(); + for component in path.components() { + match component { + Component::CurDir => {} // skip `.` + Component::ParentDir => { + // Pop last component if it's a normal component + if let Some(last) = components.last() { + if !matches!(last, Component::RootDir | Component::Prefix(_)) { + components.pop(); + continue; + } + } + components.push(component); + } + _ => components.push(component), + } + } + let result: PathBuf = components.iter().collect(); + result.to_string_lossy().to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::lua::constructors::register_pandoc_namespace; + use crate::lua::mediabag::create_shared_mediabag; + use crate::lua::runtime::NativeRuntime; + use std::sync::Arc; + + fn create_test_lua() -> Lua { + let lua = Lua::new(); + let runtime = Arc::new(NativeRuntime::new()); + register_pandoc_namespace(&lua, runtime, create_shared_mediabag()).unwrap(); + register_quarto_api(&lua).unwrap(); + lua + } + + // ========================================================================= + // quarto.json tests + // ========================================================================= + + #[test] + fn test_quarto_json_decode() { + let lua = create_test_lua(); + let result: i32 = lua + .load(r#"return quarto.json.decode('{"a":1}').a"#) + .eval() + .unwrap(); + assert_eq!(result, 1); + } + + #[test] + fn test_quarto_json_encode() { + let lua = create_test_lua(); + let result: String = lua + .load(r#"return quarto.json.encode({a = 1})"#) + .eval() + .unwrap(); + assert!(result.contains("\"a\"")); + assert!(result.contains("1")); + } + + // ========================================================================= + // quarto.log tests + // ========================================================================= + + #[test] + fn test_quarto_log_error_runs() { + let lua = create_test_lua(); + lua.load(r#"quarto.log.error("test error")"#) + .exec() + .unwrap(); + } + + #[test] + fn test_quarto_log_warning_runs() { + let lua = create_test_lua(); + lua.load(r#"quarto.log.warning("test warning")"#) + .exec() + .unwrap(); + } + + #[test] + fn test_quarto_log_output_runs() { + let lua = create_test_lua(); + lua.load(r#"quarto.log.output("test output")"#) + .exec() + .unwrap(); + } + + #[test] + fn test_quarto_log_respects_level() { + let lua = create_test_lua(); + // Default level is 0; info requires >= 1, so this should be silent (no error) + lua.load(r#"quarto.log.info("should be silent")"#) + .exec() + .unwrap(); + // debug requires >= 2 + lua.load(r#"quarto.log.debug("should be silent")"#) + .exec() + .unwrap(); + // trace requires >= 3 + lua.load(r#"quarto.log.trace("should be silent")"#) + .exec() + .unwrap(); + } + + #[test] + fn test_quarto_log_setloglevel() { + let lua = create_test_lua(); + let old: i32 = lua + .load(r#"return quarto.log.setloglevel(2)"#) + .eval() + .unwrap(); + assert_eq!(old, 0); + + let current: i32 = lua.load(r#"return quarto.log.loglevel"#).eval().unwrap(); + assert_eq!(current, 2); + + // Now info should work (level 2 >= 1) + lua.load(r#"quarto.log.info("visible at level 2")"#) + .exec() + .unwrap(); + } + + #[test] + fn test_quarto_log_multiple_args() { + let lua = create_test_lua(); + lua.load(r#"quarto.log.output("hello", "world", 42)"#) + .exec() + .unwrap(); + } + + #[test] + fn test_quarto_log_table_arg() { + let lua = create_test_lua(); + lua.load(r#"quarto.log.output({1, 2, 3})"#).exec().unwrap(); + } + + // ========================================================================= + // quarto.utils.resolve_path tests + // ========================================================================= + + #[test] + fn test_quarto_utils_resolve_path_absolute() { + let lua = create_test_lua(); + let result: String = lua + .load(r#"return quarto.utils.resolve_path("/abs/path/file.json")"#) + .eval() + .unwrap(); + assert_eq!(result, "/abs/path/file.json"); + } + + #[test] + fn test_quarto_utils_resolve_path_relative() { + let lua = create_test_lua(); + lua.globals() + .set("_quarto_script_dir", "/some/extension/dir") + .unwrap(); + let result: String = lua + .load(r#"return quarto.utils.resolve_path("data.json")"#) + .eval() + .unwrap(); + assert_eq!(result, "/some/extension/dir/data.json"); + } + + #[test] + fn test_quarto_utils_resolve_path_relative_with_subdir() { + let lua = create_test_lua(); + lua.globals().set("_quarto_script_dir", "/ext/dir").unwrap(); + let result: String = lua + .load(r#"return quarto.utils.resolve_path("sub/data.json")"#) + .eval() + .unwrap(); + assert_eq!(result, "/ext/dir/sub/data.json"); + } + + #[test] + fn test_quarto_utils_resolve_path_no_script_dir() { + let lua = create_test_lua(); + // No _quarto_script_dir set, should return as-is + let result: String = lua + .load(r#"return quarto.utils.resolve_path("data.json")"#) + .eval() + .unwrap(); + assert_eq!(result, "data.json"); + } + + #[test] + fn test_quarto_utils_resolve_path_with_dotdot() { + let lua = create_test_lua(); + lua.globals() + .set("_quarto_script_dir", "/some/extension/dir") + .unwrap(); + let result: String = lua + .load(r#"return quarto.utils.resolve_path("../shared/data.json")"#) + .eval() + .unwrap(); + assert_eq!(result, "/some/extension/shared/data.json"); + } + + // ========================================================================= + // quarto.utils.type tests + // ========================================================================= + + #[test] + fn test_quarto_utils_type_str() { + let lua = create_test_lua(); + let result: String = lua + .load(r#"return quarto.utils.type(pandoc.Str("x"))"#) + .eval() + .unwrap(); + assert_eq!(result, "Str"); + } + + #[test] + fn test_quarto_utils_type_table() { + let lua = create_test_lua(); + let result: String = lua.load(r#"return quarto.utils.type({})"#).eval().unwrap(); + assert_eq!(result, "table"); + } + + // ========================================================================= + // normalize_path unit tests + // ========================================================================= + + #[test] + fn test_normalize_path_simple() { + assert_eq!(normalize_path(Path::new("/a/b/c")), "/a/b/c"); + } + + #[test] + fn test_normalize_path_with_dots() { + assert_eq!(normalize_path(Path::new("/a/./b/c")), "/a/b/c"); + } + + #[test] + fn test_normalize_path_with_dotdot() { + assert_eq!(normalize_path(Path::new("/a/b/../c")), "/a/c"); + } + + #[test] + fn test_normalize_path_with_dotdot_at_root() { + assert_eq!(normalize_path(Path::new("/a/../b")), "/b"); + } + + #[test] + fn test_normalize_path_relative() { + assert_eq!(normalize_path(Path::new("a/b/c")), "a/b/c"); + } + + #[test] + fn test_normalize_path_relative_with_dotdot() { + assert_eq!(normalize_path(Path::new("a/b/../c")), "a/c"); + } +} diff --git a/crates/pampa/src/lua/shortcode.rs b/crates/pampa/src/lua/shortcode.rs new file mode 100644 index 00000000..4b3c90db --- /dev/null +++ b/crates/pampa/src/lua/shortcode.rs @@ -0,0 +1,1052 @@ +/* + * lua/shortcode.rs + * Copyright (c) 2025 Posit, PBC + * + * Lua shortcode engine for loading and dispatching shortcode handlers. + * + * This module is distinct from the filter engine — no AST traversal, + * just function dispatch. A single LuaShortcodeEngine is created per + * render and reused for all shortcode invocations in the document. + */ + +use mlua::{Function, Lua, Result, Table, Value}; +use std::collections::HashMap; +use std::path::Path; +use std::sync::Arc; + +use crate::pandoc::{Block, Inline}; + +use super::constructors::register_pandoc_namespace; +use super::filter::{extract_lua_block, extract_lua_inline}; +use super::mediabag::create_shared_mediabag; +use super::quarto_api::register_quarto_api; +use super::runtime::SystemRuntime; +use super::types::{LuaBlock, LuaInline}; + +/// Context in which a shortcode is being resolved. +/// Named `ShortcodeCallContext` to avoid conflict with the existing +/// `ShortcodeContext` struct in `shortcode_resolve.rs`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ShortcodeCallContext { + Block, + Inline, + Text, +} + +impl ShortcodeCallContext { + fn as_str(&self) -> &'static str { + match self { + ShortcodeCallContext::Block => "block", + ShortcodeCallContext::Inline => "inline", + ShortcodeCallContext::Text => "text", + } + } +} + +/// Result of calling a Lua shortcode handler. +#[derive(Debug)] +pub enum LuaShortcodeResult { + Inlines(Vec), + Blocks(Vec), + Text(String), + Error(String), +} + +/// Lua shortcode engine for loading and dispatching handlers. +/// +/// This is `!Send + !Sync` because it holds a `Lua` state. It must only +/// be created as a local variable inside `ShortcodeResolveTransform::transform()`. +pub struct LuaShortcodeEngine { + lua: Lua, + handlers: HashMap, + handler_script_dirs: HashMap, + runtime: Arc, +} + +impl LuaShortcodeEngine { + /// Create engine, set up Lua state with pandoc/quarto globals. + pub fn new( + target_format: &str, + runtime: Arc, + ) -> std::result::Result { + #[cfg(target_arch = "wasm32")] + let lua = { + use mlua::StdLib; + let libs = + StdLib::COROUTINE | StdLib::TABLE | StdLib::STRING | StdLib::UTF8 | StdLib::MATH; + Lua::new_with(libs, mlua::LuaOptions::default()).map_err(LuaShortcodeError::LuaError)? + }; + #[cfg(not(target_arch = "wasm32"))] + let lua = Lua::new(); + + let mediabag = create_shared_mediabag(); + register_pandoc_namespace(&lua, runtime.clone(), mediabag) + .map_err(LuaShortcodeError::LuaError)?; + + // Register quarto.json, quarto.log, quarto.utils + register_quarto_api(&lua).map_err(LuaShortcodeError::LuaError)?; + + lua.globals() + .set("FORMAT", target_format) + .map_err(LuaShortcodeError::LuaError)?; + + // Register quarto.shortcode sub-namespace + register_shortcode_api(&lua).map_err(LuaShortcodeError::LuaError)?; + + Ok(Self { + lua, + handlers: HashMap::new(), + handler_script_dirs: HashMap::new(), + runtime, + }) + } + + /// Load a shortcode Lua script. Registers all handlers it defines. + /// Supports both return-table and environment-function conventions. + pub fn load_script( + &mut self, + script_path: &Path, + ) -> std::result::Result<(), LuaShortcodeError> { + let script_bytes = self.runtime.file_read(script_path).map_err(|e| { + LuaShortcodeError::FileReadError( + script_path.to_owned(), + std::io::Error::new(std::io::ErrorKind::Other, e.to_string()), + ) + })?; + let script_source = String::from_utf8(script_bytes).map_err(|e| { + LuaShortcodeError::FileReadError( + script_path.to_owned(), + std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()), + ) + })?; + + // Set script dir for quarto.utils.resolve_path + let script_dir = script_path + .parent() + .unwrap_or(Path::new("")) + .to_string_lossy() + .to_string(); + self.lua + .globals() + .set("_quarto_script_dir", script_dir.as_str()) + .map_err(LuaShortcodeError::LuaError)?; + + // Execute script in a sandboxed environment that inherits globals + let env = self + .lua + .create_table() + .map_err(LuaShortcodeError::LuaError)?; + let env_mt = self + .lua + .create_table() + .map_err(LuaShortcodeError::LuaError)?; + env_mt + .set("__index", self.lua.globals()) + .map_err(LuaShortcodeError::LuaError)?; + env.set_metatable(Some(env_mt)) + .map_err(LuaShortcodeError::LuaError)?; + + let chunk = self + .lua + .load(&script_source) + .set_name(script_path.to_string_lossy()) + .set_environment(env.clone()); + + let ret: Value = chunk.eval().map_err(LuaShortcodeError::LuaError)?; + + // Convention 1: script returns a table of handlers + if let Value::Table(ref table) = ret { + for pair in table.pairs::() { + let (name, value) = pair.map_err(LuaShortcodeError::LuaError)?; + if is_callable(&value) { + let key = self + .lua + .create_registry_value(value) + .map_err(LuaShortcodeError::LuaError)?; + self.handler_script_dirs + .insert(name.clone(), script_dir.clone()); + self.handlers.insert(name, key); + } + } + return Ok(()); + } + + // Convention 2: scan environment for callable values + for pair in env.pairs::() { + let (name, value) = pair.map_err(LuaShortcodeError::LuaError)?; + if is_callable(&value) { + // Skip globals that were inherited (only register new ones) + if let Ok(global_val) = self.lua.globals().get::(name.as_str()) { + if same_lua_value(&value, &global_val) { + continue; + } + } + let key = self + .lua + .create_registry_value(value) + .map_err(LuaShortcodeError::LuaError)?; + self.handler_script_dirs + .insert(name.clone(), script_dir.clone()); + self.handlers.insert(name, key); + } + } + + Ok(()) + } + + /// Call a named shortcode handler. + /// Returns None if no handler is registered for the name. + pub fn call( + &self, + name: &str, + args: &ShortcodeArgs, + context: ShortcodeCallContext, + ) -> Option { + let reg_key = self.handlers.get(name)?; + let func: Function = self.lua.registry_value(reg_key).ok()?; + + // Set script dir for this handler's extension + if let Some(dir) = self.handler_script_dirs.get(name) { + let _ = self.lua.globals().set("_quarto_script_dir", dir.as_str()); + } + + Some(self.call_handler(name, func, args, context)) + } + + /// Check if a handler is registered for the given name. + pub fn has_handler(&self, name: &str) -> bool { + self.handlers.contains_key(name) + } + + fn call_handler( + &self, + name: &str, + func: Function, + args: &ShortcodeArgs, + context: ShortcodeCallContext, + ) -> LuaShortcodeResult { + match self.build_and_call(func, args, context) { + Ok(result) => result, + Err(e) => { + LuaShortcodeResult::Error(format!("Shortcode '{}' handler error: {}", name, e)) + } + } + } + + fn build_and_call( + &self, + func: Function, + args: &ShortcodeArgs, + context: ShortcodeCallContext, + ) -> std::result::Result { + // Build args table: pandoc.List of {value = string} or {name = key, value = string} + let lua_args = self.build_args_table(args)?; + + // Build kwargs table + let lua_kwargs = self.build_kwargs_table(args)?; + + // Build meta proxy table + let lua_meta = self.build_meta_table(args)?; + + // Build raw_args list + let lua_raw_args = self.build_raw_args(args)?; + + // Context string + let ctx_str = context.as_str(); + + let ret: Value = func.call((lua_args, lua_kwargs, lua_meta, lua_raw_args, ctx_str))?; + + Ok(convert_return_value(&self.lua, ret)) + } + + fn build_args_table(&self, args: &ShortcodeArgs) -> Result { + let table = self.lua.create_table()?; + let mut idx = 1; + for arg in &args.positional { + let entry = self.lua.create_table()?; + entry.set("value", arg.as_str())?; + table.set(idx, entry)?; + idx += 1; + } + for (key, val) in &args.keyword { + let entry = self.lua.create_table()?; + entry.set("name", key.as_str())?; + entry.set("value", val.as_str())?; + table.set(idx, entry)?; + idx += 1; + } + Ok(Value::Table(table)) + } + + fn build_kwargs_table(&self, args: &ShortcodeArgs) -> Result { + let table = self.lua.create_table()?; + for (key, val) in &args.keyword { + table.set(key.as_str(), val.as_str())?; + } + Ok(Value::Table(table)) + } + + fn build_meta_table(&self, args: &ShortcodeArgs) -> Result { + let table = self.lua.create_table()?; + for (key, val) in &args.metadata { + table.set(key.as_str(), val.as_str())?; + } + Ok(Value::Table(table)) + } + + fn build_raw_args(&self, args: &ShortcodeArgs) -> Result { + let table = self.lua.create_table()?; + for (i, arg) in args.positional.iter().enumerate() { + table.set(i + 1, arg.as_str())?; + } + Ok(Value::Table(table)) + } +} + +/// Arguments prepared for a shortcode handler call. +/// This is a simplified representation that the caller builds from +/// the Shortcode struct and document metadata. +pub struct ShortcodeArgs { + pub positional: Vec, + pub keyword: Vec<(String, String)>, + pub metadata: Vec<(String, String)>, +} + +/// Errors from the shortcode engine. +#[derive(Debug)] +pub enum LuaShortcodeError { + FileReadError(std::path::PathBuf, std::io::Error), + LuaError(mlua::Error), +} + +impl std::fmt::Display for LuaShortcodeError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + LuaShortcodeError::FileReadError(path, err) => { + write!( + f, + "Failed to read shortcode script '{}': {}", + path.display(), + err + ) + } + LuaShortcodeError::LuaError(err) => { + write!(f, "Lua shortcode error: {}", err) + } + } + } +} + +impl std::error::Error for LuaShortcodeError {} + +/// Register `quarto.shortcode` sub-namespace with helper functions. +fn register_shortcode_api(lua: &Lua) -> Result<()> { + let quarto: Table = lua.globals().get("quarto")?; + + let shortcode_ns = lua.create_table()?; + + // quarto.shortcode.read_arg(args, n) + shortcode_ns.set( + "read_arg", + lua.create_function(|_lua, (args, n): (Table, usize)| -> Result { + // 1-based index + let entry: Value = args.get(n)?; + match entry { + Value::Table(t) => t.get("value"), + Value::Nil => Ok(Value::Nil), + other => Ok(other), + } + })?, + )?; + + // quarto.shortcode.error_output(name, message, context) + shortcode_ns.set( + "error_output", + lua.create_function( + |lua, (name, message, context): (String, String, String)| -> Result { + let err_text = format!("[Shortcode Error ({}): {}]", name, message); + let make_strong_inline = |text: String| -> Inline { + Inline::Strong(crate::pandoc::Strong { + content: vec![Inline::Str(crate::pandoc::Str { + text, + source_info: Default::default(), + })], + source_info: Default::default(), + }) + }; + match context.as_str() { + "block" => { + let para = lua.create_userdata(LuaBlock(Block::Paragraph( + crate::pandoc::Paragraph { + content: vec![make_strong_inline(err_text)], + source_info: Default::default(), + }, + )))?; + Ok(Value::UserData(para)) + } + "inline" => { + let strong = + lua.create_userdata(LuaInline(make_strong_inline(err_text)))?; + Ok(Value::UserData(strong)) + } + _ => Ok(Value::String(lua.create_string(&err_text)?)), + } + }, + )?, + )?; + + quarto.set("shortcode", shortcode_ns)?; + Ok(()) +} + +/// Convert a Lua return value to a LuaShortcodeResult. +fn convert_return_value(_lua: &Lua, ret: Value) -> LuaShortcodeResult { + match ret { + Value::Nil => LuaShortcodeResult::Error("Shortcode returned nil".to_string()), + Value::String(s) => { + LuaShortcodeResult::Text(s.to_str().map(|s| s.to_string()).unwrap_or_default()) + } + Value::UserData(ud) => { + if let Ok(inline) = extract_lua_inline(&ud) { + LuaShortcodeResult::Inlines(vec![inline]) + } else if let Ok(block) = extract_lua_block(&ud) { + LuaShortcodeResult::Blocks(vec![block]) + } else { + LuaShortcodeResult::Error( + "Shortcode returned unsupported userdata type".to_string(), + ) + } + } + Value::Table(table) => classify_table_result(&table), + _ => LuaShortcodeResult::Error("Shortcode returned unsupported type".to_string()), + } +} + +/// Classify a table return as Inlines or Blocks. +fn classify_table_result(table: &mlua::Table) -> LuaShortcodeResult { + let len = table.raw_len(); + if len == 0 { + return LuaShortcodeResult::Inlines(vec![]); + } + + let mut inlines = Vec::new(); + let mut blocks = Vec::new(); + let mut has_blocks = false; + + for i in 1..=len { + let value: std::result::Result = table.get(i); + match value { + Ok(Value::UserData(ud)) => { + if let Ok(inline) = extract_lua_inline(&ud) { + inlines.push(inline); + } else if let Ok(block) = extract_lua_block(&ud) { + has_blocks = true; + blocks.push(block); + } + } + _ => {} + } + } + + if has_blocks { + LuaShortcodeResult::Blocks(blocks) + } else if !inlines.is_empty() { + LuaShortcodeResult::Inlines(inlines) + } else { + LuaShortcodeResult::Error( + "Shortcode returned table with no recognizable elements".to_string(), + ) + } +} + +fn is_callable(value: &Value) -> bool { + matches!(value, Value::Function(_)) +} + +/// Check if two Lua values are the same reference. +fn same_lua_value(a: &Value, b: &Value) -> bool { + match (a, b) { + (Value::Function(fa), Value::Function(fb)) => { + // Compare by pointer identity + fa == fb + } + _ => false, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::lua::runtime::NativeRuntime; + use std::path::PathBuf; + use tempfile::TempDir; + + fn make_runtime() -> Arc { + Arc::new(NativeRuntime::new()) + } + + fn write_script(dir: &std::path::Path, name: &str, content: &str) -> PathBuf { + let path = dir.join(name); + std::fs::write(&path, content).unwrap(); + path + } + + fn make_empty_args() -> ShortcodeArgs { + ShortcodeArgs { + positional: vec![], + keyword: vec![], + metadata: vec![], + } + } + + #[test] + fn test_load_script_return_table() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "hello.lua", + r#" +return { + hello = function(args, kwargs, meta, raw_args, context) + return pandoc.Str("hello-world") + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + assert!(engine.has_handler("hello")); + } + + #[test] + fn test_load_script_env_function() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "hello.lua", + r#" +function hello(args, kwargs, meta, raw_args, context) + return pandoc.Str("hello-world") +end +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + assert!(engine.has_handler("hello")); + } + + #[test] + fn test_call_returns_inlines() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "hello.lua", + r#" +return { + hello = function(args, kwargs, meta, raw_args, context) + return pandoc.Inlines{pandoc.Str("hi")} + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine + .call("hello", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Inlines(inlines) => { + assert_eq!(inlines.len(), 1); + match &inlines[0] { + Inline::Str(s) => assert_eq!(s.text, "hi"), + other => panic!("Expected Str, got {:?}", other), + } + } + other => panic!("Expected Inlines, got {:?}", other), + } + } + + #[test] + fn test_call_returns_blocks() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "brk.lua", + r#" +return { + brk = function(args, kwargs, meta, raw_args, context) + return pandoc.RawBlock("html", "
") + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine + .call("brk", &make_empty_args(), ShortcodeCallContext::Block) + .unwrap(); + match result { + LuaShortcodeResult::Blocks(blocks) => { + assert_eq!(blocks.len(), 1); + match &blocks[0] { + Block::RawBlock(rb) => { + assert_eq!(rb.format, "html"); + assert_eq!(rb.text, "
"); + } + other => panic!("Expected RawBlock, got {:?}", other), + } + } + other => panic!("Expected Blocks, got {:?}", other), + } + } + + #[test] + fn test_call_returns_string() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "ver.lua", + r#" +return { + ver = function(args, kwargs, meta, raw_args, context) + return "1.0.0" + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine + .call("ver", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "1.0.0"), + other => panic!("Expected Text, got {:?}", other), + } + } + + #[test] + fn test_call_returns_nil() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "bad.lua", + r#" +return { + bad = function(args, kwargs, meta, raw_args, context) + return nil + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine + .call("bad", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Error(msg) => { + assert!(msg.contains("nil"), "Expected nil error, got: {}", msg); + } + other => panic!("Expected Error, got {:?}", other), + } + } + + #[test] + fn test_call_unknown_handler() { + let runtime = make_runtime(); + let engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + + let result = engine.call( + "nonexistent", + &make_empty_args(), + ShortcodeCallContext::Inline, + ); + assert!(result.is_none()); + } + + #[test] + fn test_handler_receives_args() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "echo.lua", + r#" +return { + echo = function(args, kwargs, meta, raw_args, context) + return args[1].value + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let args = ShortcodeArgs { + positional: vec!["world".to_string()], + keyword: vec![], + metadata: vec![], + }; + let result = engine + .call("echo", &args, ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "world"), + other => panic!("Expected Text, got {:?}", other), + } + } + + #[test] + fn test_handler_receives_kwargs() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "kwarg.lua", + r#" +return { + kwarg = function(args, kwargs, meta, raw_args, context) + return kwargs.greeting + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let args = ShortcodeArgs { + positional: vec![], + keyword: vec![("greeting".to_string(), "howdy".to_string())], + metadata: vec![], + }; + let result = engine + .call("kwarg", &args, ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "howdy"), + other => panic!("Expected Text, got {:?}", other), + } + } + + #[test] + fn test_handler_receives_meta() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "meta.lua", + r#" +return { + meta_reader = function(args, kwargs, meta, raw_args, context) + return meta.title + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let args = ShortcodeArgs { + positional: vec![], + keyword: vec![], + metadata: vec![("title".to_string(), "My Doc".to_string())], + }; + let result = engine + .call("meta_reader", &args, ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "My Doc"), + other => panic!("Expected Text, got {:?}", other), + } + } + + #[test] + fn test_handler_receives_context() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "ctx.lua", + r#" +return { + ctx = function(args, kwargs, meta, raw_args, context) + return context + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine + .call("ctx", &make_empty_args(), ShortcodeCallContext::Block) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "block"), + other => panic!("Expected Text('block'), got {:?}", other), + } + + let result = engine + .call("ctx", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "inline"), + other => panic!("Expected Text('inline'), got {:?}", other), + } + + let result = engine + .call("ctx", &make_empty_args(), ShortcodeCallContext::Text) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "text"), + other => panic!("Expected Text('text'), got {:?}", other), + } + } + + #[test] + fn test_later_script_overrides_earlier() { + let tmp = TempDir::new().unwrap(); + let script1 = write_script( + tmp.path(), + "first.lua", + r#" +return { + greeting = function(args, kwargs, meta, raw_args, context) + return "first" + end +} +"#, + ); + let script2 = write_script( + tmp.path(), + "second.lua", + r#" +return { + greeting = function(args, kwargs, meta, raw_args, context) + return "second" + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script1).unwrap(); + engine.load_script(&script2).unwrap(); + + let result = engine + .call("greeting", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "second"), + other => panic!("Expected Text('second'), got {:?}", other), + } + } + + #[test] + fn test_read_arg_helper() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "readarg.lua", + r#" +return { + readarg = function(args, kwargs, meta, raw_args, context) + local val = quarto.shortcode.read_arg(args, 1) + return val + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let args = ShortcodeArgs { + positional: vec!["test-value".to_string()], + keyword: vec![], + metadata: vec![], + }; + let result = engine + .call("readarg", &args, ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "test-value"), + other => panic!("Expected Text('test-value'), got {:?}", other), + } + } + + #[test] + fn test_shortcode_resolve_path() { + let tmp = TempDir::new().unwrap(); + // Write a data file next to the script + std::fs::write(tmp.path().join("data.json"), r#"{"key":"value"}"#).unwrap(); + + let script = write_script( + tmp.path(), + "resolver.lua", + r#" +return { + resolver = function(args, kwargs, meta, raw_args, context) + return quarto.utils.resolve_path("data.json") + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine + .call("resolver", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => { + let expected = tmp.path().join("data.json").to_string_lossy().to_string(); + assert_eq!(s, expected); + } + other => panic!("Expected Text with resolved path, got {:?}", other), + } + } + + #[test] + fn test_shortcode_quarto_json() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "jsontest.lua", + r#" +return { + jsontest = function(args, kwargs, meta, raw_args, context) + local t = quarto.json.decode('{"x":42}') + return tostring(t.x) + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine + .call("jsontest", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "42"), + other => panic!("Expected Text('42'), got {:?}", other), + } + } + + #[test] + fn test_shortcode_quarto_log() { + let tmp = TempDir::new().unwrap(); + let script = write_script( + tmp.path(), + "logtest.lua", + r#" +return { + logtest = function(args, kwargs, meta, raw_args, context) + quarto.log.warning("test warning from shortcode") + return "ok" + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script).unwrap(); + + let result = engine + .call("logtest", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result { + LuaShortcodeResult::Text(s) => assert_eq!(s, "ok"), + other => panic!("Expected Text('ok'), got {:?}", other), + } + } + + #[test] + fn test_shortcode_resolve_path_multi_extension() { + // Test that script dirs are tracked per handler, not globally + let tmp1 = TempDir::new().unwrap(); + let tmp2 = TempDir::new().unwrap(); + + let script1 = write_script( + tmp1.path(), + "ext1.lua", + r#" +return { + ext1 = function(args, kwargs, meta, raw_args, context) + return quarto.utils.resolve_path("data1.json") + end +} +"#, + ); + + let script2 = write_script( + tmp2.path(), + "ext2.lua", + r#" +return { + ext2 = function(args, kwargs, meta, raw_args, context) + return quarto.utils.resolve_path("data2.json") + end +} +"#, + ); + + let runtime = make_runtime(); + let mut engine = LuaShortcodeEngine::new("html", runtime).unwrap(); + engine.load_script(&script1).unwrap(); + engine.load_script(&script2).unwrap(); + + // ext1 should resolve relative to tmp1 + let result1 = engine + .call("ext1", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result1 { + LuaShortcodeResult::Text(s) => { + let expected = tmp1.path().join("data1.json").to_string_lossy().to_string(); + assert_eq!(s, expected); + } + other => panic!("Expected Text for ext1, got {:?}", other), + } + + // ext2 should resolve relative to tmp2 + let result2 = engine + .call("ext2", &make_empty_args(), ShortcodeCallContext::Inline) + .unwrap(); + match result2 { + LuaShortcodeResult::Text(s) => { + let expected = tmp2.path().join("data2.json").to_string_lossy().to_string(); + assert_eq!(s, expected); + } + other => panic!("Expected Text for ext2, got {:?}", other), + } + } +} diff --git a/crates/quarto-core/src/extension/read.rs b/crates/quarto-core/src/extension/read.rs index af0da5e5..6cc18f63 100644 --- a/crates/quarto-core/src/extension/read.rs +++ b/crates/quarto-core/src/extension/read.rs @@ -216,7 +216,7 @@ fn parse_formats( /// Keys in extension format config whose values are file paths relative to /// the extension directory. -const PATH_VALUED_KEYS: &[&str] = &["template", "template-partials"]; +const PATH_VALUED_KEYS: &[&str] = &["template", "template-partials", "shortcodes"]; /// Reserved filter names that should NOT be marked as Path. /// These are special identifiers, not file paths. @@ -886,4 +886,106 @@ contributes: Some(true) ); } + + #[test] + fn test_format_shortcode_paths_marked() { + let tmp = TempDir::new().unwrap(); + let ext_dir = tmp.path().join("_extensions/test"); + let file = write_extension( + &ext_dir, + r#" +title: Test +author: Author +contributes: + formats: + html: + shortcodes: + - handler.lua +"#, + ); + + let runtime = make_runtime(); + let ext = read_extension(&file, &runtime).unwrap(); + + let html_meta = ext.contributes.formats.get("html").unwrap(); + let shortcodes = html_meta.get("shortcodes").unwrap(); + let items = shortcodes.as_array().unwrap(); + assert_eq!(items.len(), 1); + assert!( + matches!(&items[0].value, ConfigValueKind::Path(s) if s == "handler.lua"), + "expected Path(\"handler.lua\"), got {:?}", + items[0].value + ); + } + + #[test] + fn test_format_shortcode_multiple_paths_marked() { + let tmp = TempDir::new().unwrap(); + let ext_dir = tmp.path().join("_extensions/test"); + let file = write_extension( + &ext_dir, + r#" +title: Test +author: Author +contributes: + formats: + html: + shortcodes: + - hello.lua + - goodbye.lua + - utils/helper.lua +"#, + ); + + let runtime = make_runtime(); + let ext = read_extension(&file, &runtime).unwrap(); + + let html_meta = ext.contributes.formats.get("html").unwrap(); + let shortcodes = html_meta.get("shortcodes").unwrap(); + let items = shortcodes.as_array().unwrap(); + assert_eq!(items.len(), 3); + for (i, expected) in ["hello.lua", "goodbye.lua", "utils/helper.lua"] + .iter() + .enumerate() + { + assert!( + matches!(&items[i].value, ConfigValueKind::Path(s) if s == *expected), + "expected Path(\"{}\"), got {:?}", + expected, + items[i].value + ); + } + } + + #[test] + fn test_shortcode_marking_doesnt_affect_other_keys() { + let tmp = TempDir::new().unwrap(); + let ext_dir = tmp.path().join("_extensions/test"); + let file = write_extension( + &ext_dir, + r#" +title: Test +author: Author +contributes: + formats: + html: + shortcodes: + - handler.lua + toc: true + theme: cosmo +"#, + ); + + let runtime = make_runtime(); + let ext = read_extension(&file, &runtime).unwrap(); + + let html_meta = ext.contributes.formats.get("html").unwrap(); + // shortcodes should be marked + let shortcodes = html_meta.get("shortcodes").unwrap(); + let items = shortcodes.as_array().unwrap(); + assert!(matches!(&items[0].value, ConfigValueKind::Path(_))); + // other keys should be unchanged + assert_eq!(html_meta.get("toc").unwrap().as_bool(), Some(true)); + assert_eq!(html_meta.get("theme").unwrap().as_str(), Some("cosmo")); + } } diff --git a/crates/quarto-core/src/pipeline.rs b/crates/quarto-core/src/pipeline.rs index b696474f..4145941e 100644 --- a/crates/quarto-core/src/pipeline.rs +++ b/crates/quarto-core/src/pipeline.rs @@ -297,9 +297,7 @@ pub async fn parse_qmd_to_ast( let stages: Vec> = vec![ Box::new(ParseDocumentStage::new()), Box::new(EngineExecutionStage::new()), - // elliot note: I want this function to give an un-processed AST - // Box::new(MetadataMergeStage::new()), - // Box::new(AstTransformsStage::new()), + Box::new(MetadataMergeStage::new()), ]; let (output, warnings) = run_pipeline(content, source_name, ctx, runtime, stages).await?; @@ -423,13 +421,23 @@ pub async fn render_qmd_to_html( /// ## Finalization Phase /// 10. `AppendixStructureTransform` - Consolidate appendix content into container /// 11. `ResourceCollectorTransform` - Collect image dependencies -pub fn build_transform_pipeline() -> TransformPipeline { +pub fn build_transform_pipeline( + shortcode_paths: Vec, + extensions: Vec, + runtime: std::sync::Arc, + target_format: String, +) -> TransformPipeline { let mut pipeline: TransformPipeline = TransformPipeline::new(); // === NORMALIZATION PHASE === pipeline.push(Box::new(CalloutTransform::new())); pipeline.push(Box::new(CalloutResolveTransform::new())); - pipeline.push(Box::new(ShortcodeResolveTransform::new())); + pipeline.push(Box::new(ShortcodeResolveTransform::with_lua_support( + shortcode_paths, + extensions, + runtime, + target_format, + ))); pipeline.push(Box::new(MetadataNormalizeTransform::new())); pipeline.push(Box::new(TitleBlockTransform::new())); pipeline.push(Box::new(SectionizeTransform::new())); diff --git a/crates/quarto-core/src/stage/stages/ast_transforms.rs b/crates/quarto-core/src/stage/stages/ast_transforms.rs index f28597cc..96d032dc 100644 --- a/crates/quarto-core/src/stage/stages/ast_transforms.rs +++ b/crates/quarto-core/src/stage/stages/ast_transforms.rs @@ -54,20 +54,30 @@ use crate::transform::TransformPipeline; /// /// Returns an error if any transform in the pipeline fails. pub struct AstTransformsStage { - pipeline: TransformPipeline, + /// Custom pipeline (set via `with_pipeline`). If `None`, the pipeline + /// is built just-in-time in `run()` using `StageContext` data. + custom_pipeline: Option, } impl AstTransformsStage { - /// Create an AstTransformsStage with the standard transform pipeline. + /// Create an AstTransformsStage that builds the pipeline at run time. + /// + /// The pipeline is constructed in `run()` using data from `StageContext` + /// (extensions, runtime, format) needed by the shortcode transform. pub fn new() -> Self { Self { - pipeline: build_transform_pipeline(), + custom_pipeline: None, } } /// Create an AstTransformsStage with a custom transform pipeline. + /// + /// The provided pipeline is used as-is, bypassing `StageContext`-based + /// construction. Used in tests that only need built-in handlers. pub fn with_pipeline(pipeline: TransformPipeline) -> Self { - Self { pipeline } + Self { + custom_pipeline: Some(pipeline), + } } } @@ -105,7 +115,29 @@ impl PipelineStage for AstTransformsStage { )); }; - let transform_count = self.pipeline.len(); + // Build the JIT pipeline if no custom pipeline was provided + let jit_pipeline; + let pipeline = if let Some(ref p) = self.custom_pipeline { + p + } else { + // Build pipeline JIT using StageContext data needed by shortcode transform + let document_dir = ctx + .document + .input + .parent() + .unwrap_or(std::path::Path::new(".")); + let shortcode_paths = + crate::transforms::extract_shortcode_paths(&doc.ast.meta, document_dir); + jit_pipeline = build_transform_pipeline( + shortcode_paths, + ctx.extensions.clone(), + ctx.runtime.clone(), + ctx.format.identifier.as_str().to_string(), + ); + &jit_pipeline + }; + + let transform_count = pipeline.len(); trace_event!( ctx, EventLevel::Debug, @@ -125,7 +157,7 @@ impl PipelineStage for AstTransformsStage { render_ctx.artifacts = std::mem::take(&mut ctx.artifacts); // Execute the transform pipeline - let result = self.pipeline.execute(&mut doc.ast, &mut render_ctx); + let result = pipeline.execute(&mut doc.ast, &mut render_ctx); // Transfer artifacts back to StageContext ctx.artifacts = render_ctx.artifacts; diff --git a/crates/quarto-core/src/stage/stages/metadata_merge.rs b/crates/quarto-core/src/stage/stages/metadata_merge.rs index 2643570f..3ef0e92a 100644 --- a/crates/quarto-core/src/stage/stages/metadata_merge.rs +++ b/crates/quarto-core/src/stage/stages/metadata_merge.rs @@ -1657,4 +1657,42 @@ mod tests { let layer = build_extension_metadata_layer(&[], "html"); assert!(layer.is_none()); } + + #[test] + fn test_extension_format_shortcode_paths_rebased_through_merge() { + // Extension contributes formats.html.shortcodes: [handler.lua] + // After mark_path_valued_keys (Phase 3.1), these become ConfigValueKind::Path. + // build_extension_metadata_layer should preserve the Path kind so that + // adjust_paths_to_document_dir() can rebase them during full merge. + let shortcodes_array = ConfigValue { + value: ConfigValueKind::Array(vec![ConfigValue { + value: ConfigValueKind::Path("handler.lua".to_string()), + source_info: SourceInfo::default(), + merge_op: Default::default(), + }]), + source_info: SourceInfo::default(), + merge_op: Default::default(), + }; + + let mut formats = std::collections::HashMap::new(); + formats.insert( + "html".to_string(), + config_map(vec![("shortcodes", shortcodes_array)]), + ); + let ext = make_extension("myext", formats); + + let layer = build_extension_metadata_layer(&[ext], "myext-html"); + assert!(layer.is_some()); + let (cv, ext_path) = layer.unwrap(); + assert_eq!(ext_path, PathBuf::from("/extensions/myext")); + + // Verify shortcodes key is present with Path-typed entries + let sc = cv.get("shortcodes").expect("shortcodes key should exist"); + let items = sc.as_array().expect("shortcodes should be an array"); + assert_eq!(items.len(), 1); + match &items[0].value { + ConfigValueKind::Path(p) => assert_eq!(p, "handler.lua"), + other => panic!("Expected Path, got {:?}", other), + } + } } diff --git a/crates/quarto-core/src/transforms/mod.rs b/crates/quarto-core/src/transforms/mod.rs index d9f59aa5..53936aaf 100644 --- a/crates/quarto-core/src/transforms/mod.rs +++ b/crates/quarto-core/src/transforms/mod.rs @@ -45,7 +45,7 @@ pub use footnotes::FootnotesTransform; pub use metadata_normalize::MetadataNormalizeTransform; pub use resource_collector::ResourceCollectorTransform; pub use sectionize::SectionizeTransform; -pub use shortcode_resolve::ShortcodeResolveTransform; +pub use shortcode_resolve::{ShortcodeResolveTransform, extract_shortcode_paths}; pub use title_block::TitleBlockTransform; pub use toc_generate::TocGenerateTransform; pub use toc_render::TocRenderTransform; diff --git a/crates/quarto-core/src/transforms/shortcode_resolve.rs b/crates/quarto-core/src/transforms/shortcode_resolve.rs index f6c7703b..7d4af216 100644 --- a/crates/quarto-core/src/transforms/shortcode_resolve.rs +++ b/crates/quarto-core/src/transforms/shortcode_resolve.rs @@ -43,11 +43,17 @@ use quarto_pandoc_types::shortcode::{Shortcode, ShortcodeArg}; use quarto_pandoc_types::table::Table; use quarto_source_map::SourceInfo; +use std::path::PathBuf; +use std::sync::Arc; + use quarto_analysis::AnalysisContext; use crate::Result; +use crate::extension::discover::find_extension; +use crate::extension::types::Extension; use crate::render::RenderContext; use crate::transform::AstTransform; +use quarto_system_runtime::SystemRuntime; /// Error information for shortcode resolution failures. pub struct ShortcodeError { @@ -61,6 +67,8 @@ pub struct ShortcodeError { pub enum ShortcodeResult { /// Resolved to inline content Inlines(Vec), + /// Resolved to block content (for block-context shortcodes) + Blocks(Vec), /// Error - renders visible content AND emits diagnostic Error(ShortcodeError), /// Shortcode should be preserved as literal text (e.g., escaped shortcodes) @@ -75,6 +83,15 @@ pub struct ShortcodeContext<'a> { pub source_info: &'a SourceInfo, } +/// Whether the shortcode is being resolved in block or inline context. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ResolutionContext { + /// Shortcode is the sole content of a Para/Plain — may return Blocks + Block, + /// Shortcode is inline among other content — must return Inlines + Inline, +} + /// Trait for shortcode handlers. /// /// Each built-in shortcode (meta, var, env, etc.) implements this trait. @@ -83,7 +100,12 @@ pub trait ShortcodeHandler: Send + Sync { fn name(&self) -> &str; /// Resolve the shortcode to content. - fn resolve(&self, shortcode: &Shortcode, ctx: &ShortcodeContext) -> ShortcodeResult; + fn resolve( + &self, + shortcode: &Shortcode, + ctx: &ShortcodeContext, + resolution_ctx: ResolutionContext, + ) -> ShortcodeResult; } /// Handler for the `meta` shortcode. @@ -99,7 +121,12 @@ impl ShortcodeHandler for MetaShortcodeHandler { "meta" } - fn resolve(&self, shortcode: &Shortcode, ctx: &ShortcodeContext) -> ShortcodeResult { + fn resolve( + &self, + shortcode: &Shortcode, + ctx: &ShortcodeContext, + _resolution_ctx: ResolutionContext, + ) -> ShortcodeResult { // Get the key from positional args let key = match shortcode.positional_args.first() { Some(ShortcodeArg::String(s)) => s.clone(), @@ -221,29 +248,110 @@ fn flatten_blocks_to_inlines(blocks: &[Block]) -> Vec { } /// Transform that resolves shortcodes in the AST. +/// +/// Supports both built-in Rust handlers and Lua shortcode scripts loaded from +/// extensions or user-specified paths. +/// +/// The `LuaShortcodeEngine` (which holds a `!Send + !Sync` Lua state) is created +/// inside `transform()` as a stack-local variable, never stored as a field. pub struct ShortcodeResolveTransform { handlers: Vec>, + /// Shortcode script paths from merged metadata (absolute) + lua_shortcode_paths: Vec, + /// Extensions for name-based shortcode lookup + extensions: Vec, + /// System runtime for reading Lua files + runtime: Option>, + /// Target format string (e.g., "html") for Lua FORMAT global + target_format: String, } impl ShortcodeResolveTransform { - /// Create a new shortcode resolve transform with default handlers. + /// Create a new shortcode resolve transform with default handlers only. + /// Used in tests that don't need Lua support. pub fn new() -> Self { Self { handlers: vec![Box::new(MetaShortcodeHandler)], + lua_shortcode_paths: Vec::new(), + extensions: Vec::new(), + runtime: None, + target_format: String::new(), + } + } + + /// Create a shortcode resolve transform with Lua support. + /// + /// The `LuaShortcodeEngine` is NOT created here (it's `!Send + !Sync`). + /// It is created on the stack inside `transform()`. + pub fn with_lua_support( + lua_shortcode_paths: Vec, + extensions: Vec, + runtime: Arc, + target_format: String, + ) -> Self { + Self { + handlers: vec![Box::new(MetaShortcodeHandler)], + lua_shortcode_paths, + extensions, + runtime: Some(runtime), + target_format, } } /// Resolve a shortcode using the appropriate handler. - fn resolve_shortcode(&self, shortcode: &Shortcode, ctx: &ShortcodeContext) -> ShortcodeResult { + /// + /// Priority: built-in Rust handlers > loaded Lua handlers > extension name lookup. + fn resolve_shortcode( + &self, + shortcode: &Shortcode, + ctx: &ShortcodeContext, + resolution_ctx: ResolutionContext, + lua_engine: &mut Option, + ) -> ShortcodeResult { // Handle escaped shortcodes - preserve as literal text if shortcode.is_escaped { return ShortcodeResult::Preserve; } - // Find and call handler + // 1. Try built-in Rust handlers first for handler in &self.handlers { if handler.name() == shortcode.name { - return handler.resolve(shortcode, ctx); + return handler.resolve(shortcode, ctx, resolution_ctx); + } + } + + // 2. Try Lua engine (loaded handlers) + if let Some(engine) = lua_engine.as_mut() { + // If handler is already loaded, call it + if engine.has_handler(&shortcode.name) { + return dispatch_lua_shortcode(engine, shortcode, ctx, resolution_ctx); + } + + // 3. Try name-based extension lookup (on-demand loading) + if let Some(ext) = find_extension(&shortcode.name, &self.extensions) { + if !ext.contributes.shortcodes.is_empty() { + for script_path in &ext.contributes.shortcodes { + if let Err(e) = engine.load_script(script_path) { + let diagnostic = + DiagnosticMessageBuilder::warning("Shortcode script error") + .problem(format!( + "Failed to load shortcode script `{}`: {}", + script_path.display(), + e + )) + .with_location(ctx.source_info.clone()) + .build(); + return ShortcodeResult::Error(ShortcodeError { + key: shortcode.name.clone(), + diagnostic, + }); + } + } + // Retry after loading extension scripts + if engine.has_handler(&shortcode.name) { + return dispatch_lua_shortcode(engine, shortcode, ctx, resolution_ctx); + } + } } } @@ -266,6 +374,133 @@ impl Default for ShortcodeResolveTransform { } } +/// Dispatch a shortcode call to the Lua engine and convert the result. +fn dispatch_lua_shortcode( + engine: &mut pampa::lua::LuaShortcodeEngine, + shortcode: &Shortcode, + ctx: &ShortcodeContext, + resolution_ctx: ResolutionContext, +) -> ShortcodeResult { + let args = shortcode_to_lua_args(shortcode, ctx.metadata); + let call_ctx = match resolution_ctx { + ResolutionContext::Block => pampa::lua::ShortcodeCallContext::Block, + ResolutionContext::Inline => pampa::lua::ShortcodeCallContext::Inline, + }; + match engine.call(&shortcode.name, &args, call_ctx) { + Some(result) => lua_result_to_shortcode_result(result, ctx.source_info), + None => { + let diagnostic = DiagnosticMessageBuilder::warning("Shortcode handler not found") + .problem(format!( + "Lua handler for shortcode `{}` was not found", + shortcode.name + )) + .with_location(ctx.source_info.clone()) + .build(); + ShortcodeResult::Error(ShortcodeError { + key: shortcode.name.clone(), + diagnostic, + }) + } + } +} + +/// Convert a q2 `Shortcode` to pampa's `ShortcodeArgs` for Lua dispatch. +fn shortcode_to_lua_args( + shortcode: &Shortcode, + metadata: &ConfigValue, +) -> pampa::lua::ShortcodeArgs { + let positional: Vec = shortcode + .positional_args + .iter() + .filter_map(|arg| match arg { + ShortcodeArg::String(s) => Some(s.clone()), + ShortcodeArg::Number(n) => Some(n.to_string()), + ShortcodeArg::Boolean(b) => Some(b.to_string()), + _ => None, + }) + .collect(); + + let keyword: Vec<(String, String)> = shortcode + .keyword_args + .iter() + .filter_map(|(key, value)| match value { + ShortcodeArg::String(s) => Some((key.clone(), s.clone())), + ShortcodeArg::Number(n) => Some((key.clone(), n.to_string())), + ShortcodeArg::Boolean(b) => Some((key.clone(), b.to_string())), + _ => None, + }) + .collect(); + + // Extract top-level metadata as string key-value pairs for Lua + let meta_entries: Vec<(String, String)> = if let Some(entries) = metadata.as_map_entries() { + entries + .iter() + .filter_map(|entry| { + entry + .value + .as_str() + .map(|v| (entry.key.clone(), v.to_string())) + }) + .collect() + } else { + Vec::new() + }; + + pampa::lua::ShortcodeArgs { + positional, + keyword, + metadata: meta_entries, + } +} + +/// Convert a pampa `LuaShortcodeResult` back to a q2 `ShortcodeResult`. +fn lua_result_to_shortcode_result( + result: pampa::lua::LuaShortcodeResult, + source_info: &SourceInfo, +) -> ShortcodeResult { + match result { + pampa::lua::LuaShortcodeResult::Inlines(inlines) => ShortcodeResult::Inlines(inlines), + pampa::lua::LuaShortcodeResult::Blocks(blocks) => ShortcodeResult::Blocks(blocks), + pampa::lua::LuaShortcodeResult::Text(text) => { + ShortcodeResult::Inlines(vec![Inline::Str(Str { + text, + source_info: SourceInfo::default(), + })]) + } + pampa::lua::LuaShortcodeResult::Error(msg) => { + let diagnostic = DiagnosticMessageBuilder::warning("Shortcode error") + .problem(msg) + .with_location(source_info.clone()) + .build(); + ShortcodeResult::Error(ShortcodeError { + key: "lua-shortcode".to_string(), + diagnostic, + }) + } + } +} + +/// Extract shortcode paths from merged metadata. +/// +/// After metadata merge, `meta["shortcodes"]` contains an array of paths +/// (either `ConfigValueKind::Path` from extensions or `Scalar` from user frontmatter). +pub fn extract_shortcode_paths(meta: &ConfigValue, document_dir: &std::path::Path) -> Vec { + let Some(sc_val) = meta.get("shortcodes") else { + return vec![]; + }; + let Some(items) = sc_val.as_array() else { + return vec![]; + }; + items + .iter() + .filter_map(|item| match &item.value { + ConfigValueKind::Path(s) => Some(document_dir.join(s)), + ConfigValueKind::Scalar(_) => item.as_str().map(|s| document_dir.join(s)), + _ => None, + }) + .collect() +} + impl AstTransform for ShortcodeResolveTransform { fn name(&self) -> &str { "shortcode-resolve" @@ -275,8 +510,52 @@ impl AstTransform for ShortcodeResolveTransform { // Collect diagnostics during traversal let mut diagnostics: Vec = Vec::new(); + // Create Lua engine on the stack if we have paths, extensions, or a runtime. + // The engine is !Send + !Sync so it cannot be stored as a field. + let mut lua_engine = if (!self.lua_shortcode_paths.is_empty() + || !self.extensions.is_empty()) + && self.runtime.is_some() + { + let runtime = self.runtime.as_ref().unwrap().clone(); + match pampa::lua::LuaShortcodeEngine::new(&self.target_format, runtime) { + Ok(mut engine) => { + // Load scripts from metadata-specified paths + for path in &self.lua_shortcode_paths { + if let Err(e) = engine.load_script(path) { + diagnostics.push( + DiagnosticMessageBuilder::warning("Shortcode script error") + .problem(format!( + "Failed to load shortcode script `{}`: {}", + path.display(), + e + )) + .build(), + ); + } + } + Some(engine) + } + Err(e) => { + diagnostics.push( + DiagnosticMessageBuilder::warning("Lua shortcode engine error") + .problem(format!("Failed to create Lua shortcode engine: {}", e)) + .build(), + ); + None + } + } + } else { + None + }; + // Resolve shortcodes in all blocks - resolve_blocks(&mut ast.blocks, self, &ast.meta, &mut diagnostics); + resolve_blocks( + &mut ast.blocks, + self, + &ast.meta, + &mut diagnostics, + &mut lua_engine, + ); // Add any diagnostics to the render context for diagnostic in diagnostics { @@ -288,14 +567,87 @@ impl AstTransform for ShortcodeResolveTransform { } /// Resolve shortcodes in a vector of blocks. +/// +/// Uses index-based iteration because block-context shortcodes can splice +/// multiple blocks in place of a single Para/Plain. fn resolve_blocks( blocks: &mut Vec, transform: &ShortcodeResolveTransform, metadata: &ConfigValue, diagnostics: &mut Vec, + lua_engine: &mut Option, ) { - for block in blocks.iter_mut() { - resolve_block(block, transform, metadata, diagnostics); + let mut i = 0; + while i < blocks.len() { + // Check for block-context shortcode: Para/Plain with exactly one non-escaped Shortcode + if let Some(shortcode) = single_shortcode_in_para_or_plain(&blocks[i]) { + let shortcode_owned = shortcode.clone(); + let ctx = ShortcodeContext { + metadata, + source_info: &shortcode_owned.source_info, + }; + match transform.resolve_shortcode( + &shortcode_owned, + &ctx, + ResolutionContext::Block, + lua_engine, + ) { + ShortcodeResult::Blocks(new_blocks) => { + let n = new_blocks.len(); + blocks.splice(i..=i, new_blocks); + i += n.max(1); + continue; + } + ShortcodeResult::Inlines(inlines) => { + replace_shortcode_in_block(&mut blocks[i], inlines); + i += 1; + continue; + } + ShortcodeResult::Error(error) => { + diagnostics.push(error.diagnostic); + let error_inline = make_error_inline(&error.key); + replace_shortcode_in_block(&mut blocks[i], vec![error_inline]); + i += 1; + continue; + } + ShortcodeResult::Preserve => { + let literal = shortcode_to_literal(&shortcode_owned); + replace_shortcode_in_block(&mut blocks[i], vec![literal]); + i += 1; + continue; + } + } + } + // General case: recurse into block + resolve_block(&mut blocks[i], transform, metadata, diagnostics, lua_engine); + i += 1; + } +} + +/// Check if a block is a Para/Plain with exactly one non-escaped Shortcode inline. +fn single_shortcode_in_para_or_plain(block: &Block) -> Option<&Shortcode> { + let content = match block { + Block::Paragraph(Paragraph { content, .. }) | Block::Plain(Plain { content, .. }) => { + content + } + _ => return None, + }; + if content.len() != 1 { + return None; + } + match &content[0] { + Inline::Shortcode(sc) if !sc.is_escaped => Some(sc), + _ => None, + } +} + +/// Replace the single Shortcode inline in a Para/Plain with the given inlines. +fn replace_shortcode_in_block(block: &mut Block, inlines: Vec) { + match block { + Block::Paragraph(Paragraph { content, .. }) | Block::Plain(Plain { content, .. }) => { + *content = inlines; + } + _ => {} } } @@ -305,53 +657,54 @@ fn resolve_block( transform: &ShortcodeResolveTransform, metadata: &ConfigValue, diagnostics: &mut Vec, + lua_engine: &mut Option, ) { match block { Block::Plain(Plain { content, .. }) | Block::Paragraph(Paragraph { content, .. }) => { - resolve_inlines(content, transform, metadata, diagnostics); + resolve_inlines(content, transform, metadata, diagnostics, lua_engine); } Block::LineBlock(LineBlock { content, .. }) => { for line in content { - resolve_inlines(line, transform, metadata, diagnostics); + resolve_inlines(line, transform, metadata, diagnostics, lua_engine); } } Block::Header(Header { content, .. }) => { - resolve_inlines(content, transform, metadata, diagnostics); + resolve_inlines(content, transform, metadata, diagnostics, lua_engine); } Block::BlockQuote(BlockQuote { content, .. }) => { - resolve_blocks(content, transform, metadata, diagnostics); + resolve_blocks(content, transform, metadata, diagnostics, lua_engine); } Block::OrderedList(OrderedList { content, .. }) => { for item in content { - resolve_blocks(item, transform, metadata, diagnostics); + resolve_blocks(item, transform, metadata, diagnostics, lua_engine); } } Block::BulletList(BulletList { content, .. }) => { for item in content { - resolve_blocks(item, transform, metadata, diagnostics); + resolve_blocks(item, transform, metadata, diagnostics, lua_engine); } } Block::DefinitionList(DefinitionList { content, .. }) => { for (term, defs) in content { - resolve_inlines(term, transform, metadata, diagnostics); + resolve_inlines(term, transform, metadata, diagnostics, lua_engine); for def in defs { - resolve_blocks(def, transform, metadata, diagnostics); + resolve_blocks(def, transform, metadata, diagnostics, lua_engine); } } } Block::Figure(Figure { content, caption, .. }) => { - resolve_blocks(content, transform, metadata, diagnostics); + resolve_blocks(content, transform, metadata, diagnostics, lua_engine); if let Some(short) = &mut caption.short { - resolve_inlines(short, transform, metadata, diagnostics); + resolve_inlines(short, transform, metadata, diagnostics, lua_engine); } if let Some(long) = &mut caption.long { - resolve_blocks(long, transform, metadata, diagnostics); + resolve_blocks(long, transform, metadata, diagnostics, lua_engine); } } Block::Div(Div { content, .. }) => { - resolve_blocks(content, transform, metadata, diagnostics); + resolve_blocks(content, transform, metadata, diagnostics, lua_engine); } Block::Table(Table { caption, @@ -362,29 +715,47 @@ fn resolve_block( }) => { // Table caption if let Some(short) = &mut caption.short { - resolve_inlines(short, transform, metadata, diagnostics); + resolve_inlines(short, transform, metadata, diagnostics, lua_engine); } if let Some(long) = &mut caption.long { - resolve_blocks(long, transform, metadata, diagnostics); + resolve_blocks(long, transform, metadata, diagnostics, lua_engine); } // Table head for row in &mut head.rows { for cell in &mut row.cells { - resolve_blocks(&mut cell.content, transform, metadata, diagnostics); + resolve_blocks( + &mut cell.content, + transform, + metadata, + diagnostics, + lua_engine, + ); } } // Table bodies for body in bodies { for row in &mut body.body { for cell in &mut row.cells { - resolve_blocks(&mut cell.content, transform, metadata, diagnostics); + resolve_blocks( + &mut cell.content, + transform, + metadata, + diagnostics, + lua_engine, + ); } } } // Table foot for row in &mut foot.rows { for cell in &mut row.cells { - resolve_blocks(&mut cell.content, transform, metadata, diagnostics); + resolve_blocks( + &mut cell.content, + transform, + metadata, + diagnostics, + lua_engine, + ); } } } @@ -393,14 +764,14 @@ fn resolve_block( for slot in custom.slots.values_mut() { match slot { quarto_pandoc_types::custom::Slot::Block(b) => { - resolve_block(b, transform, metadata, diagnostics); + resolve_block(b, transform, metadata, diagnostics, lua_engine); } quarto_pandoc_types::custom::Slot::Blocks(bs) => { - resolve_blocks(bs, transform, metadata, diagnostics); + resolve_blocks(bs, transform, metadata, diagnostics, lua_engine); } quarto_pandoc_types::custom::Slot::Inline(i) => { let mut inlines = vec![i.as_ref().clone()]; - resolve_inlines(&mut inlines, transform, metadata, diagnostics); + resolve_inlines(&mut inlines, transform, metadata, diagnostics, lua_engine); if inlines.len() == 1 { **i = inlines.pop().unwrap(); } @@ -408,7 +779,7 @@ fn resolve_block( // back into a single Inline slot - keep the original } quarto_pandoc_types::custom::Slot::Inlines(is) => { - resolve_inlines(is, transform, metadata, diagnostics); + resolve_inlines(is, transform, metadata, diagnostics, lua_engine); } } } @@ -430,16 +801,23 @@ fn resolve_inlines( transform: &ShortcodeResolveTransform, metadata: &ConfigValue, diagnostics: &mut Vec, + lua_engine: &mut Option, ) { let mut i = 0; while i < inlines.len() { if let Inline::Shortcode(shortcode) = &inlines[i] { + let shortcode_owned = shortcode.clone(); let shortcode_ctx = ShortcodeContext { metadata, - source_info: &shortcode.source_info, + source_info: &shortcode_owned.source_info, }; - match transform.resolve_shortcode(shortcode, &shortcode_ctx) { + match transform.resolve_shortcode( + &shortcode_owned, + &shortcode_ctx, + ResolutionContext::Inline, + lua_engine, + ) { ShortcodeResult::Inlines(replacement) => { // Replace shortcode with resolved inlines let replacement_len = replacement.len(); @@ -448,6 +826,13 @@ fn resolve_inlines( // but even if they do, we don't want infinite loops) i += replacement_len.max(1); } + ShortcodeResult::Blocks(blocks) => { + // Graceful degradation: flatten blocks to inlines + let replacement = flatten_blocks_to_inlines(&blocks); + let replacement_len = replacement.len(); + inlines.splice(i..=i, replacement); + i += replacement_len.max(1); + } ShortcodeResult::Error(error) => { // Emit diagnostic diagnostics.push(error.diagnostic); @@ -458,14 +843,20 @@ fn resolve_inlines( } ShortcodeResult::Preserve => { // Convert escaped shortcode to literal text - let literal = shortcode_to_literal(shortcode); + let literal = shortcode_to_literal(&shortcode_owned); inlines[i] = literal; i += 1; } } } else { // Recurse into inline containers - recurse_inline(&mut inlines[i], transform, metadata, diagnostics); + recurse_inline( + &mut inlines[i], + transform, + metadata, + diagnostics, + lua_engine, + ); i += 1; } } @@ -477,6 +868,7 @@ fn recurse_inline( transform: &ShortcodeResolveTransform, metadata: &ConfigValue, diagnostics: &mut Vec, + lua_engine: &mut Option, ) { match inline { Inline::Emph(Emph { content, .. }) @@ -489,45 +881,45 @@ fn recurse_inline( | Inline::Insert(Insert { content, .. }) | Inline::Delete(Delete { content, .. }) | Inline::Highlight(Highlight { content, .. }) => { - resolve_inlines(content, transform, metadata, diagnostics); + resolve_inlines(content, transform, metadata, diagnostics, lua_engine); } Inline::Quoted(Quoted { content, .. }) => { - resolve_inlines(content, transform, metadata, diagnostics); + resolve_inlines(content, transform, metadata, diagnostics, lua_engine); } Inline::Cite(Cite { content, .. }) => { - resolve_inlines(content, transform, metadata, diagnostics); + resolve_inlines(content, transform, metadata, diagnostics, lua_engine); } Inline::Link(Link { content, .. }) | Inline::Image(Image { content, .. }) => { - resolve_inlines(content, transform, metadata, diagnostics); + resolve_inlines(content, transform, metadata, diagnostics, lua_engine); } Inline::Note(Note { content, .. }) => { - resolve_blocks(content, transform, metadata, diagnostics); + resolve_blocks(content, transform, metadata, diagnostics, lua_engine); } Inline::Span(Span { content, .. }) => { - resolve_inlines(content, transform, metadata, diagnostics); + resolve_inlines(content, transform, metadata, diagnostics, lua_engine); } Inline::EditComment(EditComment { content, .. }) => { - resolve_inlines(content, transform, metadata, diagnostics); + resolve_inlines(content, transform, metadata, diagnostics, lua_engine); } Inline::Custom(custom) => { // Resolve shortcodes in custom inline node slots for slot in custom.slots.values_mut() { match slot { quarto_pandoc_types::custom::Slot::Inlines(is) => { - resolve_inlines(is, transform, metadata, diagnostics); + resolve_inlines(is, transform, metadata, diagnostics, lua_engine); } quarto_pandoc_types::custom::Slot::Inline(i) => { let mut inlines = vec![i.as_ref().clone()]; - resolve_inlines(&mut inlines, transform, metadata, diagnostics); + resolve_inlines(&mut inlines, transform, metadata, diagnostics, lua_engine); if inlines.len() == 1 { **i = inlines.pop().unwrap(); } } quarto_pandoc_types::custom::Slot::Blocks(bs) => { - resolve_blocks(bs, transform, metadata, diagnostics); + resolve_blocks(bs, transform, metadata, diagnostics, lua_engine); } quarto_pandoc_types::custom::Slot::Block(b) => { - resolve_block(b, transform, metadata, diagnostics); + resolve_block(b, transform, metadata, diagnostics, lua_engine); } } } @@ -741,7 +1133,7 @@ mod tests { source_info: &shortcode.source_info, }; - let result = handler.resolve(&shortcode, &ctx); + let result = handler.resolve(&shortcode, &ctx, ResolutionContext::Inline); match result { ShortcodeResult::Inlines(inlines) => { assert_eq!(inlines.len(), 1); @@ -773,7 +1165,7 @@ mod tests { source_info: &shortcode.source_info, }; - let result = handler.resolve(&shortcode, &ctx); + let result = handler.resolve(&shortcode, &ctx, ResolutionContext::Inline); match result { ShortcodeResult::Error(err) => { assert_eq!(err.key, "meta:nonexistent"); @@ -794,7 +1186,7 @@ mod tests { source_info: &shortcode.source_info, }; - let result = handler.resolve(&shortcode, &ctx); + let result = handler.resolve(&shortcode, &ctx, ResolutionContext::Inline); match result { ShortcodeResult::Error(err) => { assert_eq!(err.key, "meta"); @@ -820,7 +1212,8 @@ mod tests { source_info: &shortcode.source_info, }; - let result = transform.resolve_shortcode(&shortcode, &ctx); + let result = + transform.resolve_shortcode(&shortcode, &ctx, ResolutionContext::Inline, &mut None); assert!(matches!(result, ShortcodeResult::Preserve)); } @@ -835,7 +1228,8 @@ mod tests { source_info: &shortcode.source_info, }; - let result = transform.resolve_shortcode(&shortcode, &ctx); + let result = + transform.resolve_shortcode(&shortcode, &ctx, ResolutionContext::Inline, &mut None); match result { ShortcodeResult::Error(err) => { assert_eq!(err.key, "unknown"); @@ -966,4 +1360,543 @@ mod tests { // Verify warning was emitted assert_eq!(ctx.diagnostics.len(), 1); } + + /// A test handler that returns Blocks when in block context. + struct BlockTestHandler; + impl ShortcodeHandler for BlockTestHandler { + fn name(&self) -> &str { + "block-test" + } + fn resolve( + &self, + _shortcode: &Shortcode, + _ctx: &ShortcodeContext, + resolution_ctx: ResolutionContext, + ) -> ShortcodeResult { + match resolution_ctx { + ResolutionContext::Block => ShortcodeResult::Blocks(vec![Block::HorizontalRule( + quarto_pandoc_types::block::HorizontalRule { + source_info: SourceInfo::default(), + }, + )]), + ResolutionContext::Inline => ShortcodeResult::Inlines(vec![Inline::Str(Str { + text: "inline-fallback".to_string(), + source_info: SourceInfo::default(), + })]), + } + } + } + + #[test] + fn test_block_shortcode_replaces_para() { + let transform = ShortcodeResolveTransform { + handlers: vec![Box::new(BlockTestHandler)], + lua_shortcode_paths: Vec::new(), + extensions: Vec::new(), + runtime: None, + target_format: String::new(), + }; + + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![Inline::Shortcode(make_shortcode("block-test", vec![]))], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + // The Para should be replaced by a HorizontalRule + assert_eq!(ast.blocks.len(), 1); + assert!( + matches!(&ast.blocks[0], Block::HorizontalRule(_)), + "Expected HorizontalRule, got {:?}", + ast.blocks[0] + ); + } + + #[test] + fn test_inline_shortcode_in_para_stays_inline() { + let transform = ShortcodeResolveTransform { + handlers: vec![Box::new(BlockTestHandler)], + lua_shortcode_paths: Vec::new(), + extensions: Vec::new(), + runtime: None, + target_format: String::new(), + }; + + // Para with text + shortcode — not a block-context shortcode + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![ + Inline::Str(Str { + text: "Before: ".to_string(), + source_info: dummy_source_info(), + }), + Inline::Shortcode(make_shortcode("block-test", vec![])), + ], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + // Should remain a Paragraph with resolved inline content + assert_eq!(ast.blocks.len(), 1); + if let Block::Paragraph(para) = &ast.blocks[0] { + assert_eq!(para.content.len(), 2); + if let Inline::Str(s) = ¶.content[1] { + assert_eq!(s.text, "inline-fallback"); + } else { + panic!("Expected Str inline, got {:?}", para.content[1]); + } + } else { + panic!("Expected Paragraph"); + } + } + + /// Handler that always returns Blocks regardless of context. + struct AlwaysBlockHandler; + impl ShortcodeHandler for AlwaysBlockHandler { + fn name(&self) -> &str { + "always-block" + } + fn resolve( + &self, + _shortcode: &Shortcode, + _ctx: &ShortcodeContext, + _resolution_ctx: ResolutionContext, + ) -> ShortcodeResult { + ShortcodeResult::Blocks(vec![Block::Paragraph(Paragraph { + content: vec![Inline::Str(Str { + text: "from-block".to_string(), + source_info: SourceInfo::default(), + })], + source_info: SourceInfo::default(), + })]) + } + } + + #[test] + fn test_block_result_in_inline_context() { + let transform = ShortcodeResolveTransform { + handlers: vec![Box::new(AlwaysBlockHandler)], + lua_shortcode_paths: Vec::new(), + extensions: Vec::new(), + runtime: None, + target_format: String::new(), + }; + + // Para with text + shortcode — inline context, handler returns Blocks + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![ + Inline::Str(Str { + text: "Before: ".to_string(), + source_info: dummy_source_info(), + }), + Inline::Shortcode(make_shortcode("always-block", vec![])), + ], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + // Blocks should be flattened to inlines + if let Block::Paragraph(para) = &ast.blocks[0] { + assert_eq!(para.content.len(), 2); + if let Inline::Str(s) = ¶.content[1] { + assert_eq!(s.text, "from-block"); + } else { + panic!("Expected Str inline, got {:?}", para.content[1]); + } + } else { + panic!("Expected Paragraph"); + } + } + + #[test] + fn test_escaped_shortcode_block_context() { + let transform = ShortcodeResolveTransform { + handlers: vec![Box::new(MetaShortcodeHandler)], + lua_shortcode_paths: Vec::new(), + extensions: Vec::new(), + runtime: None, + target_format: String::new(), + }; + + // Escaped shortcode alone in Para — should preserve as literal + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![Inline::Shortcode(Shortcode { + is_escaped: true, + name: "meta".to_string(), + positional_args: vec![ShortcodeArg::String("title".to_string())], + keyword_args: HashMap::new(), + source_info: dummy_source_info(), + })], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + // Should be converted to literal text in the Para + if let Block::Paragraph(para) = &ast.blocks[0] { + assert_eq!(para.content.len(), 1); + if let Inline::Str(s) = ¶.content[0] { + assert_eq!(s.text, "{{< meta title >}}"); + } else { + panic!("Expected Str inline, got {:?}", para.content[0]); + } + } else { + panic!("Expected Paragraph"); + } + } + + // === Lua integration tests (3.4.6) === + // These tests require the native Lua runtime + #[cfg(not(target_arch = "wasm32"))] + mod lua_integration { + use super::*; + use crate::extension::types::{Contributes, Extension, ExtensionId}; + use std::io::Write; + use tempfile::TempDir; + + fn make_runtime() -> Arc { + Arc::new(quarto_system_runtime::NativeRuntime::new()) + } + + fn write_lua_script(dir: &std::path::Path, name: &str, content: &str) -> PathBuf { + let path = dir.join(name); + let mut f = std::fs::File::create(&path).unwrap(); + f.write_all(content.as_bytes()).unwrap(); + path + } + + fn make_extension(name: &str, shortcode_paths: Vec) -> Extension { + Extension { + id: ExtensionId::new(name), + title: name.to_string(), + author: String::new(), + version: None, + quarto_required: None, + path: PathBuf::from("/extensions").join(name), + contributes: Contributes { + shortcodes: shortcode_paths, + ..Default::default() + }, + } + } + + #[test] + fn test_lua_shortcode_from_metadata_paths() { + let tmp = TempDir::new().unwrap(); + let script_path = write_lua_script( + tmp.path(), + "hello.lua", + r#"return { hello = function(args) return "Hello from Lua" end }"#, + ); + + let runtime = make_runtime(); + let transform = ShortcodeResolveTransform::with_lua_support( + vec![script_path], + Vec::new(), + runtime, + "html".to_string(), + ); + + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![Inline::Shortcode(make_shortcode("hello", vec![]))], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + if let Block::Paragraph(para) = &ast.blocks[0] { + if let Inline::Str(s) = ¶.content[0] { + assert_eq!(s.text, "Hello from Lua"); + } else { + panic!("Expected Str inline, got {:?}", para.content[0]); + } + } else { + panic!("Expected Paragraph"); + } + assert!(ctx.diagnostics.is_empty()); + } + + #[test] + fn test_lua_shortcode_by_extension_name() { + let tmp = TempDir::new().unwrap(); + let script_path = write_lua_script( + tmp.path(), + "greet.lua", + r#"return { greet = function(args) return "Extension greeting" end }"#, + ); + + let ext = make_extension("greet", vec![script_path]); + let runtime = make_runtime(); + + // No metadata paths — extension discovered by name + let transform = ShortcodeResolveTransform::with_lua_support( + Vec::new(), + vec![ext], + runtime, + "html".to_string(), + ); + + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![Inline::Shortcode(make_shortcode("greet", vec![]))], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + if let Block::Paragraph(para) = &ast.blocks[0] { + if let Inline::Str(s) = ¶.content[0] { + assert_eq!(s.text, "Extension greeting"); + } else { + panic!("Expected Str inline, got {:?}", para.content[0]); + } + } else { + panic!("Expected Paragraph"); + } + assert!(ctx.diagnostics.is_empty()); + } + + #[test] + fn test_rust_handler_overrides_lua() { + let tmp = TempDir::new().unwrap(); + // Lua script defines a "meta" handler that should be ignored + let script_path = write_lua_script( + tmp.path(), + "meta.lua", + r#"return { meta = function(args) return "FROM LUA" end }"#, + ); + + let runtime = make_runtime(); + let transform = ShortcodeResolveTransform::with_lua_support( + vec![script_path], + Vec::new(), + runtime, + "html".to_string(), + ); + + let meta = ConfigValue::new_map( + vec![make_map_entry( + "title", + ConfigValue::new_string("Rust Title", dummy_source_info()), + )], + dummy_source_info(), + ); + + let mut ast = Pandoc { + meta, + blocks: vec![Block::Paragraph(Paragraph { + content: vec![Inline::Shortcode(make_shortcode("meta", vec!["title"]))], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + // Built-in Rust handler should win over Lua + if let Block::Paragraph(para) = &ast.blocks[0] { + if let Inline::Str(s) = ¶.content[0] { + assert_eq!(s.text, "Rust Title"); + } else { + panic!("Expected Str inline, got {:?}", para.content[0]); + } + } else { + panic!("Expected Paragraph"); + } + } + + #[test] + fn test_unknown_shortcode_error() { + let runtime = make_runtime(); + let transform = ShortcodeResolveTransform::with_lua_support( + Vec::new(), + Vec::new(), + runtime, + "html".to_string(), + ); + + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![Inline::Shortcode(make_shortcode("nonexistent", vec![]))], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + // Should produce error inline + if let Block::Paragraph(para) = &ast.blocks[0] { + if let Inline::Strong(strong) = ¶.content[0] { + if let Inline::Str(s) = &strong.content[0] { + assert_eq!(s.text, "?nonexistent"); + } else { + panic!("Expected Str in Strong"); + } + } else { + panic!("Expected Strong inline, got {:?}", para.content[0]); + } + } else { + panic!("Expected Paragraph"); + } + assert_eq!(ctx.diagnostics.len(), 1); + } + + #[test] + fn test_extension_shortcode_block_context() { + let tmp = TempDir::new().unwrap(); + let script_path = write_lua_script( + tmp.path(), + "break.lua", + r#"return { ["break"] = function(args) return pandoc.HorizontalRule() end }"#, + ); + + let ext = make_extension("break", vec![script_path]); + let runtime = make_runtime(); + + let transform = ShortcodeResolveTransform::with_lua_support( + Vec::new(), + vec![ext], + runtime, + "html".to_string(), + ); + + // Shortcode alone in Para → block context + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![Inline::Shortcode(make_shortcode("break", vec![]))], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + // Para should be replaced by HorizontalRule + assert_eq!(ast.blocks.len(), 1); + assert!( + matches!(&ast.blocks[0], Block::HorizontalRule(_)), + "Expected HorizontalRule, got {:?}", + ast.blocks[0] + ); + assert!(ctx.diagnostics.is_empty()); + } + + #[test] + fn test_full_transform_block_shortcode_rawblock() { + let tmp = TempDir::new().unwrap(); + let script_path = write_lua_script( + tmp.path(), + "pagebreak.lua", + r#"return { pagebreak = function(args) return pandoc.RawBlock("html", "
") end }"#, + ); + + let runtime = make_runtime(); + let transform = ShortcodeResolveTransform::with_lua_support( + vec![script_path], + Vec::new(), + runtime, + "html".to_string(), + ); + + let mut ast = Pandoc { + meta: ConfigValue::default(), + blocks: vec![Block::Paragraph(Paragraph { + content: vec![Inline::Shortcode(make_shortcode("pagebreak", vec![]))], + source_info: dummy_source_info(), + })], + }; + + let project = make_test_project(); + let doc = DocumentInfo::from_path("/project/doc.qmd"); + let format = Format::html(); + let binaries = BinaryDependencies::new(); + let mut ctx = RenderContext::new(&project, &doc, &format, &binaries); + + transform.transform(&mut ast, &mut ctx).unwrap(); + + // Para should be replaced by RawBlock + assert_eq!(ast.blocks.len(), 1); + match &ast.blocks[0] { + Block::RawBlock(rb) => { + assert_eq!(rb.format, "html"); + assert!(rb.text.contains("page-break")); + } + other => panic!("Expected RawBlock, got {:?}", other), + } + assert!(ctx.diagnostics.is_empty()); + } + } } diff --git a/crates/quarto/tests/smoke-all/extensions/block-shortcode/_extensions/break/_extension.yml b/crates/quarto/tests/smoke-all/extensions/block-shortcode/_extensions/break/_extension.yml new file mode 100644 index 00000000..762035b6 --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/block-shortcode/_extensions/break/_extension.yml @@ -0,0 +1,5 @@ +title: Block Shortcode Extension +author: Test +contributes: + shortcodes: + - break.lua diff --git a/crates/quarto/tests/smoke-all/extensions/block-shortcode/_extensions/break/break.lua b/crates/quarto/tests/smoke-all/extensions/block-shortcode/_extensions/break/break.lua new file mode 100644 index 00000000..7419075f --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/block-shortcode/_extensions/break/break.lua @@ -0,0 +1,5 @@ +return { + ["break"] = function(args) + return pandoc.RawBlock("html", '
') + end +} diff --git a/crates/quarto/tests/smoke-all/extensions/block-shortcode/test.qmd b/crates/quarto/tests/smoke-all/extensions/block-shortcode/test.qmd new file mode 100644 index 00000000..73821d6b --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/block-shortcode/test.qmd @@ -0,0 +1,16 @@ +--- +title: Block Shortcode Test +format: html +_quarto: + tests: + html: + noErrors: true + ensureHtmlElements: + - ["hr.ext-break"] +--- + +Before the break. + +{{< break >}} + +After the break. diff --git a/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/_extensions/myext/_extension.yml b/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/_extensions/myext/_extension.yml new file mode 100644 index 00000000..68915edd --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/_extensions/myext/_extension.yml @@ -0,0 +1,7 @@ +title: Format Shortcode Extension +author: Test +contributes: + formats: + html: + shortcodes: + - greeting.lua diff --git a/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/_extensions/myext/greeting.lua b/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/_extensions/myext/greeting.lua new file mode 100644 index 00000000..ac3ca3ba --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/_extensions/myext/greeting.lua @@ -0,0 +1,5 @@ +return { + greeting = function(args) + return "FORMAT-SHORTCODE-ACTIVE" + end +} diff --git a/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/test.qmd b/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/test.qmd new file mode 100644 index 00000000..47d7c4fb --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/format-with-shortcodes/test.qmd @@ -0,0 +1,11 @@ +--- +title: Format Shortcode Test +_quarto: + tests: + myext-html: + noErrors: true + ensureFileRegexMatches: + - ["FORMAT-SHORTCODE-ACTIVE"] +--- + +Result: {{< greeting >}} diff --git a/crates/quarto/tests/smoke-all/extensions/shortcode-extension/_extensions/hello/_extension.yml b/crates/quarto/tests/smoke-all/extensions/shortcode-extension/_extensions/hello/_extension.yml new file mode 100644 index 00000000..0d5895d3 --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/shortcode-extension/_extensions/hello/_extension.yml @@ -0,0 +1,5 @@ +title: Hello Shortcode Extension +author: Test +contributes: + shortcodes: + - hello.lua diff --git a/crates/quarto/tests/smoke-all/extensions/shortcode-extension/_extensions/hello/hello.lua b/crates/quarto/tests/smoke-all/extensions/shortcode-extension/_extensions/hello/hello.lua new file mode 100644 index 00000000..6402d39e --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/shortcode-extension/_extensions/hello/hello.lua @@ -0,0 +1,5 @@ +return { + hello = function(args) + return "HELLO-SHORTCODE-ACTIVE" + end +} diff --git a/crates/quarto/tests/smoke-all/extensions/shortcode-extension/test.qmd b/crates/quarto/tests/smoke-all/extensions/shortcode-extension/test.qmd new file mode 100644 index 00000000..9c4e336a --- /dev/null +++ b/crates/quarto/tests/smoke-all/extensions/shortcode-extension/test.qmd @@ -0,0 +1,12 @@ +--- +title: Shortcode Extension Test +format: html +_quarto: + tests: + html: + noErrors: true + ensureFileRegexMatches: + - ["HELLO-SHORTCODE-ACTIVE"] +--- + +Result: {{< hello >}} diff --git a/hub-client/e2e/helpers/previewExtraction.ts b/hub-client/e2e/helpers/previewExtraction.ts index f59237b9..e185056a 100644 --- a/hub-client/e2e/helpers/previewExtraction.ts +++ b/hub-client/e2e/helpers/previewExtraction.ts @@ -14,23 +14,60 @@ import type { Page } from '@playwright/test'; * comment on each render. We wait for: * 1. An iframe with class `preview-active` to exist * 2. Its body to have non-empty innerHTML (content rendered) + * + * If `consoleErrors` is provided, the wait will abort early when a fatal + * browser error is detected (e.g. WebSocket failure, WASM crash), avoiding + * a long timeout with no diagnostic info. */ export async function waitForPreviewRender( page: Page, - opts: { timeout?: number } = {}, + opts: { timeout?: number; consoleErrors?: string[] } = {}, ): Promise { const timeout = opts.timeout ?? 30000; + const consoleErrors = opts.consoleErrors; + + // Poll for render completion, but also check for fatal console errors + // so we can fail fast with a useful message instead of timing out. + const pollInterval = 250; + const deadline = Date.now() + timeout; + + while (Date.now() < deadline) { + // Check if any fatal console errors have been collected + if (consoleErrors && consoleErrors.length > 0) { + // Only treat unrecoverable WASM traps as immediately fatal. + // Lua panics ("panicked at ... lua error") are transient — they happen + // when extension files haven't synced yet, and the app retries on re-render. + // Network errors (500s) may also be transient. + const fatal = consoleErrors.find( + (e) => + e.includes('unreachable') || + e.includes('RuntimeError'), + ); + if (fatal) { + throw new Error( + `Fatal browser error during render wait: ${fatal}\nAll console errors:\n${consoleErrors.join('\n')}`, + ); + } + } - // Wait for the active preview iframe to have rendered content - // The render marker comment () indicates completion - await page.waitForFunction( - () => { + // Check if the preview iframe has rendered content + const rendered = await page.evaluate(() => { const iframe = document.querySelector('iframe.preview-active') as HTMLIFrameElement | null; if (!iframe?.contentDocument?.body) return false; - const html = iframe.contentDocument.body.innerHTML; - return html.length > 0; - }, - { timeout }, + return iframe.contentDocument.body.innerHTML.length > 0; + }); + + if (rendered) return; + + await page.waitForTimeout(pollInterval); + } + + // Timed out — build a useful error message + const errorContext = consoleErrors?.length + ? `\nConsole errors:\n${consoleErrors.join('\n')}` + : '\nNo console errors captured.'; + throw new Error( + `Timed out after ${timeout}ms waiting for preview iframe to render.${errorContext}`, ); } diff --git a/hub-client/e2e/smoke-all.spec.ts b/hub-client/e2e/smoke-all.spec.ts index 186f3da6..01b6d84e 100644 --- a/hub-client/e2e/smoke-all.spec.ts +++ b/hub-client/e2e/smoke-all.spec.ts @@ -49,12 +49,35 @@ test.describe('smoke-all E2E tests', () => { } test(testName, async ({ page }) => { + // Collect console errors and HTTP failures for diagnostics + const consoleErrors: string[] = []; + page.on('console', (msg) => { + if (msg.type() === 'error') { + consoleErrors.push(msg.text()); + } + }); + page.on('pageerror', (err) => { + consoleErrors.push(err.message); + }); + page.on('response', (resp) => { + if (resp.status() >= 500) { + consoleErrors.push(`HTTP ${resp.status()} from ${resp.url()}`); + } + }); + const serverUrl = getServerUrl(); - // Create Automerge project with all fixture files + // Create Automerge project with all fixture files. + // Sync the render target (QMD) LAST so that extension/filter files + // are already in the VFS when the Preview component's first render fires. + const sortedFiles = [...fixture.projectFiles].sort((a, b) => { + const aIsTarget = a.path === fixture.renderPath ? 1 : 0; + const bIsTarget = b.path === fixture.renderPath ? 1 : 0; + return aIsTarget - bIsTarget; + }); const indexDocId = await createProjectOnServer( serverUrl, - fixture.projectFiles.map((f) => ({ + sortedFiles.map((f) => ({ path: f.path, content: f.content, contentType: 'text' as const, @@ -77,7 +100,10 @@ test.describe('smoke-all E2E tests', () => { // Wait for render (or error) if (!spec.expectsError) { - await waitForPreviewRender(page, { timeout: 45000 }); + await waitForPreviewRender(page, { + timeout: 45000, + consoleErrors, + }); } else { // For expected errors, wait a bit for the render attempt to complete await page.waitForTimeout(5000); diff --git a/hub-client/src/components/PreviewRouter.test.ts b/hub-client/src/components/PreviewRouter.test.ts deleted file mode 100644 index 3a5b70ea..00000000 --- a/hub-client/src/components/PreviewRouter.test.ts +++ /dev/null @@ -1,102 +0,0 @@ -import { describe, it, expect } from 'vitest'; -import { getQ2Format } from './render/getQ2Format'; - -/** Helper to build an AST JSON string with a MetaString format value */ -function metaString(value: string): string { - return JSON.stringify({ meta: { format: { t: 'MetaString', c: value } } }); -} - -/** Helper to build an AST JSON string with a MetaInlines format value */ -function metaInlines(value: string): string { - return JSON.stringify({ - meta: { format: { t: 'MetaInlines', c: [{ t: 'Str', c: value }] } }, - }); -} - -describe('getQ2Format', () => { - // --- q2-* formats should be returned --- - - it('returns "q2-slides" from MetaString', () => { - expect(getQ2Format(metaString('q2-slides'))).toBe('q2-slides'); - }); - - it('returns "q2-slides" from MetaInlines', () => { - expect(getQ2Format(metaInlines('q2-slides'))).toBe('q2-slides'); - }); - - it('returns "q2-debug" from MetaString', () => { - expect(getQ2Format(metaString('q2-debug'))).toBe('q2-debug'); - }); - - it('returns "q2-debug" from MetaInlines', () => { - expect(getQ2Format(metaInlines('q2-debug'))).toBe('q2-debug'); - }); - - // --- non-q2 formats should return null --- - - it('returns null for "html" (MetaString)', () => { - expect(getQ2Format(metaString('html'))).toBeNull(); - }); - - it('returns null for "html" (MetaInlines)', () => { - expect(getQ2Format(metaInlines('html'))).toBeNull(); - }); - - it('returns null for "pdf"', () => { - expect(getQ2Format(metaString('pdf'))).toBeNull(); - }); - - it('returns null for "docx"', () => { - expect(getQ2Format(metaString('docx'))).toBeNull(); - }); - - it('returns "revealjs" from MetaString', () => { - expect(getQ2Format(metaString('revealjs'))).toBe('revealjs'); - }); - - it('returns "revealjs" from MetaInlines', () => { - expect(getQ2Format(metaInlines('revealjs'))).toBe('revealjs'); - }); - - it('returns null for "epub"', () => { - expect(getQ2Format(metaString('epub'))).toBeNull(); - }); - - // --- edge cases --- - - it('returns null when meta has no format key', () => { - expect(getQ2Format(JSON.stringify({ meta: { title: { t: 'MetaString', c: 'Hello' } } }))).toBeNull(); - }); - - it('returns null when meta is empty', () => { - expect(getQ2Format(JSON.stringify({ meta: {} }))).toBeNull(); - }); - - it('returns null when there is no meta', () => { - expect(getQ2Format(JSON.stringify({}))).toBeNull(); - }); - - it('returns null for invalid JSON', () => { - expect(getQ2Format('not json')).toBeNull(); - }); - - it('returns null for empty string', () => { - expect(getQ2Format('')).toBeNull(); - }); - - it('returns null for empty MetaInlines array', () => { - expect(getQ2Format(JSON.stringify({ - meta: { format: { t: 'MetaInlines', c: [] } }, - }))).toBeNull(); - }); - - it('returns null for unknown meta type', () => { - expect(getQ2Format(JSON.stringify({ - meta: { format: { t: 'MetaMap', c: {} } }, - }))).toBeNull(); - }); - - it('returns null for format value that is just "q2-" with no suffix', () => { - expect(getQ2Format(metaString('q2-'))).toBe('q2-'); - }); -});