Make fastpylight highlighting optional, shrinking the wheel from 8.8MB to 0.95MB

Cargo.toml

@@ -28,14 +28,11 @@ codegen-units = 16
 [dependencies]
 base64 = "0.23.0"
 fast5ever = { git = "https://github.com/AnswerDotAI/fast5ever" }
-fastpylight = { git = "https://github.com/AnswerDotAI/fastpylight", default-features = false, features = ["standard-languages", "themes"], optional = true }
 html-escape = ">=0.2"
 pyo3 = { version = ">=0.28", optional = true }
 unicode-properties = "0.1"
 
 [features]
-default = ["hl"]
-hl = ["dep:fastpylight"]
 python = ["dep:pyo3"]
 extension-module = ["python", "pyo3/extension-module"]
 

DEV.md

@@ -96,6 +96,13 @@ gen_docs()
 
 Rust renders provisional markup and does no HTML parsing. `python/mdhtml/__init__.py` sends that markup through `mdhtml2dom`, backed by [fast5ever](https://github.com/AnswerDotAI/fast5ever) (html5ever with an arena DOM and Python bindings), so parsing, tree construction, and serialization are the WHATWG algorithms as one engine spells them. The README describes the public API and `docs/DIALECT.md` defines the resulting DOM contract.
 
+Non-Markdown syntax highlighting is an optional Python-layer adapter rather
+than a Rust dependency. Python imports fastpylight lazily and passes its result
+through `HtmlExportOptions::hl_fn`; the base Rust crate therefore carries no
+fastpylight or tree-sitter code. Without the `hl` extra, `mdhtml2html` leaves
+those code blocks plain and reports a warning, while Markdown fences continue
+to use mdhtml's own highlighter.
+
 `ops()` is the semantic-operation view over that DOM. Its traversal follows both ordinary children and inert `template.content`, returning live fast5ever nodes so source-specific pipelines can detach or replace operations without adding mutation policy to mdhtml.
 
 ## Render callbacks

README.md

@@ -61,6 +61,13 @@ Install via pip to get both the Python API and the `md2mdhtml` CLI:
 pip install mdhtml
 ```
 
+The base install has no syntax-highlighter dependency. Install `mdhtml[hl]` for
+fastpylight highlighting and the theme assets used by `md2html` and `viewmd`:
+
+```bash
+pip install 'mdhtml[hl]'
+```
+
 The CLI reads Markdown from stdin or from an optional file path and writes an MDHTML fragment to stdout:
 
 ```bash
@@ -72,7 +79,7 @@ md2mdhtml --implicit_figures input.md > out.html
 md2mdhtml --no-bare_autolinks input.md > out.html
 ```
 
-`md2html` goes the rest of the way, lowering that fragment to a finished HTML page: references baked, headings and captions numbered, code highlighted (```` ```markdown ```` fences by mdhtml itself, everything else by fastpylight), mustache tokens shown as styled pills, and the assets those features need (`dialect_css`, light and dark fastpylight themes, KaTeX plus `math_js`) composed into the page. With no `--out` it writes the page under `~/.cache/md2html/` and opens it in a browser, inlining local images so the page renders from anywhere; piped, it writes to stdout instead, and `--out -` forces that even at a terminal. `--fragment` emits the body alone. `--frontmatter` recognizes a leading metadata block (see below), and ```mermaid fences become diagrams drawn in place by mermaid.js. References default to `--refs=ids`, which shows each reference's target id and never fails on a draft; `--refs=resolve` numbers them and raises on a broken one, and `--refs=lenient` numbers what it can and warns about the rest.
+`md2html` goes the rest of the way, lowering that fragment to a finished HTML page: references baked, headings and captions numbered, code highlighted (```` ```markdown ```` fences by mdhtml itself, everything else by the optional fastpylight extra), mustache tokens shown as styled pills, and the assets those features need (`dialect_css`, light and dark fastpylight themes, KaTeX plus `math_js`) composed into the page. With no `--out` it writes the page under `~/.cache/md2html/` and opens it in a browser, inlining local images so the page renders from anywhere; piped, it writes to stdout instead, and `--out -` forces that even at a terminal. `--fragment` emits the body alone. `--frontmatter` recognizes a leading metadata block (see below), and ```mermaid fences become diagrams drawn in place by mermaid.js. References default to `--refs=ids`, which shows each reference's target id and never fails on a draft; `--refs=resolve` numbers them and raises on a broken one, and `--refs=lenient` numbers what it can and warns about the rest.
 
 ```bash
 md2html input.md
@@ -365,7 +372,7 @@ The result is still a body fragment (a str subclass carrying a `warnings` list;
 - `{=html}` raw data is decoded and spliced in place; raw data for other formats is removed. Malformed payloads are dropped with a warning.
 - A `colwidths` attribute lowers to a `<colgroup>`; `fr` values share the width remaining after fixed lengths.
 - A `width` attribute on a table lowers to an inline style width (bare number = px; invalid values stay visible); it merges last, so it beats `colwidths`' `width:100%`.
-- Code blocks with a language are highlighted (natively, via the statically linked fastpylight engine): `hl='spans'` (default) emits `hl-*` classed spans, `hl='api'` wraps the block in the `<hl-code>` element for the CSS Custom Highlight API, and `hl=None` leaves code untouched. Two per-block hooks customize this: `hl_lang(text, lang)` may return a corrected language before highlighting (e.g. mapping a `%%sql` first line to `sql`), and `code_wrap(html, lang, text)` may return replacement markup for the finished block (a copy-button wrapper, a mermaid `pre`).
+- Code blocks with a language are highlighted through the optional [fastpylight](https://github.com/AnswerDotAI/fastpylight) package (`pip install 'mdhtml[hl]'`): `hl='spans'` (default) emits `hl-*` classed spans, `hl='api'` wraps the block in the `<hl-code>` element for the CSS Custom Highlight API, and `hl=None` leaves code untouched. Without fastpylight installed, code blocks render plain and a warning reports it (```` ```markdown ```` fences always self-highlight, with no dependency). Rust consumers get the same seam as the `hl_fn` slot on `HtmlExportOptions`: a `(code, lang, mode)` hook returning highlighted markup. Two per-block hooks customize this: `hl_lang(text, lang)` may return a corrected language before highlighting (e.g. mapping a `%%sql` first line to `sql`), and `code_wrap(html, lang, text)` may return replacement markup for the finished block (a copy-button wrapper, a mermaid `pre`).
 - `toc=True` prepends a `<nav class="toc">` of the headings.
 - `auto_ids` (on by default) derives pandoc-style ids for headings without one, deduplicated per export — pass `auto_ids=False` for fragments sharing a page.
 - A `div` classed `details` lowers to a `<details>` element; a first-child heading becomes its `<summary>` (id kept, excluded from the TOC and numbering). Non-HTML exporters degrade it to a bold label line; the class word is reserved by [the dialect's converter obligations](docs/DIALECT.md#converter-obligations).
@@ -445,4 +452,3 @@ maturin develop && pytest -q
 ```
 
 The spec-conformance suite is `tests/test_conformance.py`: it renders the fixtures under `tests/source/` and compares normalized HTML trees. Run just that file with `pytest tests/test_conformance.py -v` to see per-example ids.
-

pyproject.toml

@@ -29,6 +29,7 @@ Issues = "https://github.com/AnswerDotAI/mdhtml/issues"
 
 [project.optional-dependencies]
 fill = ["execnb>=0.3.2"]
+hl = ["fastpylight>=0.1.6"]
 dev = ["fastship>=0.0.14", "maturin~=1.0", "pytest", "execnb>=0.3.2", "fastpylight>=0.1.6", "math-core~=0.7.0"]
 
 [project.entry-points.fastaudit_safe_native]

python/mdhtml/__init__.py

@@ -6,7 +6,7 @@
 from fast5ever import Element, Node, parse_fragment as mdhtml2dom
 from ._native import (blocks as _blocks, edit_nodes as _edit_nodes, highlight_md, mdhtml2md,
     md2mdhtml as _md2mdhtml, wiki2mdhtml as _wiki2mdhtml)
-from .export import dialect_css, math_js, meta_table, theme_css, themes, mdhtml2html
+from .export import dialect_css, math_js, meta_table, mdhtml2html
 from .md import _normalize_offsets, md2gfm
 from .fill import frontmatter_data, instantiate, fill_md, tokens
 from .typst import mdhtml2pdf, mdhtml2typst

python/mdhtml/export.py

@@ -6,10 +6,10 @@
 
 from fast5ever import Element
 from ._native import HeadingNums, Resolver as _Resolver, group_plan, anchors, ref_tokens, ref_variant, target_kind
-from ._native import REFTYPES, SCHEMES, decode_raw as _decode_raw, dialect_css, export_html as _export_html, math_js as _math_js, theme_css, themes
+from ._native import REFTYPES, SCHEMES, decode_raw as _decode_raw, dialect_css, export_html as _export_html, math_js as _math_js
 
 
-__all__ = ["SCHEMES", "REFTYPES", "ref_tokens", "ref_variant", "target_kind", "anchors", "decode_raw", "tmpl_node", "group_plan", "HeadingNums", "Resolver", "mdhtml2html", "math_js", "meta_table", "dialect_css", "theme_css", "themes"]
+__all__ = ["SCHEMES", "REFTYPES", "ref_tokens", "ref_variant", "target_kind", "anchors", "decode_raw", "tmpl_node", "group_plan", "HeadingNums", "Resolver", "mdhtml2html", "math_js", "meta_table", "dialect_css"]
 
 
 _HEADS = {"h1", "h2", "h3", "h4", "h5", "h6"}
@@ -62,6 +62,24 @@ def _els(el): return [c for c in el.children if isinstance(c, Element)]
 def _text(el): return " ".join(el.to_text().split())
 
 
+def _fastpylight():
+    "The fastpylight module, imported lazily: highlighting and themes install via `pip install 'mdhtml[hl]'`."
+    try: import fastpylight
+    except ImportError as e: raise ImportError("highlighting and themes need fastpylight: pip install 'mdhtml[hl]'") from e
+    return fastpylight
+
+
+def _hl_fn(hl):
+    "A per-block highlighter callback wrapping fastpylight, or None when it isn't installed (the exporter then warns)."
+    try: fp = _fastpylight()
+    except ImportError: return None
+    f = fp.highlight_spans if hl == "spans" else fp.highlight
+    def go(text, lang):
+        try: return f(text, lang)
+        except ValueError: return None  # unknown language: the block stays plain
+    return go
+
+
 def mdhtml2html(src, dest=None, reftypes: dict | None = None, number_headings=None, hl: str | None = "spans", auto_ids: bool = True,
     toc: bool = False, refs: str = "resolve", id_prefix: str = "", fn_salt: str = "", hl_lang=None, code_wrap=None) -> Html:
     """Lower MDHTML (a string or DocumentFragment; never mutated) to finished HTML: cross-references
@@ -84,10 +102,13 @@ def mdhtml2html(src, dest=None, reftypes: dict | None = None, number_headings=No
     distinct. Per code block, `hl_lang(text, lang)` may
     return a corrected language (`lang` is None for a bare fence), and `code_wrap(html, lang, text)`
     may return replacement markup for the highlighted block (None keeps it; `text` is unescaped).
+    Highlighting comes from the optional fastpylight package (`pip install 'mdhtml[hl]'`);
+    without it, code blocks render plain and a warning reports it.
     Returns an `Html` str carrying `.warnings`; `dest` also writes it to a file."""
     if refs not in ("resolve", "ids", "lenient"): raise ValueError(f"unknown refs mode {refs!r}")
     if not isinstance(src, str): src = src.to_html()
-    out, warnings = _export_html(src, reftypes, number_headings, hl, toc, refs, id_prefix, fn_salt, hl_lang, code_wrap, auto_ids)
+    hl_fn = None if hl is None else _hl_fn(hl)
+    out, warnings = _export_html(src, reftypes, number_headings, hl, toc, refs, id_prefix, fn_salt, hl_lang, code_wrap, hl_fn, auto_ids)
     res = Html(out, warnings)
     if dest is not None: Path(dest).write_text(res, encoding="utf-8")
     return res

python/mdhtml/md2html.py

@@ -9,7 +9,8 @@
 from fastcore.meta import delegates
 from fastcore.script import call_parse
 
-from . import dialect_css, math_js, meta_table, mdhtml2dom, theme_css, mdhtml2html, md2mdhtml
+from . import dialect_css, math_js, meta_table, mdhtml2dom, mdhtml2html, md2mdhtml
+from .export import _fastpylight
 from .mustache import MUSTACHE, mustache_pill
 from fast5ever import Element
 from ._cli import parse_args, read_src
@@ -53,7 +54,7 @@ def _code_wrap(html, lang, text):
 
 def page(body, title="mdhtml", theme="vscode_light", dark_theme="vscode_dark", preview=False, math=True, head=()):
     "A standalone HTML page around an exported `body` fragment, with the assets its features need; `head` chunks (`<style>`, `<script>`, `<link>`, ...) are inserted verbatim at the end of `<head>`"
-    hl = "".join(f"@media (prefers-color-scheme: {m}) {{\n{theme_css(t)}}}\n" for m, t in (("light", theme), ("dark", dark_theme)))
+    hl = "".join(f"@media (prefers-color-scheme: {m}) {{\n{_fastpylight().theme_css(t)}}}\n" for m, t in (("light", theme), ("dark", dark_theme)))
     css = PAGE_CSS + dialect_css(preview=preview) + hl
     katex = (f'<link rel="stylesheet" href="{KATEX}/katex.min.css">\n'
         f'<script type="module">import katex from "{KATEX}/katex.mjs";\n{math_js()}</script>') if math else ""
@@ -87,7 +88,7 @@ def main(
     number_headings: NumMode = None,  # Heading numbering scheme
     toc: bool = False,  # Prepend a table of contents
     hl: HlMode = HlMode.spans,  # Code highlighting: classed spans, the Highlight API, or off
-    theme: str = "vscode_light",  # Code colors in light mode: any name from `mdhtml.themes()`
+    theme: str = "vscode_light",  # Code colors in light mode: any name from `fastpylight.themes()`
     dark_theme: str = "vscode_dark",  # Code colors in dark mode
     templates: bool = True,  # Show mustache `{{tokens}}` as styled pills
     auto_ids: bool = True,  # Derive ids for headings

python/mdhtml/viewmd.py

@@ -10,7 +10,8 @@
 from aidialog.dialog import dlg2md
 from aidialog.ipynb import read_ipynb
 
-from . import DASHES, replacements, theme_css, mdhtml2html, md2mdhtml
+from . import DASHES, replacements, mdhtml2html, md2mdhtml
+from .export import _fastpylight
 from .mustache import MUSTACHE, mustache_pill
 from ._cli import parse_args, read_src
 from . import meta_table
@@ -33,7 +34,7 @@ def _copy_wrap(html, lang, text):
 
 def assets():
     "The viewer's stylesheet, controls, and script, as one blob appended to the page body"
-    hl = "".join(theme_css(t, f'[data-hl="{t}"] pre code') for _, lt, dk in THEMES for t in (lt, dk))
+    hl = "".join(_fastpylight().theme_css(t, f'[data-hl="{t}"] pre code') for _, lt, dk in THEMES for t in (lt, dk))
     opts = "".join(f'<option value="{lbl}">{lbl}</option>' for lbl, _, _ in THEMES)
     return (f"<style>{VIEW_CSS}{hl}</style>{CONTROLS.replace('__OPTS__', opts)}"
         f"<script>{VIEW_JS.replace('__THEMES__', json.dumps(THEMES))}</script>")

src/chunk.rs

@@ -1,8 +1,8 @@
 //! Fast hierarchical Markdown chunking, following the existing Wikipedia
 //! pipeline's H2, H3, H4, then paragraph passes.
 
-use crate::{Block, Document, Options, render_md};
 use crate::block::parse_block_boundaries;
+use crate::{Block, Document, Options, render_md};
 use std::ops::Range;
 
 #[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -244,8 +244,7 @@ fn range_words(range: &StructuralRange, word_totals: &[usize]) -> usize {
 fn structural_sections(range: &StructuralRange, level: Option<u8>, starts: &[ChunkStart]) -> Vec<StructuralRange> {
     let mut cuts = vec![range.blocks.start];
     cuts.extend((range.blocks.start + 1..range.blocks.end).filter(|&i| level.is_none_or(|level| starts[i] == ChunkStart::Heading(level))));
-    cuts
-        .iter()
+    cuts.iter()
         .enumerate()
         .map(|(i, &start)| StructuralRange {
             blocks: start..cuts.get(i + 1).copied().unwrap_or(range.blocks.end),

src/export_html.rs

@@ -36,9 +36,13 @@ pub enum RefsMode {
 }
 
 /// Per-code-block hooks. Errors short-circuit the export; the pyo3 bridge
-/// stores the original Python exception and re-raises it.
+/// stores the original Python exception and re-raises it. `HlHook` receives
+/// `(code, lang, mode)` and returns highlighted markup: inner spans or a whole
+/// `<pre><code>` block for `Spans`, `<hl-code toks=...>` component markup for
+/// `Api`. `None` leaves the block plain.
 pub type HlLangHook<'a> = &'a (dyn Fn(&str, Option<&str>) -> Result<Option<String>, String> + Send + Sync);
 pub type CodeWrapHook<'a> = &'a (dyn Fn(&str, Option<&str>, &str) -> Result<Option<String>, String> + Send + Sync);
+pub type HlHook<'a> = &'a (dyn Fn(&str, &str, HlMode) -> Result<Option<String>, String> + Send + Sync);
 
 #[derive(Default)]
 pub struct HtmlExportOptions<'a> {
@@ -51,6 +55,7 @@ pub struct HtmlExportOptions<'a> {
     pub fn_salt: String,
     pub hl_lang: Option<HlLangHook<'a>>,
     pub code_wrap: Option<CodeWrapHook<'a>>,
+    pub hl_fn: Option<HlHook<'a>>,
     pub auto_ids: bool,
 }
 
@@ -207,7 +212,7 @@ impl Exporter {
                 self.table_width(t);
             }
         }
-        let hl_on = opts.hl.is_some() && cfg!(feature = "hl");
+        let hl_on = opts.hl.is_some();
         if hl_on || opts.hl_lang.is_some() || opts.code_wrap.is_some() {
             for &pre in &els {
                 if ename(&self.dom, pre) == Some("pre") {
@@ -564,30 +569,47 @@ impl Exporter {
                 }
             }
         }
-        #[cfg_attr(not(feature = "hl"), allow(unused_mut))] // reassigned only in api mode
         let mut cur = pre;
-        #[cfg(feature = "hl")]
         if opts.hl.is_some()
             && let Some(l) = &lang
         {
             // The dialect highlights itself: `md` fences go through
-            // `highlight_md` (always span-shaped), everything else through
-            // fastpylight.
+            // `highlight_md` (always span-shaped); every other language goes
+            // to the `hl_fn` hook, when one is provided.
             let own = matches!(l.as_str(), "markdown" | "md");
-            let inner = if own {
+            if !own && opts.hl_fn.is_none() && !self.warnings.iter().any(|w| w.starts_with("highlighting requested")) {
+                self.warnings
+                    .push("highlighting requested but no highlighter provided: code blocks render plain (from Python, pip install 'mdhtml[hl]')".to_string());
+            }
+            let markup = if own {
                 Some(crate::highlight::highlight_md(&text, "hl-"))
             } else if opts.hl == Some(HlMode::Spans) {
-                fastpylight::highlighted_inner(&text, l, "hl-").ok()
+                match opts.hl_fn {
+                    Some(hook) => hook(&text, l, HlMode::Spans)?,
+                    None => None,
+                }
             } else {
                 None
             };
-            if let Some(inner) = inner {
-                let frag = parse_fragment(&inner, "body");
-                let imported = self.dom.import(&frag, DOCUMENT);
+            if let Some(markup) = markup {
+                // A hook may return inner spans or a whole `<pre><code>` block;
+                // either way our own wrapper elements and their attributes stay.
+                let frag = parse_fragment(&markup, "body");
+                let mut src = DOCUMENT;
+                if let [root] = el_children(&frag, DOCUMENT)[..]
+                    && ename(&frag, root) == Some("pre")
+                    && let Some(c) = el_children(&frag, root).into_iter().find(|&c| ename(&frag, c) == Some("code"))
+                {
+                    src = c;
+                }
                 self.dom.clear_children(code);
-                self.dom.append_child(code, imported).unwrap();
+                for child in frag.children(src).to_vec() {
+                    let imported = self.dom.import(&frag, child);
+                    self.dom.append_child(code, imported).unwrap();
+                }
             } else if opts.hl == Some(HlMode::Api)
-                && let Ok(markup) = fastpylight::highlight_component(&text, l)
+                && let Some(hook) = opts.hl_fn
+                && let Some(markup) = hook(&text, l, HlMode::Api)?
             {
                 let frag = parse_fragment(&markup, "body");
                 let root = el_children(&frag, DOCUMENT).first().copied();