[DNM] WebAssembly user-defined functions (experiment)

[DAlperin]

Do not merge. This is an experiment, up for discussion and to see what CI thinks. It is one large stack; if we pursue it, it would land as separate PRs per concern.

What this is

User-defined scalar functions compiled to WebAssembly:

\set module `base64 < udfs.wasm | tr -d '\n'`
CREATE FUNCTION jaro_winkler(a text, b text) RETURNS double precision
    LANGUAGE wasm STRICT
    USING BASE64 $1 \bind :module \g

SELECT jaro_winkler('martha', 'marhta');  -- 0.9611

Behind enable_wasm_functions, off by default and on in CI. The design doc is doc/developer/design/20260930_wasm_udfs.md.

Design in brief

• ABI. Guests implement the arrow-udf 3.0 ABI: a wasm32-wasip1 module with one export per signature, called with an Arrow record batch. Modules built with the arrow-udf crate's #[function] macro load unchanged, including vectorized guests that take whole Arrow columns through batch_fn.
• Determinism. Differential retracts by recomputing, so a function must give the same answer for the same input on every replica, every time. The host runs each batch in a fresh instance with fuel limits (no epochs), canonical NaNs, relaxed SIMD and threads disabled, and its own WASI preview 1 shim: seeded random_get, a clock frozen at the epoch, no filesystem or network. A row's result is defined as the result of calling the function on that row alone. A batch that traps or exhausts its limits is bisected until every row has its own outcome.
• Batching. One guest call per batch of up to 16K rows, not per row. The batched MFP evaluator precomputes every UDF call in a step over the live rows, then evaluates the step per row and replays the recorded results. So results, error precedence and short-circuiting in CASE, COALESCE, AND and OR are exactly what row-at-a-time evaluation gives. Batching happens in Mfp operators and inside persist_source's decode, where filter pushdown is kept. Join closures, Reduce key and value plans, and peeks still call one row at a time.
• Boundaries. mz_expr depends only on a WasmInvoker trait. wasmtime lives in the new mz-wasm-udf crate, linked by clusterd and the mz CLI, not environmentd. Calls are never constant-folded, so environmentd validates modules but never runs them.
• Distribution. A new compute command, InstallWasmModule, ships module bytes to replicas before any dataflow or peek that uses them. It is kept across history compaction and replayed on reconnect.
• SQL. CREATE FUNCTION ... LANGUAGE wasm [STRICT] USING BASE64 { '...' | $n } [WITH (EXPORT, FUEL, MEMORY)], DROP FUNCTION, and SHOW [REDACTED] CREATE FUNCTION. Statement text is capped at 1 MiB, and arrow-udf modules exceed that once base64-encoded, so a module is usually sent as a bind parameter. Planning substitutes it into create_sql. Statement logging records CREATE FUNCTION parameters as <REDACTED>.
• CLI. mz udf check | call | sql | create | pull | replay. These validate a module, call it locally with the same runtime and limits a cluster uses, check that results do not depend on batching, and debug failing rows from a region.
• Columnar evaluation. The design doc's future work section describes how this folds into general columnar scalar evaluation. A UDF call becomes one more function with a batch form, and the replay evaluator goes away.

Commits

1. design: the design doc.
2. wasm-udf: a guest fixture module (16 functions exercising types, errors, traps, fuel, memory, state and WASI), checked in gzipped with its build script.
3. wasm-udf-abi: signatures, type mapping and module validation.
4. expr: the VariadicFunc::Wasm variant, the invoker trait, and the batched MFP evaluator.
5. wasm-udf: the deterministic runtime.
6. compute: module installation on replicas, and batching in Mfp operators.
7. sql: CREATE FUNCTION and DROP FUNCTION, catalog and adapter.
8. wasm-udf: vectorized guests and configurable batch sizes.
9. mz: the udf subcommands.
10. expr, storage: precompute-and-replay for exact per-row semantics, and batching in persist_source.
11. sql, mz: modules as bind parameters, and statement logging redaction.
12. design: how this fits into columnar scalar evaluation.

Tests

• mz-wasm-udf-abi unit tests: signature encoding, type mapping, module validation.
• mz-wasm-udf integration tests against the fixture: types, NULLs, errors, traps, fuel, memory, bisection, determinism of WASI imports, batch sizing, and batch_fn guests.
• mz-expr tests for the batched evaluator. Batched and per-row results match, including error precedence, filters before a call, and calls under If and AND.
• Parser datadriven cases for CREATE FUNCTION, including USING BASE64 $n.
• src/environmentd/tests/wasm_udf.rs end to end:
  • create, query and materialize functions;
  • retraction, restart and re-planning from the catalog;
  • DROP FUNCTION and its dependency checks;
  • validation errors and the feature flag;
  • a 40K-row materialized view through the batched persist_source path, with UDF errors in an untaken CASE branch that must not surface.

Known gaps

Table functions, aggregates, overloads, CREATE OR REPLACE FUNCTION, introspection for batch sizes and fuel, and batching in joins, Reduce and peeks. No security review of running guest code in clusterd yet. These are listed in the design doc's open questions and future work.

Generated with Claude Code