Wasmi Benchmarking Suite

This includes execution and compilation benchmarks for the Wasmi interpreter and other Wasm runtimes.

Runtimes

The following Wasm runtimes and configurations are included.

Runtime	Kind	Configurations	Description
wasmi-v0.31	Interpreter		The old version of the Wasmi Wasm stack-machine interpreter.
wasmi-v1	Interpreter	eager eager.unchecked lazy lazy.unchecked lazy-translation	The Wasmi v1 interpreter.
wasmi-v2	Interpreter	eager eager.unchecked lazy lazy.unchecked lazy-translation	The new and experimental Wasmi v2 interpreter with its various optimizations.
wasmtime.cranelift	Optimizing JIT		Wasmtime's optimizing JIT backend for the fastest performance.
wasmtime.winch	Baseline JIT		Wasmtime's baseline JIT for faster startup performance.
wasmtime.pulley	Interpreter		Wasmtime's portable interpreter backend.
wasmer.cranelift	Optimizing JIT		Wasmer's optimizing JIT backend based on Cranelift.
wasmer.singlepass	Baseline JIT		Wasmer's baseline JIT for faster startup performance.
v8	Multi-Tier JIT		Google's high performance JS runtime.
wasm3	Interpreter	eager lazy	A fast and well-established Wasm interpreter.
stitch	Interpreter		Experimental and very fast zero-dependencies Wasm interpreter.
wamr	Interpreter		The WebAssembly Micro Runtime (WAMR) fast interpreter.
tinywasm	Interpreter		Wasm interpreter written in pure safe Rust.
toywasm	Interpreter		Feature-rich in-place WebAssembly interpreter with low memory usage.
spacewasm	Interpreter		NASA JPL's no_std Wasm 1.0 (MVP) interpreter for on-board spacecraft use.
submilli-wasm	Interpreter		Wasm 3.0 interpreter with a Wasmtime-compatible API, built for fast compilation and single-execution workloads.
wasmedge	Interpreter		The interpreter of the WasmEdge Wasm runtime.
fizzy	Interpreter		A fast, deterministic, and pedantic WebAssembly interpreter.
dlr-wasm-interpreter	Interpreter		An in-place Wasm interpreter by the DLR-FT for use in aviation.
silverfir-nano.jit	Optimizing JIT		The optimizing JIT of the compact WebAssembly 3.0 Silverfir-nano runtime.
silverfir-nano.interpreter	Template Interpreter		Silverfir-nano's folded stack machine interpreter.
wasmz	Interpreter		A fast WebAssembly interpreter written in Zig.

Configuration Explanation

• eager: All function bodies are compiled and validated immediately.
• eager.unchecked: Function bodies are compiled eagerly but Wasm validation is skipped.
• lazy: Function bodies are only compiled and validated upon first use.
• lazy.unchecked: Function bodies are only compiled upon first use and Wasm validation is skipped.
• lazy-translation: Function bodies are lazily compiled but eagerly validated.

Note: by default runtimes compile and validate function bodies eagerly.

Usage

Run all benchmarks via:

cargo bench

Note: compilation might take some minutes since we are compiling a lot of Wasm runtimes with very high optimization settings.

Filter benchmarks via

• startup: for startup (compile + instantiate) benchmarks.
• execute: for execution benchmarks.
• The runtime ID, e.g. wasmi-v0.31 or wasm3.
• The runtime configuration on top of the runtime ID, e.g. wasmi-v0.32.lazy.
• Single test names, e.g. counter (execute) or ffmpeg (startup)

Examples

Run all runtimes on the counter execution benchmark test case:

cargo bench execute/counter

Run all Wasm3 test cases with its eager compilation configuration:

cargo bench wasm3.eager

Test Cases

Execution Benchmarks

Tests the execution performance of the Wasm runtime, prefixed by execute/.

Test Case	Description
counter-local	Simple loop that counts a single local down from some number.
counter-param	Simple loop that counts down from some number via a control parameter.
counter-global	Simple loop that counts a global down from some number.
fibonacci-rec	Recursive fibonacci calculation. Call-intense workload.
fibonacci-iter	Iterative fibonacci calculation. Compute intense workload.
fibonacci-tail	Tail-call based fibonacci calculation.
sort	Executes Rust's standard sort_unstable on integers.
sort-dyn	Executes Rust's standard sort_unstable on integers with dynamically called comparators.
prime_sieve	Executes a Rust sieve of eratosthenes implementation.
matrix-mul	Naive matrix multiplication implementation. Makes heavy use of linear memory and floats.
nbody	N-body physics simulation.
argon2	Password hashing library. Compute- and memory intense workload.
tiny-keccak	Tiny Rust implementation of Keccak crptography hashing.
mandelbrot	Classic Rust mandelbrot implementation.
spectralnorm	Computes the eigenvalue using the power method.
compression	Compresses some input using the miniz_oxide crate.
word-count	Count unqiue words in a string input via hash table inserts and look-ups.
json-parse	Parses a JSON file using serde_json.
reverse-complement	Converts a DNA sequence into its reverse, complement.
regex-redux	Match DNA 8-mers and substitute magic patterns.
bulk-ops	Tests performance of memory.{copy,fill} from the Wasm bulk-memory-operations proposal.

Startup Benchmarks

Tests the startup performance of the Wasm runtime, prefixed by startup/.

Test Case	Description
bz2	Medium-sized compression library with huge function bodies. (WASI required)
pulldown-cmark	Medium-sized markdown renderer. (WASI required)
spidermonkey	The firefox Javascript execution engine. (large, WASI required)
ffmpeg	Huge multimedia library. (WASI required)
coremark	CoreMark benchmarking compilation. (kinda small, no WASI)
argon2	Password hashing library. (small, no WASI)
erc20	ink! based ERC-20 implementation. (tiny, no WASI)

Running Benchmarks & Rendering Plots

In order to run the benchmarks and simultaneously plot diagrams of their results use the following command:

cargo criterion --bench criterion --message-format=json | cargo run --bin plot

This generates plots in the target/wasmi-benchmarks folder for all the benchmark groups. In order to use this you may need to install cargo-criterion via cargo install cargo-criterion.

Next to the per test case folders, a geomean-execute.svg and a geomean-startup.svg plot summarize each category: every runtime as the geometric mean of its per test case ratios against a theoretical optimal runtime that is the fastest one for each test case.

In case you want to collect data first and plot later you can also instead store the benchmark results into a file and use the file to plot the data later:

cargo criterion --bench criterion --message-format=json > results.json
cat results.json | cargo run --bin plot

Rendered plots are hosted at wasmi-benchmarks-plots.

Plots: Geomean

The following rendered plots show the geomean across all execute and startup benchmarks for all Wasm runtimes supported by wasmi-benchmarks respectively.

Coremark

This benchmark suite also contains a Coremark test which can be run via

cargo run --profile bench

This will run Coremark using all available Wasm VMs and print their Coremark scores to the console.

Runtime & Benchmark Support

Not every Wasm runtime can run every benchmark test case: some lack support for Wasm proposals used by a test case, and some fail to instantiate certain modules. The matrices below show which execute and startup test cases each runtime supports. A ✅ means the runtime runs the test case, a ❌ means it does not. These are derived directly from each runtime's can_run implementation (under runtimes/*/lib.rs); runtimes without a can_run run every test case.

Execution Support

Test Case	wasmi v0.31	wasmi v0.32	wasmi v1	wasmi v2	wasmtime cranelift	wasmtime winch	wasmtime pulley	wasmer cranelift	wasmer singlepass	v8	wasm3	stitch	wamr	tinywasm	toywasm	spacewasm	wasmedge	fizzy	dlr-wasm-interpreter	silverfir-nano jit	silverfir-nano interpreter	submilli-wasm	wasmz
counter-local	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
counter-param	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	✅	❌	✅	✅	✅	✅	✅
counter-global	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
fibonacci-rec	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
fibonacci-iter	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
fibonacci-tail	✅	✅	✅	✅	✅	❌	✅	❌	❌	✅	⚠️	❌	✅	✅	✅	❌	✅	❌	❌	✅	✅	✅	✅
sort	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
sort-dyn	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
prime_sieve	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
matrix-mul	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
nbody	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
argon2	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
tiny-keccak	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
mandelbrot	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
spectralnorm	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
compression	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
word-count	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
json-parse	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
reverse-complement	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
regex-redux	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
bulk-ops	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	✅	❌	✅	✅	✅	✅	✅

⚠️: Wasm3 requires a C compiler with musttail support when compiling its sources.

Startup Support

Test Case	wasmi v0.31	wasmi v0.32	wasmi v1	wasmi v2	wasmtime cranelift	wasmtime winch	wasmtime pulley	wasmer cranelift	wasmer singlepass	v8	wasm3	stitch	wamr	tinywasm	toywasm	spacewasm	wasmedge	fizzy	dlr-wasm-interpreter	silverfir-nano jit	silverfir-nano interpreter	submilli-wasm	wasmz
bz2	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
pulldown-cmark	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
spidermonkey	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
ffmpeg	✅	✅	✅	✅	⚠️	✅	⚠️	⚠️	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	⚠️	✅	✅	✅
coremark	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
argon2	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅
erc20	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅

• Note: wasmtime.winch only runs on x86_64 and aarch64; this matrix assumes such a host. On other architectures it supports no test cases.
• Note: Cells marked with ⚠️ may take very long to complete.

License

Licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.