Hello all,
I'm Mat, author of AnyBlox. In our paper we needed to compile Vortex for Wasm for decoding, which I did by vendoring the crate in and brutally pruning everything that wouldn't compile (like I/O, tokio dependencies, etc.). I recently wanted to see if it would be possible to make the project less hacky and revert to using latest versions of all vendored crates.
The situation is much better than roughly a year ago, as Vortex compiles for wasm32-unknown-unknown. However, it doesn't actually work. This is due to the dependency on getrandom, which simply does not work on bare-metal Wasm. I can see there is a workaround in place:
|
[workspace.dependencies.getrandom_v03] |
|
features = ["wasm_js"] |
|
package = "getrandom" |
|
version = "0.4.0" |
|
[target.wasm32-unknown-unknown] |
|
rustflags = ['--cfg', 'getrandom_backend="wasm_js"', '-C', 'target-feature=+atomics'] |
This, however, requires wasm-bindgen to be used to run the Wasm library, which is not a general solution -- in particular it does not work for AnyBlox, which JIT-compiles Wasm to x86 with no runtime library. If you try to run a decoder using mainline Vortex via wasmtime without WASI or bindgen you'll get:
unknown import: `__wbindgen_placeholder__::__wbindgen_describe` has not been defined
My assumption is that actual randomness is not actually needed for reading and decoding Vortex files (correct me if I'm wrong). If that is the case then the workaround to this issue would be to use existing getrandom facilities to provide a custom backend that always fails (or returns a random number determined by a fair dice roll). Currently this is impossible, because the wasm_js feature forces bindgen into a weird path (already filed an issue there: rust-random/getrandom#840). An additional pain point is that this feature puts all of wasm-bindgen into the tree, which causes the resulting wasm file to be huge (over 20MB). This very negatively impacts JIT times for wasm->x86 later down the line and, in context of AnyBlox, unnecessarily bloats self-contained datasets.
It would be best if wasm_js was an optional feature that can be controlled from the top-level of Vortex. It is disabled by default in getrandom itself specifically to not force users into the JS backend. It's explicitly discouraged for libraries in their README:
WARNING: We strongly recommend against enabling this feature in libraries (except for tests) since it is known to break non-Web WASM builds and further since the usage of wasm-bindgen causes significant bloat to Cargo.lock (on all targets).
I agree with that as well -- if someone is compiling to wasm32-unknown-unknown instead of explicitly targeting emscripten or WASI, then they are probably clever enough to deal with the randomness issue themselves (by explicitly pulling in getrandom and enabling wasm_js or providing a custom backend).
Hello all,
I'm Mat, author of AnyBlox. In our paper we needed to compile Vortex for Wasm for decoding, which I did by vendoring the crate in and brutally pruning everything that wouldn't compile (like I/O, tokio dependencies, etc.). I recently wanted to see if it would be possible to make the project less hacky and revert to using latest versions of all vendored crates.
The situation is much better than roughly a year ago, as Vortex compiles for
wasm32-unknown-unknown. However, it doesn't actually work. This is due to the dependency ongetrandom, which simply does not work on bare-metal Wasm. I can see there is a workaround in place:vortex/Cargo.toml
Lines 305 to 308 in a78897f
vortex/.cargo/config.toml
Lines 6 to 7 in a78897f
This, however, requires wasm-bindgen to be used to run the Wasm library, which is not a general solution -- in particular it does not work for AnyBlox, which JIT-compiles Wasm to x86 with no runtime library. If you try to run a decoder using mainline Vortex via
wasmtimewithout WASI or bindgen you'll get:My assumption is that actual randomness is not actually needed for reading and decoding Vortex files (correct me if I'm wrong). If that is the case then the workaround to this issue would be to use existing
getrandomfacilities to provide a custom backend that always fails (or returns a random number determined by a fair dice roll). Currently this is impossible, because thewasm_jsfeature forces bindgen into a weird path (already filed an issue there: rust-random/getrandom#840). An additional pain point is that this feature puts all of wasm-bindgen into the tree, which causes the resulting wasm file to be huge (over 20MB). This very negatively impacts JIT times for wasm->x86 later down the line and, in context of AnyBlox, unnecessarily bloats self-contained datasets.It would be best if
wasm_jswas an optional feature that can be controlled from the top-level of Vortex. It is disabled by default ingetrandomitself specifically to not force users into the JS backend. It's explicitly discouraged for libraries in their README:I agree with that as well -- if someone is compiling to
wasm32-unknown-unknowninstead of explicitly targeting emscripten or WASI, then they are probably clever enough to deal with the randomness issue themselves (by explicitly pulling ingetrandomand enablingwasm_jsor providing a custom backend).