-
Notifications
You must be signed in to change notification settings - Fork 2
Open
Labels
bugSomething isn't workingSomething isn't workingdogfoodFound during dogfoodingFound during dogfooding
Description
Found during dogfooding v3.1.2
Severity: Medium
Command: node scripts/benchmark.js, node scripts/embedding-benchmark.js
Reproduction
node scripts/benchmark.js
# Segfaults (exit 139) during 3rd WASM 1-file rebuild iteration
# Native benchmark results are never collected
node scripts/embedding-benchmark.js
# Crashes (exit 132, illegal instruction) on nomic-v1.5 model loading
# Results from completed models (minilm, jina-small, jina-base, nomic) are lostExpected behavior
- Each engine (WASM/native) should be wrapped in try/catch in
benchmark.jsso a crash in one doesn't prevent the other from running - Each model should be wrapped in try/catch in
embedding-benchmark.jsso one model's failure doesn't lose all results - Partial results should be reported with the failed engine/model marked as "crashed"
Actual behavior
benchmark.jsline 182:await benchmarkEngine('wasm')has no error handling — WASM segfault kills process before native runsembedding-benchmark.js: model loop has no try/catch — illegal instruction on nomic-v1.5 loses all 4 previously completed model results
Root cause
Missing try/catch isolation around engine and model benchmark loops.
Suggested fix
- In
benchmark.js, wrap eachbenchmarkEngine()call in try/catch, storenullfor the crashed engine, and continue - In
embedding-benchmark.js, wrap eachbenchmarkModel()call in try/catch, record the error in the results JSON, and continue to the next model - Consider running each engine/model in a child process (
fork()) to isolate segfaults (which can't be caught by try/catch)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingdogfoodFound during dogfoodingFound during dogfooding