Skip to content

Replace convertJSArrayToNumberVector with direct WASM frame buffer#29

Open
kalwalt wants to merge 1 commit intomainfrom
feature-frame-buffer
Open

Replace convertJSArrayToNumberVector with direct WASM frame buffer#29
kalwalt wants to merge 1 commit intomainfrom
feature-frame-buffer

Conversation

@kalwalt
Copy link
Copy Markdown
Member

@kalwalt kalwalt commented Apr 19, 2026

Summary

  • Eliminates per-frame heap copy: convertJSArrayToNumberVector<uint8_t>() walked the JS typed array element-by-element and created a fresh std::vector on every frame. For a 1280×720 RGBA frame that's ~3.7 MB allocated and freed on every call.
  • New approach (Approccio 1): C++ allocates a persistent uint8_t* buffer once via initFrameBuffer(colorSpace), exposes its WASM linear-memory byte offset via getFrameBufferPtr(), and JS writes video data directly with HEAPU8.set() before calling processFrame.
  • Lazy-init safety net: processFrame in WebARKitController.js auto-calls initFrameBuffer on first use if not called explicitly, so existing callers keep working without changes.
  • All examples updated: worker_threejs.js, worker.js, worker_bufferCopy_threejs.js, worker_jsfeatNext.js all call initFrameBuffer explicitly with the correct colorspace after tracker load.

Changes

File Change
emscripten/WebARKitJS.h Add initFrameBuffer, getFrameBufferPtr, destructor, private buffer members; update processFrame signature
emscripten/WebARKitJS.cpp Implement buffer allocation, bppForColorSpace helper, null guard, destructor
emscripten/bindings.cpp Expose initFrameBuffer and getFrameBufferPtr to JS via embind
src/WebARKitController.js Add initFrameBuffer wrapper, lazy-init + HEAPU8.set() in processFrame
examples/worker_threejs.js Explicit initFrameBuffer(RGBA)
examples/worker.js Explicit initFrameBuffer(GRAY)
examples/worker_bufferCopy_threejs.js Explicit initFrameBuffer(RGBA)
examples/worker_jsfeatNext.js Explicit initFrameBuffer(GRAY)
build/, dist/ Rebuilt artifacts

Pitfalls addressed

  • Destructor frees the buffer to avoid WASM heap leak on teardown
  • HEAPU8 is read fresh each frame (not cached) to survive ALLOW_MEMORY_GROWTH heap relocations
  • initFrameBuffer colorspace must match processFrame colorspace — all examples verified and corrected (found and fixed a GRAY/RGBA mismatch in worker_jsfeatNext.js)
  • initTrackerGray intentionally left unchanged (one-time call, copy overhead negligible)

Test plan

  • npm run build-docker — compiled cleanly, no new warnings
  • npm run build-es6 — webpack bundled successfully
  • Open examples/teblid_example.html or threejs_worker_ES6.js in browser and verify marker tracking still works

🤖 Generated with Claude Code

Allocate a persistent uint8_t* buffer in WASM linear memory once via
initFrameBuffer(colorSpace), expose its byte offset via getFrameBufferPtr(),
and write each video frame directly with HEAPU8.set() on the JS side.
This eliminates the per-frame heap allocation and element-by-element copy
that convertJSArrayToNumberVector<uint8_t> performed on every processFrame call.

- emscripten/WebARKitJS.h: add initFrameBuffer, getFrameBufferPtr, destructor,
  m_frameBuffer/m_frameBufferSize private members; update processFrame signature
- emscripten/WebARKitJS.cpp: implement buffer allocation, bppForColorSpace helper,
  null guard in processFrame, destructor frees buffer
- emscripten/bindings.cpp: expose initFrameBuffer and getFrameBufferPtr to JS
- src/WebARKitController.js: add initFrameBuffer wrapper, lazy-init in processFrame,
  HEAPU8.set() direct write instead of passing JS array to WASM
- examples/worker_threejs.js: explicit initFrameBuffer(RGBA) after tracker load
- examples/worker.js: explicit initFrameBuffer(GRAY)
- examples/worker_bufferCopy_threejs.js: explicit initFrameBuffer(RGBA)
- examples/worker_jsfeatNext.js: explicit initFrameBuffer(GRAY)
- build/dist: rebuilt artifacts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@kalwalt kalwalt self-assigned this Apr 20, 2026
@kalwalt kalwalt added enhancement New feature or request javascript Emscripten / C / C++ all about Emscripten labels Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Emscripten / C / C++ all about Emscripten enhancement New feature or request javascript

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant