BUILD_AS_WORKER: onmessage is installed too late under MODULARIZE, and is installed even on the main thread

[tovabar]

src/build_as_worker.js wraps everything in an IIFE that installs the worker's onmessage:

(() => {
  var messageBuffer = null, buffer = 0;

  function flushMessages() { ... }
  function messageResender() { ... }

  onmessage = (msg) => {
    // if main has not yet been called (mem init file, other async things), buffer messages
    if (!runtimeInitialized) {
      if (!messageBuffer) {
        messageBuffer = [];
        setTimeout(messageResender, 100);
      }
      messageBuffer.push(msg);
      return;
    }
    ...
  }
})();

Two problems follow from when and where that IIFE runs. They are independent, so I have described both here and am happy to split them.

1. The handler is installed even when the module is not running in a worker

The IIFE runs in whatever environment loads the module, and assigns onmessage unconditionally. A module linked with -sBUILD_AS_WORKER that also runs on the main thread therefore overwrites window.onmessage, taking over message delivery for the whole page.

That is not a quiet takeover. Any message the page receives from another source now reaches this handler, which either aborts:

var func = Module['_' + msg.data['funcName']];
if (!func) abort('invalid worker function to call: ' + msg.data['funcName']);

or throws a TypeError outright, if msg.data is not an object with a funcName (a plain postMessage('...') from an embedding page or another library, for instance).

A guard at the top of the IIFE seems like the right fix:

if (!ENVIRONMENT_IS_WORKER) return;

2. Under MODULARIZE, messages that arrive before the factory runs are dropped

With -sMODULARIZE the IIFE lives inside the module factory, which the application invokes whenever it is ready. A worker, however, starts receiving messages the moment it is created — so there is a window between "worker script began executing" and "factory was called" during which these messages arrive.

The existing buffering does not cover that window. messageBuffer only starts collecting from inside the handler this IIFE installs, so it protects the interval between the factory running and runtimeInitialized — not the earlier one.

The natural workaround is to install a small buffering onmessage in the worker bootstrap (or via --pre-js) that collects messages until the factory is called. That does not work either: this IIFE overwrites onmessage and starts from messageBuffer = null, so everything the earlier handler collected is discarded.

The net effect is that a MODULARIZE'd BUILD_AS_WORKER module silently loses any message sent before the application gets around to instantiating it — with no way for the application to bridge the gap.

What we ship locally is for the IIFE to adopt a pre-existing buffer if one is present:

var messageBuffer = typeof workerMessageBuffer != 'undefined' ? workerMessageBuffer : null, buffer = 0;
// Upstream only starts the resender from inside its own handler, so an adopted
// buffer would otherwise never be flushed.
if (messageBuffer) setTimeout(messageResender, 100);

The bootstrap then just pushes into workerMessageBuffer until the factory is instantiated, and the module drains it. This has been running in production for us for a while.

I am raising this as an issue rather than sending the patch straight away, because the second half proposes a convention (workerMessageBuffer) that emscripten does not currently define, and you may well prefer a different shape — an explicit Module option, or having build_as_worker.js emit its own pre-factory bootstrap. Happy to send a PR for whichever you would like, and #1 can go on its own regardless.

Related, in that they are all variations of "the generated worker's onmessage is not reachable by the application": #20192, #11962, #8854.

Version

emscripten 6.0.3; the same code is on main as of today.

[sbc100]

Its not too surprising to me that we there are issues with the BUILD_AS_WORKER setting as this doesn't have lot of testing or (IIUC) users.

I think we should certainly fix these two issues, but we should also look into increasing the level of testing for BUILD_AS_WORKER and/or considering for removal. Can you share how/why you are using -sBUILD_AS_WORKER in the first place? Is your use case not covered by Wasm Workers and/or Pthreads.

Regarding fixing these 2 issues specifically:

1. Can we also maybe also assert that generated program is always run on a worker? I don't think code build with -sBUILD_AS_WORKER should ever need to be loaded on the main thread, for example?

2. Can we perhaps make the MODULARIZE incompatible with BUILD_AS_WORKER. The benefits of MODULARIZE seem don't see very relevant under BUILD_AS_WORKER (unless I'm missing some use case?)

[tovabar]

Thanks for looking at this. Let me give the full picture, since it bears on both of your suggestions — including a correction to how I framed the first item.

Framing first. BUILD_AS_WORKER was not designed for main-thread loading, and you're right that it normally shouldn't be. Building one artifact that serves both roles is our own idea, and it doesn't fix a problem so much as improve things: one file instead of two, less total code to ship, simpler deployment. So item 1 in the description is better read as a feature request than a bug report. Item 2 is a genuine bug regardless, and I'll come back to it.

Our use case. We ship a WebAssembly SDK that renders with WebGL, embedded into host applications and pages we do not control. We link one artifact and load it two ways: on the main thread it renders into a canvas, so it cannot live in a worker; and the host also spins up workers from the same script URL to do expensive file parsing and format conversion off the render loop. Only a handful of functions are exported for that worker path.

Why not Wasm Workers or pthreads. Both require SharedArrayBuffer, so the embedding page has to be cross-origin isolated (COOP/COEP). We are a library inside somebody else's page — we cannot require the host to set those headers, and in practice many hosts either cannot or will not. BUILD_AS_WORKER with plain postMessage is the one option that works regardless of the host's header configuration, which is why we are still on it.

How much of this emscripten has already absorbed. We have historically carried three modifications to the generated JS to make the single-artifact pattern work, and two are already gone:

libeventloop.js — we used to strip the else postMessage(emscriptenMainLoopMessageId, '*') branch, because the main-thread path broke the worker. You have since fixed this upstream by adding an ENVIRONMENT_IS_WORKER guard. That is precisely the shape of fix item 1 is asking for.
A worker self-bootstrap appended to the output, so that loading the same file as a worker script instantiates the module and resolves the wasm binary from URL parameters. That one is ours and needs nothing from emscripten; it lives in --extern-post-js.
build_as_worker.js — still outstanding, and the subject of this issue.
So the pattern is less exotic than it may look: emscripten has already converged on ENVIRONMENT_IS_WORKER branching for the same class of problem, in the same generated module.

On your two suggestions.

An assert that the program only ever runs on a worker would fire on our main rendering path, and would move in the opposite direction from the libeventloop.js fix. What we need is for the BUILD_AS_WORKER handler to leave onmessage alone when it is not in a worker — a one-line no-op. Shipping two builds would double a large artifact for the sake of one message handler, which is the very thing the single-artifact pattern exists to avoid.

Making MODULARIZE incompatible with BUILD_AS_WORKER would break every configuration we ship. MODULARIZE plus EXPORT_NAME is what lets the embedding application decide when to instantiate the module, and it is what makes the "create the worker from the same script URL" pattern work at all.

And item 2 from the description stands independently of all of the above: messages arriving between worker creation and factory instantiation are lost in a plain worker-only MODULARIZE build, with no main-thread involvement.