Engine: asynchronous code in the worker's publish causes events to be lost

[josephjclark]

Inside the engine, we run job code in a worker thread. The worker thread sends events out to the main engine process through parentPort.postMessage

In production right now this is synchronous. When the message is posted the payload is processed and sent out of the child worker. This ensures that messages leave the worker_thread in the same order that they were emitted from the runtime.

In #1366 we use an async loop to read the size of a payload before sending it. This async loop is causing two problems:

1. It no longer guarantees that events will leave the worker thread in the same order as they're emitted, because of the async yield. If message a takes longer to process than message b, than message b might be piped through the port first.
2. Some messages don't get emitted at all because the worker thread is shut down while the events are being processed. When we complete the task and exit the run, I don't think there's anyway to know that messages are still processing and waiting to be emitted.

I don't think this is affecting production, but it's a possible cause of #1072. What I'm worried about is that a message emitted with a large-ish payload, but the worker thread gets killed before that message is completed.

I'd sort of expect node.js to handle that for me and not exit the worker until the message has finished posting, but it's possible that we're exiting too soon (because of code or config in the engine).