Worker framework investigation spike

[jwulf]

We want to transparently support asynchronous workers in user applications.

These workers should use the async surface to poll for work, so that they are non-blocking.

When a worker retrieves work, how this work is executed will affect the performance of the application (async/concurrency vs thread vs process)

We want to provide a sensible default, allow explicit override, and maybe even autoselect based on performance.

[jwulf]

@rwestera

Worker performance testing:

1. Create n process instances in a loop (n=100,1000,10_000)
2. Simulate some CPU bound work in the worker callback
3. Measure how fast the worker can complete the work using the three different strategies.

[jwulf]

@rwestera

1. Measure the memory use of the running application.
2. Create an I/O bound callback task, so that we can test CPU and I/O bound workloads.
3. Support multi-plexing on workload type (in addition to strategy and number of tasks).

Additionally:

Create an external tool call using sub_process.call to simulate calling an external CLI tool. We want to see if this crashes when multi-threaded vs async.

@jwulf

1. Implement max_concurrent_jobs control in the polling loop.

• Maintain an active_jobs count in the worker.
• Implement ActivatedJob class with methods for complete, fail, error, and ignore. These methods call the respective underlying SDK methods to accomplish this with the job, and have the job.job_key bound into them, so the user needs only pass in the "unknown" parameters to the method. They do not accept a job_key parameter. The ignore method decrements the active jobs count, but does nothing with the broker. This is a method that allows the job to be sent to another system and completed there.
• Pass the ActivatedJob to the worker callback.
• Force the worker callback to return the output of one of these methods (ensure completeness of worker handler logic).
• Track finalised job count through calls to these methods. Each method will decrement the active_jobs count of the worker.
• Modify jobs to activate based on this count.