Skip to content

feat: let a worker declare the process unable to serve - #36

Merged
vinismarques merged 2 commits into
mainfrom
feat/fatal-worker-error
Oct 1, 2026
Merged

vinismarques merged 2 commits into
mainfrom
feat/fatal-worker-error

Conversation

@vinismarques

@vinismarques vinismarques commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

A worker can now declare that its process can no longer serve by raising FatalWorkerError, alongside the existing ValidationError. The server answers that request with 500 and the full error chain. After that, /health and /inference answer 503, and the process sends itself SIGTERM so the orchestrator restarts it.

Why: some failures leave a process unusable without making it look unhealthy. A typical case is a sticky CUDA error such as an illegal memory access: once it happens, every later CUDA call in that process fails the same way. /health keeps passing, so jobs keep being routed to a process that can only fail them. The existing consecutive-error breaker eventually catches this, but only after several more failed jobs. On a GPU shared between processes, the stuck process can also block other clients until it exits.

Ownership: this package owns the mechanism: health, refusing requests, exiting. Deciding which errors are fatal is left to the worker.

Behaviour

  • FatalWorkerError has its own exception handler, so it doesn't feed the consecutive-500 counter. Ordinary failures and ValidationError behave as before.
  • uvicorn shuts down gracefully, so the failing request still gets its response. Checked against a real maestro-server: the 500 was delivered, the listener closed, and the process exited with code 143.
  • Once a fatal error is recorded, the process never goes back to healthy.

Testing

  • tests/test_serve.py:
    • A fatal error fails the request, switches /health to 503, refuses later requests without calling the worker, and terminates the process exactly once.
    • Ordinary failures and ValidationError leave the process serving.
  • uv run pytest, ruff and ty pass.

Bumps the version to 5.2.0 for the new public API.

A sticky CUDA error kept /health passing while every later job failed, so FatalWorkerError now fails the request, reports 503, and exits for a restart.
@vinismarques
vinismarques merged commit e6f8045 into main Oct 1, 2026
1 check passed
@vinismarques
vinismarques deleted the feat/fatal-worker-error branch October 1, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant