Repository navigation
feat: jobs survive restarts across the network (worker drain + gateway persistence); v0.2.92 - #193
Merged
Merged
Conversation
Anthony: "jobs should be resumable on restarts across the nodes".
- Workers: songs, images and videos hold a c0mpute_update::LongJob guard;
an auto-update downloads and swaps the binary but re-execs only once no
long job runs (max 30 min, so a stuck job cannot pin an old binary).
Before, every release killed whatever a node was rendering.
- Gateway: tracked requests (x-request-id) are written to
<data_dir>/gateway-jobs until they finish and their results to
<data_dir>/gateway-results (30 min). After a restart the gateway re-runs
unfinished requests under the same id and serves finished ones from
disk, so GET /v1/result/{id} still delivers. Tracked jobs also hold a
LongJob guard, so the gateway itself drains before updating.
- Loopback tests keep router() in memory; only the served gateway persists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ThreatCrush Security Scan13 finding(s) HIGH/CRITICAL: 1 | MEDIUM: 8 | LOW: 4
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anthony asked that jobs be resumable across restarts on every node.
LongJobguard while they run. An auto-update still downloads and swaps the binary, but it re-executes only once no long job is running, with a 30-minute limit so a stuck job can't pin an old binary. Before this, every release killed whatever a node was rendering.x-request-id) are written to<data_dir>/gateway-jobsuntil they finish, and their results to<data_dir>/gateway-results(kept 30 minutes). After a restart, the gateway re-runs unfinished requests under the same id and serves finished ones from disk, soGET /v1/result/{id}keeps working. Tracked jobs also hold the guard, so the gateway drains before it updates too.router()stays in memory for the loopback tests. Only the served gateway persists to disk.Checks: clippy shows no new warnings.
c0mpute-corelib tests pass 372 of 372 (including a disk round-trip across a simulated restart) andc0mpute-updatepasses 10 of 10 (including the drain test).Bumps the version to v0.2.92, which also ships #191's video pin.
🤖 Generated with Claude Code