fix(miles-pipeline): honor R11-F1 ledger contract on failed releases#33
Open
JunzheJoe wants to merge 1 commit into
Open
fix(miles-pipeline): honor R11-F1 ledger contract on failed releases#33JunzheJoe wants to merge 1 commit into
JunzheJoe wants to merge 1 commit into
Conversation
Two fixes in the same leak family as #14: 1. shutdown_hard cleared both ledger flags unconditionally even when a notify_release_gpus RPC failed, contradicting the documented R11-F1 contract ('shutdown_hard will retry the release if the flag stays True'). A second shutdown_hard/dispose call would skip the retry and leak the server-side allocation. Flags now flip per-cluster and only after a successful release. 2. Init-phase release failures (phase A step 7, phase B step 8) only logged a warning and fell through. On a pool sized for one role at a time, phase B's request_gpus (no timeout by design) would then block forever on GPUs this pipeline still holds. Both sites now raise; initialize_pipeline's except path runs shutdown_hard, which retries the release. Tests: behavioral shutdown_hard flag-retention/retry (ray stubbed, same pattern as test_scheduler_apply_plan_invariants) + AST fail-fast asserts (same pattern as test_miles_pipeline_after_training_cleanup).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two fixes in the same scheduler-ledger leak family as #14, found during the F1-F12 code review (findings F8-SHUTDOWN-FLAGS and F8-INIT-RELEASE-HANG).
1.
shutdown_hardcleared ledger flags even when the release RPC failed_notify_release_cluster_gpus's docstring documents the R11-F1 contract: "shutdown_hard will retry the release if the flag stays True". Butshutdown_harditself cleared both_actor_train_allocated/_actor_infer_allocatedunconditionally after the release loop — a transient RPC failure meant a secondshutdown_hard/disposecall skipped the retry and the scheduler ledger leaked the allocation (peer pipelines starve).Now: flags flip per-cluster, and only after a successful release; failures keep the flag set for retry.
2. Init-phase release failures fell through instead of failing fast
Phase A step 7 (release
actor_trainbefore requestingactor_infer) and phase B step 8 (release INITactor_inferbefore the GENERATION re-request) only logged a warning on failure and continued. On a pool sized for one role at a time (the exact scenario the P1-7 comment describes), the nextrequest_gpus— which blocks without timeout by design — would wait forever on GPUs this same pipeline still holds: a silent deadlock instead of an error.Now: both sites raise;
initialize_pipeline's except path runsshutdown_hard, which retries the release.Tests
tests/test_miles_pipeline_shutdown_ledger.pyshutdown_hardretries only the leaked cluster (ray stubbed via thesys.modulespattern fromtest_scheduler_apply_plan_invariants.py)test_miles_pipeline_after_training_cleanup.pyfrom fix(rlix): always resume generation after finalize #14)