fix(rlix-scheduler): warn on GENERATION request without demand signal#34
Open
JunzheJoe wants to merge 1 commit into
Open
fix(rlix-scheduler): warn on GENERATION request without demand signal#34JunzheJoe wants to merge 1 commit into
JunzheJoe wants to merge 1 commit into
Conversation
The gap-ratio planner skips a pipeline whose rollout_open_pipelines entry has no positive step_target_estimate AND no stored progress (the bootstrap-estimate branch in planner.plan_generation_gap_ratio). Such a request is never signaled and its caller blocks forever in request_gpus — silently. The hazard was documented only in MilesPipeline's docstring. Warn at enqueue time (request_gpus + notify_release_then_request_gpus) via a shared helper. Warn-only, no behavior change: some backends legitimately request first and report progress later, so rejecting outright could break them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Review finding S12-EST-NONE-HANG (F1-F12 code review, scheduler deep-dive).
request_gpus(GENERATION, step_target_estimate=None)is accepted and writesNoneinto the durablerollout_open_pipelinesregistry. On the planner side (planner.plan_generation_gap_ratio, bootstrap-estimate branch), a pipeline with no positive estimate and no stored progress is skipped entirely — its pending request is never signaled, and the caller blocks forever insiderequest_gpus(which waits without timeout by design). No log line points at the cause; today the hazard is documented only inMilesPipeline._request_cluster_gpus's docstring.This adds a
logger.warningat both enqueue sites (request_gpusandnotify_release_then_request_gpus) via a shared_warn_gen_request_without_demand_lockedhelper, naming the exact skip condition and the likely outcome.Warn-only, no behavior change: some backends may legitimately enqueue first and publish progress later, so rejecting outright could break request-then-report flows. If we later decide to hard-reject, this helper is the single place to do it.
Tests
tests/test_scheduler_gen_request_warning.py— warn / no-warn matrix:Full suite: 69 passed, 3 skipped.