Repository navigation
perf(deploy): pre-pull the worker image on every GPU node - #373
Draft
hodaaaaaaaaaa wants to merge 1 commit into
Draft
hodaaaaaaaaaa wants to merge 1 commit into
hodaaaaaaaaaa wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A new worker on a node without the worker image pulls it first, several GB that take minutes, and a cold job pays it for its trainer and its sampler.
Change
The base bundle adds the
open-rl-worker-prepullDaemonSet, which pulls the worker image onto every GPU node withopenrl.io/enabled=trueat deploy time and only sleeps. The kustomize images transformer rewrites its image like every other OpenRL image, so a release that bumps the tag re-pulls on every node. The kind overlays drop it, since their workers pull from a cluster-local registry.Based on droot's prototype in #343.
Results
Measured on GKE with eight GSM8K GRPO jobs launched together (gemma-4-E2B and Qwen3-0.6B, LoRA and full fine-tuning, A100 80GB and L4), with the worker image removed from every node before each run, two runs each: