[droot]

Faster cold starts, safe weight updates on small GPUs, and a seat limit for shared claims

This branch comes out of two weeks of measuring OpenRL under load on GKE: first one
job at a time, then 12 and 24 jobs at once on a mixed H100 and L4 fleet. Each change
below fixes one thing the measurements pointed at. The write-up with every drill is in
runs/2026-10-17_demo_baselines/notes.md; the cold-start study is in
runs/2026-10-15_cold_start/REPORT.md.

What a cold job paid before, and what it pays now

One job alone, from the client's start to the end of its first step. H100 unless noted.

Phase	Before	After	Change
Worker image pull on a node without the tag	92-169 s	0 s	pre-pull DaemonSet
Trainer weights load, Qwen3-8B, cold Filestore	903 s	13 s warm, 71 s cold	parallel prefetch before from_pretrained
vLLM torch.compile, gemma-4-e2b	100 s	1 s	compile cache on the shared volume
CUDA graph capture, gemma-4-e2b	53 s	22-30 s	power-of-two capture sizes
Sampler engine init, gemma-4-e2b	199 s	57 s	the two rows above
Training per step, Qwen3-0.6B, 64 sequences	20 s	4 s	automatic token budget

What 12 jobs at once got, same fleet, same jobs

Jobs that finished 3 steps within 180 s of kickoff. Six H100s; the L4s hurt every mix and
are out of these rows.

Drill	Setting	Result
3	binpack as on main	3 of 12
4	binpack with a seat limit of 2	7 of 12
7	the same, with the LoRA runtimes already up	8 of 12

The changes

Trainer and sampler start

• training/prefetch.py: reads a checkpoint with several threads and large blocks
  before from_pretrained. The mmap path read 260 KiB at a time with two reads in
  flight and was latency bound on NFS. OPEN_RL_PREFETCH=auto|always|off.
• The trainer picks its token budget from the model's vocabulary and the free device
  memory. The cluster never set OPEN_RL_TRAIN_TOKEN_BUDGET, so it ran one sequence
  per pass. A fixed 65536 ran gemma out of memory: 262k vocabulary, 22 GiB of logits.
• Ready records: the trainer's and sampler's ready keys carry start, load, and engine
  timestamps, and openrl_status shows them.

Sampler start on every node

• k8s/deploy/base/worker-prepull.yaml: a DaemonSet that pulls the worker image on
  every enabled node at deploy time. Workers default to imagePullPolicy: IfNotPresent.
• vLLM's compile cache lives on the shared volume (OPEN_RL_VLLM_CACHE_DIR, bundle
  default) or on a hostPath (OPEN_RL_VLLM_CACHE_HOSTPATH). The first sampler of a
  model on a GPU type compiles; every later one loads the artifact from any node. The
  cache key covers model, GPU type, and engine config, so LoRA and FFT samplers warm
  separately.
• CUDA graphs are captured for power-of-two batch sizes up to max_num_seqs
  (VLLM_CUDAGRAPH_CAPTURE_SIZES, default restores vLLM's list).

Weight updates on small devices

• A gemma-4-e2b FFT sampler died at its first weight update on an L4. vLLM's patch
  loader stages each patched tensor densely, then builds a mask and a merged copy: 2.5x
  the tensor, 11 GiB for gemma's per-layer embedding, while vLLM's share left 2 GiB.
• The sampler now wakes only the weights when it takes the GPU, applies the update, then
  wakes the KV cache (vLLM's documented order for RL updates). The headroom it keeps is
  2.5x the largest checkpoint tensor plus a margin, read from the safetensors headers,
  never below what the resident weights need (OPEN_RL_WEIGHT_UPDATE_HEADROOM_GIB
  overrides).
• The estimator adds the same room to the FFT sampler's placement claim. gemma FFT's
  claim grew from 11.7 to 23.6 GiB, so it no longer fits a 24 GB device and lands on the
  80 GB tier by its own size.

Scheduler

• --max-seats-per-claim (OPEN_RL_MAX_SEATS_PER_CLAIM, default 0 for no limit). A
  shared claim at the limit is not a candidate; the ledger's CAS re-checks it; a worker
  that finds every claim full cuts its own and waits Pending until a GPU is added or a
  seat frees. Before this, six FFT samplers time-sliced one H100 while other GPUs idled.
• kubectl get workloads shows the status reason, which names the seat limit while a
  worker waits.

Demo and drill tooling (scripts/)

• demo_jobs.py renders N in-cluster client Jobs from kind:model:count specs, with a
  per-client Hugging Face cache seeded from the shared one (twelve clients locking one
  NFS directory lost a client to a stale file handle) and optional session tags.
• demo_report.py, demo_watch.py, demo_rehearsal.sh: per-job and per-step timings
  from metrics.jsonl, a one-screen live view, and a one-command rehearsal with an
  idle check, volume trim, and an optional warm-up of the LoRA runtimes.
• cold_start_*.py|sh: the instrumented single-job measurement used for the report.
• cell_up.sh, cell_seed_hf.sh, cell_down.sh: an OpenRL cell in any GCP region.

What the drills still say

• A shared LoRA runtime started cold needs about 165 s for its first step with three
  jobs waiting. Started warm, the same three jobs finish three steps in about 90 s.
• Six H100s hold two LoRA runtimes on four GPUs and leave one pair for every FFT job.
  The FFT half queues. More large GPUs, not more policy, is what moves it.
• With small devices in the pool, binpack places by fit, and a gemma runtime that fits an
  L4 goes there and runs three times slower. This branch does not change that.

Tests

make test (260 unit tests), make -C scheduler/controller fmt-check vet manifests-check test, and lint on every tracked file. New tests cover the prefetch,
the token budget, the ready records, the capture sizes, the headroom and its floor, the
wake order, the estimator's update room, the seat limit in selection and in the ledger
CAS, and the demo tooling's defaults.