Repository navigation
Conversation
droot
added a commit
to droot/open-rl
that referenced
this pull request
Oct 29, 2026
…on a fresh cluster k8s/deploy/gke-use5: the FFT bundle plus openrl-dev#383's dashboard for open-rl-dra-use5 (us-east5, 8 x H100, DRA), with what the trial needed around it: a dedicated dashboard service account for Workload Identity, the DCGM exporter and PodMonitorings for the exporter, the scheduler's openrl_gpu_held and the sampler pods, spread placement, and a demo app. The demo app is a small in-cluster FastAPI service on the API server image, code from a ConfigMap, that starts and stops sweeps of RL client Jobs from a page and shows the fleet from the inspection API: Sweep, Fleet, Timeline (occupancy traces per accelerator or job, other-jobs overlay, drag-to-select, hover details) and Weights (the sparse delta per step). It is specific to one cluster and registry. docs/briefs/pr383-dashboard-trial: the findings report for the team and the step-by-step log with screenshots.
droot
added a commit
to droot/open-rl
that referenced
this pull request
Oct 30, 2026
…on a fresh cluster k8s/deploy/gke-use5: the FFT bundle plus openrl-dev#383's dashboard for open-rl-dra-use5 (us-east5, 8 x H100, DRA), with what the trial needed around it: a dedicated dashboard service account for Workload Identity, the DCGM exporter and PodMonitorings for the exporter, the scheduler's openrl_gpu_held and the sampler pods, spread placement, and a demo app. The demo app is a small in-cluster FastAPI service on the API server image, code from a ConfigMap, that starts and stops sweeps of RL client Jobs from a page and shows the fleet from the inspection API: Sweep, Fleet, Timeline (occupancy traces per accelerator or job, other-jobs overlay, drag-to-select, hover details) and Weights (the sparse delta per step). It is specific to one cluster and registry. docs/briefs/pr383-dashboard-trial: the findings report for the team and the step-by-step log with screenshots.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run telemetry and the Svelte dashboard, as one branch on main. It folds together a five-PR stack reviewed on ShubyM/open-rl (#44 to #48 there).
placedAt,releasedAtand GPU UUIDs on Workload status, and exportsopenrl_gpu_held.src/server/telemetry.py): onetelemetry.record(run_id, kind, **fields)call keeps the record in Redis and logs it. Worker heartbeats, command records, GPU turns and waits, placements with pod exit reasons, and one sample record per served group. Telemetry never raises into or blocks the request path./api/v1/inspect,src/server/dashboard/): facts, not verdicts, for an agent or a person. Run and session summaries, steps, commands, activity, accelerators, named PromQL queries, and logs. A relaunch lists who changed the run's Workloads, from the audit log.log_pathon the shared volume. The client changes nothing.src/server/dashboard/ui): Overview, Run (Activity, Metrics, Logs), Nodes and Health. GPU charts can show what ran, DCGM readings, or a rolling one-minute duty.Design notes are in
docs/designs/018-run-diagnostics.md. Deployment isk8s/deploy/dashboard.Checked on GKE (
open-rl-fft-test, diag stack) with exclusive and time-sliced runs, including each sampler's own vLLM charts. Three agent probes answered 14 of 14 questions from the API alone. Unit tests show the same 4 GPU-only errors as main.Open:
SAMPLING_BACKEND=vllm-pools) records no samples yet.