Repository navigation
feat(telemetry): record trainer and sampler activity for the inspection API - #396
Merged
Merged
Conversation
ShubyM
force-pushed
the
observability/run-telemetry
branch
from
October 30, 2026 18:24
3cfbc07 to
db3313e
Compare
droot
approved these changes
Oct 30, 2026
ShubyM
marked this pull request as ready for review
October 30, 2026 20:13
…gh one buffer Each process that records keeps one buffer in memory. A hook on a request path only appends to it, swallows its own errors and returns nothing. One thread drains the buffer into Redis in one pipeline every 5 s. A worker's record of itself goes in the same pipeline, so the thread is also its heartbeat. A failing or hanging Redis costs a flush. The sync Redis client now has socket timeouts, so a hung call cannot hold the thread forever. Trainers, in-process samplers and pool dispatchers keep a worker record: phase, GPU turns and the waits for them, outstanding requests by op and run, and the settings env without secrets. The time slicer's acquire and the wake and sleep inside the turn show on it. The record says exited after a last flush, which an exiting process waits up to 5 s for. The ledger counts each run's trainer requests enqueued (API server), popped and answered (trainer) in open_rl:ledger:<run>. The API server starts its own recorder, which also writes for its in-process trainer. Worker pods get POD_NAME, POD_UID and NODE_NAME for the records.
The trainer records one op record per op in open_rl:operations:<run>,
capped at the newest 2,000: operation, status, error type, timing, finite
metrics and the step it belongs to. A forward-only call records as
forward. Its own sample ops keep a separate list. A failure answered
before its op ran, such as a failed wake or parse, gets a record too.
Each successful optim_step adds to open_rl:steps:<run> and
open_rl:step_times:<run>, and a failed op to the ledger's failed count.
A step stamp counts this process's steps on top of the run's count, which
the recorder reads once per run before it writes the run's first op. A
restarted trainer continues the run's numbering.
The API server stamps the enqueue time on each command it queues, as an
openrl-enqueued-at entry in trace_context. The op record carries
queue_seconds, the time from enqueue to the op's start. The stamp rides in
trace_context because workers from before it forbid unknown command
fields but already accept any trace_context entry, so the API server can
roll ahead of the workers.
Workers still replace NaN and infinity before answering. sanitize_metrics
names the replaced metrics to telemetry first, so a NaN grad norm is
recorded in nonfinite instead of as 0.0.
GET /api/v1/inspect/runs/{id}/steps?last=&fields= groups the op records
by step, with each op's count, failures, seconds and metric spread. Ops
after the last optim_step that do no training, such as the final save,
join the last step instead of forming an empty one.
The hook is in SamplingStore.set_result, where the first saved result wins, so it sees every sampling answer a client gets exactly once, from a dispatcher or from the API server's deadline sweep and job failures. A result that loses to an earlier one is not recorded. A failed answer keeps its own record in open_rl:operations:<run>:samples. Successful answers are only counted in memory. Each flush writes one group record per run with their count and the window they span, so there is no record per sample. A run's sends are the sum of its group counts plus its failed records. There is no separate sample counter. A send that raises is sent again and is counted once, when it is answered. A failed weight update answers nothing, and the requests wait for the restarted sampler. A revision no replica can load fails each of its requests once. Samples answered before the engine dies are written by the exit's last flush. accepted_at moves to sampling_store, which now needs it for a failure saved without a send.
ShubyM
force-pushed
the
observability/run-telemetry
branch
from
October 30, 2026 20:18
db3313e to
47a7f81
Compare
droot
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Phases 6 and 7 of #390. Record what each run's trainer and sampler did, so the inspection API can show steps, failures and stuck requests.
What changed
/api/v1/inspect/runs/{run_id}/stepsreturns each step's ops, timing and metrics.What you configure
Nothing. Records go to the API server's Redis.
Compatibility
The enqueue time rides in each command's existing
trace_context, so workers on the previous image still accept commands from a new API server.Verification
make testandmake lintpass at each commit. A broken, hanging or missing Redis changes nothing a request returns./stepsmatches the existing dashboard on every step for two runs.