Repository navigation
feat(observability): show runs' pods and nodes and serve their logs - #392
Merged
ShubyM merged 4 commits intoOct 30, 2026
Merged
Conversation
ShubyM
force-pushed
the
observability/cluster-view-and-logs
branch
from
October 29, 2026 20:44
38517b2 to
5d6665a
Compare
The inspection API serves agents and the CLI as well as a dashboard, so the package is named for what it does. This moves the package and its two test modules and updates the import and the Makefile test lists. Nothing else changes.
The inventory now joins each run with its pods, nodes and Workloads from Kubernetes, gives it a lifecycle of ended, failed, waiting, placed or active, and reports the cluster's nodes and problem pods. /api/v1/inspect/logs pages a run's worker logs from Cloud Logging, reading only pods with the run's owner label, and with pod=client reads the client's logs.log from the shared volume.
…a declared path Worker logs are off until OPEN_RL_LOG_STORE names a store. cloud-logging is the only store today. It sits in a small registry, so another store such as Loki can be added later without a route change. The GKE metadata server no longer turns log reads on. OPEN_RL_GKE_TELEMETRY is gone. A run's worker pods are now found by name. The scheduler names a worker pod orw-<Workload>, and a run's Workloads start with <kind>-<owner>-. The store reads the worker namespace's pods with that prefix instead of the timeslice.io/owner label. The server no longer scans the shared volume for client directories. A client declares its files with the openrl.log_path setting, which is stored on the model. pod=client reads <log_path>/logs.log only when its resolved path is under OPEN_RL_CLIENT_LOG_ROOTS. OPEN_RL_CLIENT_RUNS_ROOT is gone.
OPEN_RL_LOG_STORE is now the one log setting. cloud-logging reads the project that the Google credentials name. cloud-logging:<resource> reads that Cloud Logging resource instead, such as a log bucket view. OPEN_RL_LOG_VIEWS, OPEN_RL_GKE_PROJECT, _CLUSTER and _LOCATION and the metadata server lookup are gone. The worker filter keeps the namespace and the anchored pod name prefix. It no longer pins the project, cluster and location, because owner ids are unique. Client logs now come from the same store. A client sets the session tag openrl.client_log_id=<id>, which is stored on the model. Its log lines carry the label openrl_client=<id>. For a client pod that is a pod label, which GKE copies to k8s-pod/openrl_client. pod=client reads the lines with either label. openrl.log_path, OPEN_RL_CLIENT_LOG_ROOTS and the file reader are gone. A store now takes a selector: a namespace's worker pods by name prefix, or a client id.
ShubyM
force-pushed
the
observability/cluster-view-and-logs
branch
from
October 29, 2026 22:32
a19d84e to
6d10bbd
Compare
2 of 9 tasks
droot
approved these changes
Oct 30, 2026
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Phase 3 of #390. Runs now show where they ran, and their logs can be read without going to the log store by hand.
What changed
server.dashboardis renamed toserver.observability(first commit, move only)./api/v1/inspect/inventoryadds each run's pods, nodes and lifecycle, and aclustersection with nodes, GPU counts and problem pods./api/v1/inspect/logs?run_id=<id>pages a run's logs from your log store. Addpod=clientfor the client's logs.What you configure
OPEN_RL_LOG_STORE=cloud-loggingon the API server. On GKE, worker logs already go to Cloud Logging. Grant the API serverroles/logging.viewer(see the GKE setup docs).OPEN_RL_LOG_STORE=cloud-logging:<log view>. It sees client logs only if that bucket's sink routes them in.openrl.client_log_id=<id>, and ship the client's logs to the same store labelledopenrl_client=<id>. For a client pod, that is a pod label.Without
OPEN_RL_LOG_STORE, log reads are off. Cloud Logging is the only store today. Others such as Loki plug in as adapters.Verification
make testandmake lintpass./logsreturns every record the existing dashboard returns.