Forward-merge release/0.9 into main - #1137
Closed
rapids-bot[bot] wants to merge 6 commits into
Closed
rapids-bot[bot] wants to merge 6 commits into
rapids-bot[bot] wants to merge 6 commits into
Conversation
#### Overview Enrich Relay's source-backed OpenTelemetry GenAI projections, add generic managed-stream time-to-first-chunk telemetry, retain the validated coding-agent executable version from transparent CLI launches, and keep metric dimensions low-cardinality. Together, these changes let downstream telemetry consumers receive a more complete standards-aligned GenAI record plus a trusted harness version without fabricating values Relay cannot authoritatively determine. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details ##### GenAI projection coverage The `gen_ai` trace projection now fills additional registry attributes when Relay already has an authoritative normalized source: - classifies marked CLI turn roots as `invoke_agent` while preserving INTERNAL span kind and avoiding invented agent names; - projects embedding dimensions and encoding formats; - projects previous response ID and reasoning level; - projects provider-reported reasoning-token usage for OpenAI Responses and Gemini; - emits `gen_ai.request.stream` only when the request explicitly enables streaming; - emits `gen_ai.output.type` only from explicit OpenAI Chat/Responses controls: `audio` maps to `speech`, explicit single `text` remains `text`, and JSON response formats map to `json`; mixed, image, and implicit modalities remain omitted; - requires the canonical request-model attribute for internal agent spans instead of interpreting a generic `model` label as proof of one configured agent model; - uses generated semantic-convention constants everywhere the pinned crate exposes them, retaining local constants only for development attributes not yet generated. This preserves the recently merged `release/0.9` GenAI tool-content behavior while applying the broader projection audit. ##### Streaming latency Managed streaming calls now capture the monotonic duration from starting provider-stream execution to the first successful protocol chunk. Relay carries that optional duration on the LLM end-event profile and exports it through both standard OTEL lanes: - trace attribute: `gen_ai.response.time_to_first_chunk`, seconds; - metric histogram: `gen_ai.client.operation.time_to_first_chunk`, `f64`, unit `s`, with standard low-cardinality GenAI client attributes. The value is omitted for non-streaming calls, streams that never yield a chunk, and metric records where Relay cannot resolve the required provider. It measures first protocol chunk, not first textual token. ##### OTLP log/trace identity Fix generic log/trace correlation when a scope's local parent was not observed by the exporters. Logs now select the same observed-event trace root as trace export; explicitly imported parent contexts retain their existing behavior. This is not MCP-specific and applies to Full, GenAI, and OpenInference projections. Previously exported records are unchanged. Validation for commit `5f1cda8c`: - Decoded HTTP OTLP regression failed before the fix and passes afterward across all three projections, local/imported ancestry, ordinary tool logs, plugin marks, and late marks. - Real Claude Code 2.1.277 and Codex 0.155.0 runs against a local MCP server each exported two MCP marks joining two tool spans, plus three existing tool-start logs joining three tool spans. Each run included MCP success, MCP failure, and an ordinary shell tool. Live MCP validation used the separate stacked classification branch; this PR includes only the generic correlation fix and regression. - `just test-rust` passed: 5,185 workspace tests (one unrelated diagnostics timeout test passed on retry) and 10, 13, and 20 plugin-example tests. - The 19 focused log-export tests, changed-file pre-commit checks, and `just docs` passed. ##### Metric cardinality and trace correlation Relay no longer injects each event's propagation-root UUID into metric attributes as `nemo_relay.session.instance_id`. That value created one time series per session and prevented otherwise identical measurements from aggregating. A regression test now records the same counter under two propagation roots and verifies that the SDK exports one data point with the combined value and no automatic session UUID dimension. Explicit attributes supplied by a metric producer remain unchanged. Trace-linked exemplars are intentionally deferred: the vendored OpenTelemetry Rust SDK 0.32 metric recording path cannot carry the originating span context, so Relay does not synthesize a correlation identifier or attach an exemplar unless a future SDK path can prove that causal association. ##### Harness version metadata Relay already probes wrapped executables with `--version` for compatibility. This change retains that parsed version with the canonical launched-agent kind and adds trusted `agent_version` metadata to matching session, tool, and LLM events. Caller, hook, and configured metadata cannot override it, and it is omitted for unwrapped sessions, dry runs, or sessions belonging to a different harness. Consumers can promote the generic metadata field with `promote_metadata_prefixes`. ##### Compatibility note The projector replaces the obsolete development attribute `gen_ai.usage.cache_creation.input_tokens` with the current registry spelling `gen_ai.usage.cache_write.input_tokens`. Consumers querying the old key must update to the new key. No values are invented for registry entries that still require hosted-agent, prompt-management, memory, workflow, retrieval-content, evaluation-event, fetch/resume, compaction, modality-specific usage, or multi-generation lifecycle contracts. Validation completed: - `just test-rust` — 5,182 workspace tests passed; - native plugin example — 10/10 passed; - gRPC worker plugin example — 13/13 passed; - language-binding plugin example — 20/20 passed; - focused metric cardinality regression — passed; - focused GenAI projection suite — 14/14 passed; - focused time-to-first-chunk lifecycle/trace/metric suite — 3/3 passed; - `nemo-relay-types` test suites — passed; - staged `uv run pre-commit run` — passed, including formatting, Clippy, cargo check, and documentation link checks. #### Where should the reviewer start? Start with `crates/core/src/observability/otel_genai.rs` for the field applicability and projection decisions. Then review `crates/core/src/stream.rs` and `crates/core/src/observability/otel_metrics.rs` for the TTFC lifecycle, metric path, and low-cardinality aggregation behavior. Harness-version capture begins in `crates/cli/src/process/launcher.rs` and is trust-bound in `crates/cli/src/sessions/mod.rs`. Focused regression coverage is in the corresponding `otel_tests.rs`, `otel_metrics_tests.rs`, `llm_api_tests.rs`, launcher tests, and session tests. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: none ## Summary by CodeRabbit * **New Features** * Relay sessions detect Claude Code and Codex versions and include them in matching observability metadata. * Streaming LLM telemetry records time to first chunk. * Usage data reports uncached input tokens across supported providers. * GenAI observability adds expanded standardized tool, usage, reasoning, cache, and retrieval metadata. * **Bug Fixes** * Agent version metadata is restricted to matching wrapped agents; unwrapped sessions and dry runs omit it. * Improved cache-write metadata mapping for framework integrations. * **Documentation** * Added guidance for agent identity, streaming response timing, and supported GenAI attributes. Authors: - Bryan Bednarski (https://github.com/bbednarski9) Approvers: - Will Killian (https://github.com/willkill07) URL: #1121
Author
|
FAILURE - Unable to forward-merge due to an error, manual merge is necessary. Do not use the IMPORTANT: When merging this PR, do not use the auto-merger (i.e. the |
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository: NVIDIA/NeMo-Relay/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
github-actions
Bot
added
size:XL
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1137 +/- ##
==========================================
+ Coverage 93.08% 93.33% +0.25%
==========================================
Files 312 372 +60
Lines 114876 130684 +15808
Branches 0 134 +134
==========================================
+ Hits 106925 121964 +15039
- Misses 7951 8719 +768
- Partials 0 1 +1
... and 63 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
…1134) #### Overview > [!WARNING] > This PR includes a minor breaking change to Rust source by adding a field to the public `OpenTelemetrySignalEndpointConfig` struct Extend `promote_resource_metadata_prefixes` from traces to OTLP logs and metrics so all three signals can project root-scope metadata into their OpenTelemetry `Resource`. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Targets `release/0.9` with the resource-promotion feature from #1124 and matching plugin-configuration helpers. Preserves the release branch's exporter configuration and service-name defaults; excludes unrelated main-branch changes and the 0.10.0 version bump. - Add `promote_resource_metadata_prefixes` to explicit log and metric endpoint configuration. Derived log and metric endpoints inherit the trace endpoint's prefixes. - Expose explicit signal-endpoint promotion through Python plugin helpers and stubs, Node.js TypeScript declarations (the JavaScript helper already preserves supplied fields), and Go plugin configuration structs. The Go trace endpoint also exposes the setting for trace resources and derived signals. Preserve existing defaults and accept the field under strict unknown-field validation. Also recognize the existing log `completed_span_context_ttl_millis` setting emitted by binding helpers under that policy. - Cover omitted, empty, valid, and invalid prefixes for both signals through each binding's serialization and native plugin validation; add a TypeScript compilation fixture and a core strict-policy regression test. - Reuse the existing metadata sanitization, supported-value validation, reserved-key protection, and configured-resource precedence rules. - Establish a resource route from each root Scope-start event and inherit it across child scopes and late marks. Expire completed routes before lookup, including the first late mark or child start beyond the TTL; the exact TTL boundary remains linked. - Cap active resource lineage at 4,096 scopes per log/metric endpoint. Preserve already-tracked long-running scopes, reject new starts at capacity before constructing providers, and emit `otel.resource_metadata_active_scope_limit`. A tracked scope end frees capacity for subsequent starts; untracked scopes' marks use the base resource. - Test TTL boundaries and decoded log/metric fallback exports, active-scope saturation, repeated-start identity, long-running scopes, rejection diagnostics, and admission after a tracked scope ends. - Route logs and metric measurements through provider pipelines keyed by the effective resource. OpenTelemetry resources are provider-scoped, so records and measurements cannot select a resource independently. - Retain and flush admitted resource-keyed providers for the subscriber lifetime. Each log or metric endpoint admits at most 16 dynamic resource pipelines, excluding its base provider; existing keys remain reusable at the pipeline cap, subject to active-scope capacity, and no pipelines are evicted. - New resource keys beyond the cap fall back to the configured base provider and record `otel.resource_metadata_pipeline_limit`. Log delivery continues; metric measurements may aggregate together after losing promoted resource distinctions. Construction failures retain their separate diagnostic and base-resource fallback. - Test the exact capacity boundary, reuse after saturation, repeated rejection diagnostics, and decoded log/metric payloads for overflow. - Document the configuration, precedence, lifecycle, diagnostics, and low-cardinality requirement. Behavioral and compatibility changes: - When the new setting is omitted, it defaults to an empty list, is omitted during serialization, creates no resource-keyed providers, and preserves the existing exported resource and signal contents. - Metric subscribers now observe non-metric scope lifecycle events so they can establish resource lineage. With promotion disabled these events do not emit metrics, but they add classification and locking work to the default metric-subscriber path. - When enabled, each admitted effective resource creates retained logger and meter provider pipelines, up to the fixed limit of 16 per endpoint and signal. Provider count and active resource lineage have separate fixed limits. Neither bounds total process memory, completed-route caches, or existing trace-correlation state. Trace pipeline behavior is unchanged. Operators must still restrict promoted values to controlled, low-cardinality dimensions; request, tenant, and user identifiers are inappropriate. - Explicit `service_name`, `service_namespace`, `service_version`, and `resource_attributes` continue to win over promoted metadata with the same key. - Adding a field to the public `OpenTelemetrySignalEndpointConfig` struct is source-breaking for downstream Rust callers that construct it with exhaustive literals. Adding fields to the Go configuration structs can also break unkeyed composite literals. Serde-based TOML and JSON configuration remains backward compatible. Scope decision: - The log/metric pipeline and active-scope caps are internal fixed safety limits, not new public configuration fields. Active scopes are not evicted by age: missing ends can occupy slots until subscriber replacement, but cannot grow the active-route map past its cap. Trace pipeline limits and shared runtimes remain outside this change. - Direct Python, Node.js, and Go log/metric subscriber APIs were considered and are intentionally out of scope. This PR exposes promotion through plugin configuration; adding direct subscriber options would require separate native bridge/FFI work and validation. Validation: - `just test-rust`: 5,184 workspace tests and 43 Rust plugin-example tests passed on the final run. The first run hit a CLI gateway-process termination failure; that test passed in isolation and on the full rerun. A standalone example attempt picked up local worker configuration; the canonical rerun isolated configuration and passed. - `just test-go`: passed, including omitted, empty, valid, and invalid trace-promotion configuration cases. - `just test-python`: 78 native bridge tests, 702 Python tests, and 24 plugin-example tests passed. - `just test-node`: 3 native tests, 427 JavaScript tests, and 26 plugin-example tests passed. - Focused lineage TTL/capacity tests and decoded OTLP log/metric fallback tests passed, including the existing dynamic-pipeline saturation regression. - `just docs`, staged `uv run pre-commit run`, and `git diff --cached --check`: passed. - These checks exercise plugin configuration and runtime export behavior. Direct subscriber API additions remain outside this PR's scope. #### Where should the reviewer start? Start with the shared promotion and lineage behavior in `crates/core/src/observability/otel_signal.rs`, then review provider selection in `otel_logs.rs` and `otel_metrics.rs`. The export-level log-and-metric inheritance test in `crates/core/tests/unit/observability/otel_metrics_tests.rs` demonstrates configured precedence and child-scope routing. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to #1124 (replacement targeting `release/0.9`; supersedes the original PR). ## Summary by CodeRabbit ## New Features - OpenTelemetry trace, log, and metric endpoints can promote selected root-event metadata into resource attributes. - Configurable promotion prefixes are available across supported client APIs and inherited by derived endpoints. - Related events retain consistent resource routing, including recently completed scopes. - Resource-specific telemetry pipelines support coordinated flushing and shutdown. ## Bug Fixes - Resource or pipeline creation failures fall back to the configured endpoint resource. - Invalid promotion-prefix configurations now produce diagnostics. - Resource and pipeline limits are enforced with fallback behavior. ## Documentation - Updated guidance covers configuration, inheritance, supported fields, limits, and recommended metadata values. Authors: - Bryan Bednarski (https://github.com/bbednarski9) Approvers: - Eric Evans II (https://github.com/ericevans-nv) - Will Killian (https://github.com/willkill07) URL: #1134
github-actions
Bot
added
lang:go
#### Overview Document the 0.9.1 patch-release changes and upgrade considerations, including PR #1134. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Add a 0.9.1 release-notes section covering every change merged to `release/0.9` after the `0.9.0` tag, including OTLP resource metadata promotion from #1134. - Add migration guidance for daemon logging configuration, OTLP signal endpoint struct literals, GenAI telemetry consumer updates, and Anthropic compaction cache bypasses. #### Where should the reviewer start? Start with `docs/about-nemo-relay/release-notes/index.mdx`, then review the matching 0.9.1 guidance in `docs/reference/migration-guides.mdx`. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: #1134 ## Summary by CodeRabbit - **Documentation** - Added NeMo Relay 0.9.1 release notes covering OpenTelemetry enhancements, managed Codex routing, daemon logging changes, Anthropic compaction cache bypasses, Codex permission handling, telemetry correlation, and dependency updates. - Added migration guidance for daemon logging configuration, OpenTelemetry compatibility updates, GenAI cache metric changes, and Anthropic compaction-related response-cache bypasses. - Documented the remaining limitation affecting Anthropic compaction replay. Authors: - Will Killian (https://github.com/willkill07) Approvers: - Bryan Bednarski (https://github.com/bbednarski9) - Maryam Najafian (https://github.com/mnajafian-nv) URL: #1139
#### Overview Fix false `otel.logs_dropped` errors during shutdown when resource metadata promotion creates multiple OTLP log providers. A provider could drain successfully and still report another provider's pending records as dropped, causing plugin teardown to fail even when every record was delivered. Targets `release/0.9`. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details Give each resource-specific log provider independent delivery counters. Retain child diagnostics in the endpoint owner so actual queue drops and export failures remain visible in the aggregate delivery summary. Register child diagnostics only after provider construction succeeds; the existing resource-pipeline cap bounds retention. No public API or configuration changes. The reproduction queues one record in one provider and three in another. Previously, shutting down the first provider reported three dropped logs before the second provider drained; all four records ultimately reached the exporters. Tests now cover independent accounting, aggregate failure reporting, and decoded OTLP delivery for the base resource plus two promoted resources, with both direct shutdown and flush-then-shutdown. Validation on macOS ARM64: - `just test-rust`: 5,202 workspace tests and 43 example tests passed for the fix. The subsequently added multi-resource OTLP regression passed separately and in 100 consecutive runs, each exercising both shutdown paths. - `just test-python`: 78 bridge tests, 702 Python tests, and 24 example tests passed. - `just test-node`: 3 bridge tests, 427 Node tests, and 26 example tests passed. - Tokenomics MR36 content and metrics sidecar smoke checks passed with resource promotion enabled. The source checkout reports version 0.9.0; the metrics smoke used a temporary manifest range accepting both that source build and the installed 0.9.1 Python runtime. - Staged repository hooks passed, including formatting, Clippy, and compile checks. The initial Rust suite run had one CLI test-hook failure while a concurrent CLI build replaced its test-enabled executable. The complete suite passed when rerun without that concurrent build. Linux/Windows CI remains to be run; experimental Go bindings were not separately tested. #### Where should the reviewer start? Start with `ensure_dynamic_log_pipeline` and `LogDeliveryDiagnostics::failure_counts` in `crates/core/src/observability/otel_logs.rs`. Then review `resource_log_providers_drain_without_false_delivery_failures` in `crates/core/tests/unit/observability/otel_metrics_tests.rs`, which checks actual OTLP resource identities and record counts across three providers. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: none ## Summary by CodeRabbit * **Bug Fixes** * Improved OpenTelemetry log delivery diagnostics for dynamic pipelines and provider shutdowns. * Prevented logs pending in other providers from being incorrectly reported as dropped. * Failure summaries now include export failures across all related pipelines. * **Tests** * Added coverage for shutdown, flushing, resource-scoped events, successful delivery, and diagnostic reporting. Authors: - Bryan Bednarski (https://github.com/bbednarski9) Approvers: - Will Killian (https://github.com/willkill07) URL: #1141
…iptions (#1143) #### Overview Codex still shows `Hook failed: hook returned invalid permission-request JSON output` on 0.9.1. Codex adds `tool_input.description`, the approval reason, to PermissionRequest but not to PreToolUse, so escalated commands never match the recorded tool and get denied. The deny then goes out as `{"decision","reason"}`, which Codex rejects. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Drop `description` from built-in Codex tools before matching. MCP tools keep their arguments as-is. - Return Codex's `hookSpecificOutput` deny shape from both the gateway and the managed daemon. - Update the tests to use Codex's documented payloads and assert the full deny shape. - Tried it locally with `nemo-relay run --agent codex` and three escalated commands. Every hook completed with no denies and no `Hook failed`. The same prompt fails on 0.9.1. Identical duplicate calls still deny as ambiguous. I left that alone since it looked like deliberate hardening. #### Where should the reviewer start? `crates/cli/src/agents/codex/adapter.rs` #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to #1112 and [RELAY-887](https://linear.app/nvidia/issue/RELAY-887/fix-invalid-codex-permissionrequest-validation-in-nemo-relay) ## Summary by CodeRabbit * **Bug Fixes** * Permission denials now use Codex’s expected response format and include a denial message. * Non-MCP permission requests no longer include the tool description in their arguments. MCP requests continue to pass the description through. * Denial responses are consistent across managed and server-based permission checks, including unmatched or ambiguous tool requests. Authors: - Ethan Look-Potts (https://github.com/elookpotts-nvidia) Approvers: - Will Killian (https://github.com/willkill07) URL: #1143
#### Overview Add unauthenticated, read-only operational endpoints to the managed NeMo Relay daemon so service managers and Fleet-style deployment tooling can verify the daemon process, determine its configured deployment mode, enumerate currently assigned worker generations, and inspect the broker's last-known status for those workers. This targets the `release/0.9` branch and preserves fail-open behavior: daemon health remains independent of individual route state, and pass-through or activating routes are intentionally absent from worker listings because they do not have an assigned worker generation. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details This PR adds the following daemon HTTP routes: - `GET /healthz` - Returns `200` while the daemon HTTP process is serving requests. - Reports the service name, Relay version, daemon process instance ID, and global `deployment_mode` (`managed` or `pass_through`). - `GET /_nemo_relay/v1/workers` - Returns a deterministic, sorted list of currently assigned worker UUIDs. - `GET /_nemo_relay/v1/workers/status` - Returns a deterministic aggregate list of last-known broker status for currently assigned workers. - `GET /_nemo_relay/v1/workers/<instance_uuid>/status` - Returns one assigned worker status object. - Returns `404` with `worker not found` when the UUID is unknown, drained, replaced, or no longer assigned. All successful operational responses include `Cache-Control: no-store`. Unknown-worker responses also include `Cache-Control: no-store`. Worker status is derived under the broker registry read lock and includes: - worker generation UUID; - broker lifecycle state (`ready`, `draining`, or `recovering` for an assigned worker); - number of MCP references retaining the route; - last-known authenticated control-channel availability; and - requests accepted by the broker and still in flight at snapshot time. These values are instantaneous broker snapshots, not active network probes. Global pass-through, transient fail-open, empty, and activating routes do not have assigned worker UUIDs and are omitted from worker results. `/healthz` continues to describe daemon process health and global deployment mode rather than deriving health from route or worker state. The endpoints are intentionally unauthenticated and expose only bounded operational metadata. Responses exclude route credentials, machine-user fingerprints, worker endpoints, MCP session IDs, activation IDs and grants, provider credentials, configuration contents, and internal worker session credentials. Documentation calls out worker UUIDs and activity counts as operational metadata and directs remote deployments to restrict the daemon listener through normal binding, firewall, and reverse-proxy policy. The implementation promotes a credential-free registry snapshot from test-only lifecycle inspection into a production operational surface, keeps output ordering deterministic, and adds coverage for: - managed and global pass-through daemon health modes; - worker listing, aggregate status, and individual lookup; - unknown-worker `404` behavior; - omission of fail-open routes without assigned workers; - ready-worker and transient fail-open registry state in the same broker; and - response redaction of route tokens, worker credentials, and worker endpoints. Documentation now covers the HTTP contract, response shape, field semantics, empty-list ambiguity, worker-backed deployment verification, troubleshooting, architecture placement, macOS/Linux/Windows runbooks, and remote-listener exposure. Validation performed: - `cargo test -p nemo-relay-cli daemon_health_is_a_public_process_probe` - `cargo test -p nemo-relay-cli worker_status -- --nocapture` - focused registry status tests for ready and fail-open routes - live daemon/MCP/worker runs covering: - managed daemon with no routes; - managed daemon with a ready worker; - global `--pass-through` registration; - managed worker activation failure transitioning to fail-open; and - simultaneous ready-worker and fail-open routes in one broker - live HTTP checks for all four operational routes and unknown-worker `404` - `cargo fmt --all -- --check` - `uv run pre-commit run` (including docs link checks, Clippy, and Cargo check where applicable) - `just docs` The full `just test-rust` suite was not rerun after the final endpoint reshape. An earlier run on this branch executed 5,201 tests: 5,193 passed and eight existing configuration-isolation tests failed while reading ambient provider-auth configuration. The endpoint-specific tests, live-process tests, Clippy, Cargo check, formatting, and documentation checks all pass on the final branch state. No existing public route is renamed or removed, and no package or binding API changes. #### Where should the reviewer start? Start with `crates/cli/src/daemon/broker/server.rs` for the route definitions and response contract, then review `crates/cli/src/daemon/broker/registry.rs` for the lock-bounded, credential-free worker snapshots. The most important design decision is that `/healthz` reports process health and global deployment mode while worker endpoints report only currently assigned generations; fail-open and activating routes are deliberately omitted rather than represented as workers. The focused HTTP contract coverage is in `crates/cli/tests/coverage/daemon/server_tests.rs`, and the registry-state coverage is in `crates/cli/tests/coverage/daemon/registry_tests.rs`. The complete user-facing contract begins in `docs/daemon/reference.mdx` under **Probe Daemon Health** and **Inspect Worker Status**. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: none ## Summary by CodeRabbit * **New Features** * Added daemon health checks reporting service version, instance, and deployment mode. * Added worker-status endpoints to list assigned workers and view their state, activity, and control availability. Responses omit credentials, are not cached, and return 404 for unknown workers. * **Documentation** * Updated daemon setup, verification, reference, and troubleshooting guidance for health and worker-status checks, including access restrictions and how to interpret empty worker lists. Authors: - Bryan Bednarski (https://github.com/bbednarski9) Approvers: - Maryam Najafian (https://github.com/mnajafian-nv) - Will Killian (https://github.com/willkill07) URL: #1142
2 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to release/0.9 that creates a PR to keep main up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.