feat(switchyard-server): Add /v1/routing/session-stats

[grahamking]

To match Python FastAPI server.

• Pass flag --routing-log-file file.jsonl
• Writes one line for each request to that file

We now have three ways of reporting basically the same numbers:
Prometheus metrics, in-memory stats, and on-disk routing stats.

Assisted-by: Codex:GPT 5.6 Sol medium
Signed-off-by: Graham King grahamk@nvidia.com

Summary by CodeRabbit

• New Features

  • Added optional routing logs that record completed requests, including model, tier, session, and token usage details.
  • Added session statistics endpoint for viewing call and token totals grouped by model.
  • Added support for streaming responses, recording usage after streams finish.

• Documentation

  • Documented routing-log configuration and session statistics behavior, including availability and error conditions.

[grahamking]

@CodeRabbit review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Walkthrough

The server adds optional JSONL routing logs, records terminal usage for aggregate and streaming responses, and exposes session-level statistics through a conditional endpoint.

Changes

Session routing log

Layer / File(s)	Summary
Routing record persistence crates/switchyard-server/src/routing_log.rs, crates/switchyard-server/src/lib.rs, crates/switchyard-server/Cargo.toml	The server writes routing records with request context, model, tier, timestamp, and token usage. It rescans JSONL records to build per-session and per-model totals.
Server configuration and statistics endpoint crates/switchyard-server/src/cli.rs, crates/switchyard-server/src/lib.rs, crates/switchyard-server/README.md	The CLI configures a routing-log path. ServerState enables the log and conditionally registers /v1/routing/session-stats. The README documents the flag and endpoint.
Request recording and validation crates/switchyard-server/src/usage_metrics.rs, crates/switchyard-server/src/lib.rs, crates/switchyard-server/tests/server.rs	Request headers flow into terminal usage handling. Aggregate responses and completed streams append routing records. Integration coverage verifies session totals and record timestamps.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Poem

A rabbit logs each routed byte,
In JSON lines through day and night.
Sessions bloom with counted calls,
Models answer through the walls.
“Stats are ready!” hops the hare.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the primary change: adding the /v1/routing/session-stats endpoint.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/switchyard-server/src/lib.rs`:
- Around line 110-127: The routing-log implementation must avoid blocking async
request workers: update SharedRoutingLog::append to enqueue records through a
bounded dedicated writer rather than locking the file and performing synchronous
write I/O, and update get_session_stats/snapshot_session to use bounded blocking
work while maintaining a per-session aggregate or index instead of rescanning
the JSONL file to EOF for every query. Apply the corresponding changes at
crates/switchyard-server/src/lib.rs lines 110-127 and 864-873.

In `@crates/switchyard-server/src/routing_log.rs`:
- Around line 27-113: Document the routing-log API and lifecycle behavior across
all affected sites: in crates/switchyard-server/src/routing_log.rs lines 27-113,
add /// documentation for RoutingLog::new, RoutingLog::append, snapshot, and
RoutingLogContext::from_headers; in lines 152-219, document the JSON validation
and aggregation performed by SessionStatsSnapshot::add_record and add_counter.
In crates/switchyard-server/src/lib.rs lines 96-128, document that
SharedRoutingLog serializes writers but does not synchronize snapshots. In
crates/switchyard-server/src/usage_metrics.rs lines 17-79, document observe and
add a block comment explaining clean-stream terminal recording and failure
behavior.
- Around line 28-40: Update ServerResult::new to inspect the existing log’s
final byte before opening it for appending, and write a newline when the file is
non-empty and does not already end with '\n', while preserving existing
newline-terminated logs. Add a restart-recovery test covering a partial final
JSON record followed by a new record and verify snapshot() retains both records.
- Around line 36-40: Update the file-opening logic in the routing-log
initialization flow around OpenOptions to enforce owner-only permissions (0600)
for both newly created and existing files before writing records. Use the
platform-appropriate permission-setting API and ensure the permission update
occurs before the file is used, while preserving the existing routing_log_error
handling.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c92e9501-fd73-45eb-b885-1e21ab4c4d02

📥 Commits

Reviewing files that changed from the base of the PR and between 6ff491a and f229df5.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock, !Cargo.lock

📒 Files selected for processing (7)

• crates/switchyard-server/Cargo.toml
• crates/switchyard-server/README.md
• crates/switchyard-server/src/cli.rs
• crates/switchyard-server/src/lib.rs
• crates/switchyard-server/src/routing_log.rs
• crates/switchyard-server/src/usage_metrics.rs
• crates/switchyard-server/tests/server.rs

[nachiketb-nvidia]

LGTM, feature parity with python looks like its exactly what we need, for benchmarking and intake