feat(analytics): add route reliability and latency analytics

[Wibias]

Summary

RI-03 of the Router Intelligence / Routing Control Plane programme. Adds
source-backed routing analytics computed from the request-history index
(RI-02), never from repeated full JSONL scans.

Read-only: this PR cannot change routing behavior (ADR-10 - no automatic
self-tuning).

Metrics

GET /api/routing-analytics (filters: provider, model, profileId,
surface, from, to) returns:

• Success / failure / cancelled rates (client cancellations are never
  provider failures)
• Fallback rate (multi-attempt share) and attempt counts
• p50/p95/p99 total duration and time-to-first-token (nearest-rank
  percentiles, with TTFT sample count + coverage)
• Incomplete-stream rate (terminalStatus: "incomplete")
• Cooldown-triggering failure count (status 429, or attempts with
  rate-limit-429 / key-429 / oauth-401 / anthropic-oauth-429
  recovery kinds)
• Estimated cost per successful request and total for successful requests
  (reuses usage/cost.ts pricing; unknown prices stay unknown, never zero)
• Usage coverage and price coverage
• Provider/model/account breakdown with per-bucket sample counts and p50
• Profile breakdown (from RI-01 decision traces) with revision
• Minimum-sample confidence (high >= 100, medium >= 20, else low)
• Explicit historyTruncated flag when the analysis cap (50,000 rows) is hit

Scope

• src/routing/analytics.ts - computeRoutingAnalytics() over the SQLite
  index (bounded, deterministic, single SQL pass + JS aggregation).
• src/server/management/routing-analytics-routes.ts - the endpoint,
  registered in management-api.ts.
• tests/routing-analytics.test.ts - 8 tests.

Privacy / security

• No prompts, credentials, or raw bodies: the analysis reads the same
  privacy-bounded columns the index already stores.
• Cost estimates are derived locally from usage + prices; no new data leaves
  the process.
• bun run privacy:scan passes.

Compatibility

• Additive API; /api/logs, /api/request-history, and usage.jsonl
  contracts unchanged.
• No routing behavior change.

Dependency

• Stacked on RI-02 (PR feat(control-plane): add indexed cursor-paginated request history #1004) and RI-01 (PR feat(routing): record durable route decision traces #1003). Branch base:
  feat/ri-02-request-history-index head 7efb6e842. PR target is dev;
  the diff includes earlier PR commits until they merge.

Non-goals

• No routing-decision changes, no self-tuning, no profile weights mutation.
• No policy profiles (RI-04..08) - the profileId filter/breakdown only
  reads RI-01 traces.
• No explainability surfaces (RI-09), no GUI (RI-10).

Local verification (exact)

• bun x tsc --noEmit -> PASSED (0 errors)
• bun run test tests/routing-analytics.test.ts -> 8/8 pass
• Focused regression suites (analytics, request-history, route-decision-
  trace, request-log, usage-log, combos) -> 144/144 pass
• bun run privacy:scan -> passed

Notes for reviewers

• Classification is explicit and documented in classifyRow(): status 499 /
  client_cancel -> cancelled; terminalStatus: "incomplete" -> failure;
  status >= 400 -> failure; otherwise success.
• Unknown prices produce null estimates and priceCoverage: 0 - never a
  fake zero cost.

Summary by CodeRabbit

• New Features

  • Added read-only routing analytics for request outcomes, latency, fallbacks, usage, costs, and provider/model/profile breakdowns.
  • Added filtering by provider, model, profile, surface, and time range.
  • Added a management API endpoint to retrieve routing analytics in JSON format.
  • Reports data confidence and indicates when history is truncated.

• Bug Fixes

  • Added validation for invalid analytics time ranges with clear error responses.

• Tests

  • Added comprehensive coverage for analytics calculations, filtering, truncation, and API responses.

[coderabbitai]

Warning

Review limit reached

@Wibias, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c2479ad5-dc5f-4062-b152-31b310160951

📥 Commits

Reviewing files that changed from the base of the PR and between e732d02 and b25ffa4.

📒 Files selected for processing (4)

• devlog/_plan/260804_router_intelligence/001_pr_stack_status.md
• src/routing/analytics.ts
• src/server/management/routing-analytics-routes.ts
• tests/routing-analytics.test.ts

📝 Walkthrough

Walkthrough

This change adds read-only routing analytics from the request-history SQLite index. It computes filtered metrics and breakdowns, exposes them through GET /api/routing-analytics, adds comprehensive tests, and updates RI-02 and RI-03 stack records.

Changes

Routing analytics

Layer / File(s)	Summary
History-backed analytics computation src/routing/analytics.ts, src/routing/history/indexer.ts	Defines analytics filters and result types, reads bounded and filtered history rows, classifies outcomes, computes latency, usage, cost, cooldown, confidence, and provider/model/profile breakdowns. Exposes requestHistoryDb() for access to the open history index.
Management API exposure and validation src/server/management-api.ts, src/server/management/routing-analytics-routes.ts, tests/routing-analytics.test.ts	Registers GET /api/routing-analytics, validates timestamp ranges, returns analytics JSON, and tests metrics, filters, truncation, environment isolation, and HTTP responses.
Stack status and acceptance record devlog/_plan/260804_router_intelligence/001_pr_stack_status.md	Marks RI-02 as merged and RI-03 as open and resynchronizing. Records RI-03 fixes, verification results, and pending final commit status.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant ManagementAPI
  participant RoutingAnalyticsRoute
  participant RequestHistorySQLite
  Client->>ManagementAPI: GET /api/routing-analytics
  ManagementAPI->>RoutingAnalyticsRoute: dispatch request
  RoutingAnalyticsRoute->>RoutingAnalyticsRoute: validate filters and timestamps
  RoutingAnalyticsRoute->>RequestHistorySQLite: compute filtered analytics
  RequestHistorySQLite-->>RoutingAnalyticsRoute: analytics result
  RoutingAnalyticsRoute-->>Client: HTTP 200 JSON response

Loading

Possibly related PRs

• lidge-jun/opencodex#494: Adds request-history records consumed by the analytics module.
• lidge-jun/opencodex#1003: Adds durable route-decision and usage traces used by the analytics input.
• lidge-jun/opencodex#1004: Provides the request-history SQLite index and requestHistoryDb() accessor used by this change.

Suggested reviewers: ingwannu, lidge-jun

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding routing analytics for reliability and latency.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

opencodex/src/routing/history/indexer.ts

Lines 131 to 132 in 2069e72

	return stored.sourceSize === Number(revision.size)
	&& stored.sourceMtimeMs === Number(revision.mtimeMs);

Use stable identity for appended usage logs

When usage.jsonl grows normally, the stored sourceSize/sourceMtimeMs no longer equal the current revision, so sourceIdentityMatches() returns false and the caller goes through destroyAndRecreate()/fullRebuild() before the tail-ingest branch can run. That makes /api/request-history and /api/routing-analytics repeatedly rescan the entire ledger under normal traffic and can stall large histories; compare stable file identity separately from size/mtime and reserve rebuilds for replacement or truncation.

---

opencodex/src/routing/analytics.ts

Line 231 in 2069e72

const estimate = estimateRequestCost({

Price combo rows with their attempts

For successful combo routes, the parent usage row is persisted as provider combo and model combo/..., while the priceable provider/model pairs live in entry.attempts (the existing log DTO uses the combo attempt estimator for this case). Calling estimateRequestCost with the parent row here makes estimatedCostUsd* and priceCoverage come back null/0 for priced combo traffic, skewing the analytics exactly when fallback routes are being evaluated; use the attempt-level estimator before falling back to single-request pricing.

---

opencodex/src/routing/analytics.ts

Line 283 in 2069e72

let profile = byProfile.get(row.profileId);

Separate profile revisions in breakdowns

When a routing profile is revised but keeps the same profileId, this map key collapses all revisions into one bucket and reports whichever profileRevision was seen first. That hides regressions between policy revisions and attributes successes/failures to the wrong revision in the profileBreakdown; key the bucket by both profile id and revision, or explicitly omit/mark the revision when multiple revisions are mixed.

---

opencodex/src/routing/analytics.ts

Line 250 in 2069e72

...(row.apiKeyId ? { accountRef: row.apiKeyId } : {}),

Use routed account refs, not admission keys

For Codex pool/OAuth/provider-account routing, apiKeyId is the inbound management/admission key, not the upstream account selected by the router; loopback Codex callers often have no apiKeyId at all. Building the analytics accountRef from this column collapses all provider accounts together or splits them by client key, so account-level reliability cannot identify a bad routed account; persist and aggregate the privacy-safe routeDecision.selected.accountRef instead.

---

opencodex/src/routing/history/indexer.ts

Line 86 in 2069e72

recordOwnedConfigPath(dir, `${dir}/${HISTORY_DB_FILENAME}`);

Track SQLite WAL sidecars in ownership

This only records routing-history.sqlite in the uninstall manifest, but the database is opened in WAL mode, so normal index writes also create routing-history.sqlite-wal and routing-history.sqlite-shm. Those unowned sidecar files can be left behind and prevent the config directory from being fully removed during uninstall/cleanup; register the sidecars as owned paths too, or avoid WAL for this disposable index.

---

opencodex/src/routing/history/indexer.ts

Lines 203 to 207 in 2069e72

	if (parsed && typeof parsed === "object"
	&& typeof parsed.requestId === "string"
	&& typeof parsed.timestamp === "number"
	&& typeof parsed.provider === "string") {
	return parsed;

Validate JSONL rows before inserting them

A parseable but hand-edited line that has only requestId, timestamp, and provider passes this check, then extractRow() sends missing model, status, or durationMs into NOT NULL columns and the whole index refresh fails. Since the canonical usage ledger may contain corrupt/manual rows and other readers skip them, validate all indexed fields (or normalize and drop invalid entries) before queueing the insert.

---

opencodex/src/routing/analytics.ts

Line 225 in 2069e72

if (row.usageStatus !== "unreported") usageReported += 1;

Count only measured usage as coverage

This treats usageStatus: "unsupported" as covered because it is merely not "unreported", but unsupported rows have no token measurements and the existing usage summary excludes them from coverage. A provider that never reports usage can therefore show high usageCoverage in routing analytics; count only reported and estimated rows as measured.

---

opencodex/src/routing/analytics.ts

Line 179 in 2069e72

if (filters.surface !== undefined) add("surface = ?", filters.surface);

Normalize surface filters for Codex and Claude Desktop

The persisted Codex surface is NULL and Claude Desktop is stored as claude-desktop, so a literal surface = ? predicate makes surface=codex return no Codex rows and makes surface=claude miss Claude Desktop traffic. That leaves callers unable to analyze just the default Codex bucket and makes surface-scoped analytics disagree with the existing usage surface semantics; translate these public surface values before adding the SQL predicate.

---

opencodex/src/routing/history/indexer.ts

Line 184 in 2069e72

const buf = Buffer.allocUnsafe(length);

Stream history ingestion in bounded chunks

On a first build, manual rebuild, or any large unindexed tail, length can be the remaining size of usage.jsonl, so this allocates one buffer for the whole ledger and then turns it into one giant string/split array. Large existing histories can OOM or block the single Bun server thread while serving /api/request-history or /api/routing-analytics; read and insert complete lines in bounded chunks instead.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Wibias]

[GD] Verdict: approve-comment

TLDR

• PR: feat(analytics): add route reliability and latency analytics #1005 — feat(analytics): route reliability and latency analytics
• Head: e732d02ef41a111327dc4649e61c6b933c4a7434
• Decision: approve-comment
• Usefulness: pass — read-only routing analytics over RI-02 index (rates, percentiles, cost coverage, breakdowns)
• Bugs: fixed — invalid test decisionIds caused profile_id null in index; fixtures corrected to valid 12-char hex
• Security: pass — local index, parameterized SQL, privacy scan green, no credential/prompt leakage
• Spec/standards: pass with note — add docs-site for new API when convenient
• Reviews: no unresolved bot threads on this PR
• Base/CI: base dev post-feat(control-plane): add indexed cursor-paginated request history #1004; CI pending on e732d02 after fix push
• Gate: ship-gate ready except CI completion on new head
• Owner actions: wait for CI green on e732d02, then merge; optional docs-site
• Bottom line: RI-03 analytics is merge-ready after CI confirms the test fix

Full review

See commit e732d02 for test fix and simplify.

[coderabbitai]

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@devlog/_plan/260804_router_intelligence/001_pr_stack_status.md`:
- Around line 43-44: Update the programme status statement near the document’s
introductory summary to remove or clearly label the stale “nothing merged”
claim, reflecting that RI-01 and RI-02 are merged. Preserve the existing
historical context if needed, and keep the RI status table and PR `#1004` entry
unchanged.
- Around line 129-135: Update the “Reviewed commit” and “Final commit” entries
in the stack status plan so they remain consistent: before the final push, mark
both as pending; after pushing, record the exact reviewed and final commit SHAs.
Associate the listed verification results with the corresponding SHA instead of
claiming the reviewed commit already matches the final commit.
- Around line 127-128: Correct the lineage labels in the Base SHA entry:
identify 2a72aa4a9 as the merged RI-02 commit and canonical RI-03 base,
7efb6e842 as the pre-merge RI-02 head, and 2069e724e as an RI-03 commit rather
than an RI-02 head.

In `@src/routing/analytics.ts`:
- Around line 238-239: Update the analytics aggregation around `entry`,
`cooldownFailures`, and the result object to parse every row needed for attempt
inspection, including failed and cancelled rows, while separating
recovered-success counts from `cooldownTriggeringFailures`. Increment
`cooldownFailures` only for non-success requests, track successful requests with
cooldown-triggering attempts in a new `cooldownRecoveredRequests` counter, and
expose that counter through `RoutingAnalyticsResult` and the returned result.

In `@src/routing/history/indexer.ts`:
- Around line 570-579: Change requestHistoryDb() to return a narrow read-only
query interface exposing only the query operation needed by
computeRoutingAnalytics, rather than the live Database with run, exec, and
schema mutation methods. Update the doc comment to state that callers must use
the handle synchronously and not retain it across await or beyond
closeRequestHistoryIndex(), which invalidates the underlying database.

In `@src/server/management/routing-analytics-routes.ts`:
- Around line 38-45: Update the routing analytics endpoint’s
computeRoutingAnalytics call to read an optional maxRows query parameter, parse
and validate it consistently with the existing analytics options, and forward it
so callers can request a smaller bounded window instead of always using the
default cap. Keep the existing provider, model, profileId, surface, from, and to
handling unchanged; do not add memoization unless already supported by the
surrounding implementation.

In `@tests/routing-analytics.test.ts`:
- Around line 56-72: Add a regression test alongside the existing routing
analytics tests that appends a non-4xx incomplete stream with attempts
containing a cooldown recovery kind such as "rate-limit-429", followed by a
successful attempt. Assert computeRoutingAnalytics reports failureRate as 1 and
cooldownTriggeringFailures as 1, ensuring cooldown detection uses attempt
recoveryKinds rather than only the stream status.
- Around line 191-200: Add a focused test beside “API endpoint returns the
analytics payload” covering all routing-analytics validation branches: assert
400 responses and error codes invalid_from for malformed and empty from values,
invalid_to for a non-integer to value, and invalid_range when from exceeds to.
Reuse a local request helper to exercise handleManagementAPI consistently.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 883ffcb5-b9af-4376-9352-dbf62426398e

📥 Commits

Reviewing files that changed from the base of the PR and between 2a72aa4 and e732d02.

📒 Files selected for processing (6)

• devlog/_plan/260804_router_intelligence/001_pr_stack_status.md
• src/routing/analytics.ts
• src/routing/history/indexer.ts
• src/server/management-api.ts
• src/server/management/routing-analytics-routes.ts
• tests/routing-analytics.test.ts

[Wibias]

Merged — RI-03 (routing analytics) is on dev.