fix(api): dont 500 on cross dc response

[MasterPtato]

Fixes RVT-6105

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[railway-app]

🚅 Deployed to the rivet-pr-4449 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
frontend-cloud	✅ Success (View Logs)	Web	Mar 31, 2026 at 8:39 am
frontend-inspector	😴 Sleeping (View Logs)	Web	Mar 19, 2026 at 12:12 pm
website	❌ Build Failed (View Logs)	Web	Mar 17, 2026 at 10:41 pm
mcp-hub	✅ Success (View Logs)	Web	Mar 17, 2026 at 10:40 pm
ladle	❌ Build Failed (View Logs)	Web	Mar 17, 2026 at 10:39 pm

[MasterPtato]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• chore: envoy client #4521 : 2 dependent PRs (#4531 , #4534 )
• feat(pb): actors v3 #4463
• feat(ups): implement queue subs #4486
• feat(cache): add in flight deduping #4459
• fix(cache): clean up lib #4457
• fix(config): allow configuring topo dcs via map, fix pg ssl mode config #4456
• fix(api): dont 500 on cross dc response #4449 👈 (View in Graphite)
• chore: update byte units on frontend #4444
• fix: runner alloc idx logic, api auth for actor get #4443
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[linear]

RVT-6105 Cross-DC proxy escalates 400 client errors to 500 internal errors

[pkg-pr-new]

More templates

• actor-actions
• actor-actions-vercel
• ai-agent
• ai-agent-vercel
• example-ai-and-user-generated-actors-freestyle
• chat-room
• chat-room-vercel
• example-cloudflare-workers
• example-cloudflare-workers-hono
• collaborative-document
• collaborative-document-vercel
• cross-actor-actions
• cross-actor-actions-vercel
• example-cursors
• example-cursors-raw-websocket
• example-cursors-raw-websocket-vercel
• example-cursors-vercel
• custom-serverless
• example-elysia
• example-experimental-durable-streams-ai-agent
• example-experimental-durable-streams-ai-agent-vercel
• geo-distributed-database
• geo-distributed-database-vercel
• hello-world
• hello-world-vercel
• example-hono
• hono-react
• hono-react-vercel
• example-kitchen-sink
• example-kitchen-sink-vercel
• multi-region-vercel
• multiplayer-game
• example-multiplayer-game-patterns
• example-multiplayer-game-patterns-vercel
• multiplayer-game-vercel
• native-websockets
• native-websockets-vercel
• next-js
• per-tenant-database
• per-tenant-database-vercel
• example-raw-fetch-handler
• example-raw-fetch-handler-vercel
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-raw-websocket-handler-proxy-vercel
• example-raw-websocket-handler-vercel
• example-react
• example-react-vercel
• sandbox
• sandbox-coding-agent
• sandbox-coding-agent-vercel
• sandbox-vercel
• scheduling
• scheduling-vercel
• example-sqlite-drizzle
• example-sqlite-raw
• state
• state-vercel
• example-stream
• example-stream-vercel
• example-trpc

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@4449

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@4449

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@4449

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@4449

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@4449

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@4449

@rivetkit/sqlite-vfs

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sqlite-vfs@4449

@rivetkit/traces

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/traces@4449

@rivetkit/workflow-engine

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/workflow-engine@4449

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@4449

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@4449

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@4449

commit: 90bceb5

[claude]

PR Review: fix(api): dont 500 on cross dc response

Summary

This is a minimal, well-targeted fix in engine/packages/api-util/src/lib.rs in reqwest_to_axum_response.

Before: When a remote DC returned a non-2xx response, anyhow::bail! propagated an Err up the call stack, causing the API handler to respond with HTTP 500.

After: Logs the error via tracing::error! and falls through to return the actual remote response, preserving the original status code and body.

---

Feedback

Correctness: ✅

The fix is correct. reqwest_to_axum_response should convert a reqwest response to an axum response — a non-2xx upstream status is not a programming error, it's a valid response to be forwarded. Bailing here was the wrong abstraction.

Logging: ✅

The structured logging conversion is well done and aligns with CLAUDE.md: "Do not format parameters into the main message, instead use tracing's structured logging."

Minor observation:

?status (Debug) prints as StatusCode(404) while %status (Display) would print as 404 Not Found, which is slightly more readable. Not blocking, just worth noting.

---

Verdict

✅ Looks good to merge. Simple, correct fix with minimal blast radius.

[claude]

PR Review: fix(api) dont 500 on cross dc response

This is a small, focused bug fix that correctly addresses the issue of cross-DC responses being converted to 500 errors.

Change Summary: In reqwest_to_axum_response, when a remote DC returned a non-2xx response, anyhow::bail!() was called, which propagated as an internal server error (500) instead of forwarding the actual response status and body to the client. The fix logs the error and allows the response to pass through with its original status code.

Whats Good: reqwest_to_axum_response is intended as a raw pass-through so forwarding the original status code and body is the right approach. The new tracing::error! call correctly follows the project logging conventions. The body_bytes borrow via String::from_utf8_lossy is correctly scoped within the if block.

Minor Suggestions: (1) Consider using %status (Display) instead of ?status (Debug) for the status code - Display outputs 404 Not Found while Debug adds extra formatting noise. (2) tracing::error! is reasonable but tracing::warn! might reduce log noise if 404s are expected in normal operation. (3) The checklist items in the PR description are unchecked - worth filling in for completeness.

Overall this is a correct and minimal fix. LGTM.

[claude]

PR Review: fix(api): dont 500 on cross dc response

Summary: This is a clean, targeted fix. When a cross-DC proxy response returns a non-2xx status, the original code bailed with an anyhow error that propagated as a 500. The fix logs the error structurally and passes the original response through instead.

What's good

• The fix correctly preserves the remote DC's status code and body for the caller, which is the right behavior for a transparent proxy function.
• The structured logging (?status, ?ray_id, %body_text) follows the repo's tracing conventions.
• The log message is lowercase and uses structured fields rather than interpolation — matches CLAUDE.md style guidelines.
• Small, focused change with minimal blast radius.

Observations

Log level consideration: Using tracing::error! for all non-2xx responses may be noisy in practice. 4xx responses (e.g. 404, 400, 401) are client errors that are expected in normal operation and typically shouldn't trigger error-level alerts. Consider using tracing::warn! for 4xx and reserving tracing::error! for 5xx responses:

if status.is_client_error() {
    tracing::warn!(?status, ?ray_id, %body_text, "remote dc returned client error");
} else {
    tracing::error!(?status, ?ray_id, %body_text, "remote dc returned server error");
}

This is a minor nit — the current approach is still correct and may be intentional if all non-2xx responses from peer DCs are considered unexpected.

No test coverage needed: Given this is a proxy passthrough fix, the behavior is straightforward and doesn't require additional tests.

Overall this is a solid fix — approve.

[claude]

PR Review: fix(api): dont 500 on cross dc response

Overall: LGTM with one minor suggestion.

The fix is correct. reqwest_to_axum_response is a proxy/passthrough function, so when a remote DC returns a non-2xx response, the old anyhow::bail! was incorrectly converting that into a 500 from the proxying service itself. The response should be forwarded as-is to the caller.

Positive:

• The new tracing::error! call correctly uses structured fields (?status, ?ray_id, %body_text) and a lowercase message, matching the project's logging conventions.
• The response body is correctly preserved and forwarded after the log.

Suggestion: Log level for 4xx responses

Using error! for all non-2xx responses will fire on 4xx client errors from the remote DC (e.g., 404, 401, 422), which may be noisy or misleading since those are expected application-level errors rather than infrastructure failures. Consider differentiating by status class:

if status.is_server_error() {
    tracing::error!(?status, ?ray_id, %body_text, "remote dc returned error");
} else {
    tracing::warn!(?status, ?ray_id, %body_text, "remote dc returned non-success response");
}

This is a minor point and not a blocker — the current approach is still a clear improvement over the previous 500 behavior.

[claude]

PR Review: fix(api): dont 500 on cross dc response

The fix is correct and well-targeted. The reqwest_to_axum_response function is a raw response proxy — when a remote DC returns a 4xx/5xx, the right behavior is to forward the actual error response (preserving status code and body) back to the caller, not convert it into a 500. The previous anyhow::bail! was wrong here because it discarded the remote DC's response entirely.

Observations

%status vs ?status — Minor style point: HTTP status codes display more readably with % (Display) than ? (Debug). %body_text is correct; the status field could match:

tracing::error!(%status, ?ray_id, %body_text, "remote dc returned error");

This would log status=404 Not Found rather than the Debug representation.

Log level for 4xx — Logging all non-success responses at error! level could be noisy if clients make invalid requests that get proxied (e.g., 401, 404). Consider distinguishing 4xx (client errors) from 5xx (server errors):

if status.is_server_error() {
    tracing::error!(%status, ?ray_id, %body_text, "remote dc returned server error");
} else {
    tracing::warn!(%status, ?ray_id, %body_text, "remote dc returned client error");
}

That said, since this is internal DC-to-DC traffic, 4xx from a peer DC may legitimately indicate a problem worth surfacing as an error — so the current choice is defensible.

Contrast with parse_response — The sibling function parse_response (line 198) correctly returns Err on non-success because its callers expect a typed T and need proper error propagation. The asymmetry is intentional and correct.

Summary

The core fix is sound — replacing bail! with a log-and-continue is the right approach for a raw proxy function. The two suggestions above are minor polish items, not blockers.

[claude]

Code Review: fix(api): dont 500 on cross dc response

Summary: The PR converts a hard anyhow::bail! (which propagated as a 500 Internal Server Error) to a tracing::error! log, then continues to build and return the proxied axum::Response, preserving the original status code and body from the remote datacenter.

Correctness / Bug Fix

The fix is correct and addresses the root problem. The old code in reqwest_to_axum_response bailed out when a non-2xx status was received from a remote DC, converting any legitimate error (e.g. 400, 404, 403) into a 500 and stripping out the structured error body.

Callers of request_remote_datacenter_raw (e.g. actors/kv_get.rs, actors/delete.rs) rely on the response being passed through intact so that ApiError::into_response on the outer handler can re-serialize the RawErrorResponse with the correct status code. The fix correctly removes the bail and logs instead.

Logging Convention

The new log line follows the project's structured logging conventions correctly:

tracing::error!(?status, ?ray_id, %body_text, "remote dc returned error");

Minor nit: ?status uses the Debug formatter, which produces StatusCode(404) rather than 404 Not Found. Using %status (the Display formatter) would be more readable in logs.

Potential Information Loss

The old anyhow::bail! included the URL of the remote DC in the error string. The new log line does not log the URL. Since reqwest_to_axum_response doesn't have access to the URL at that point, this is a reasonable trade-off. The calling context already logs at tracing::debug! level in request_remote_datacenter_raw, so the tracing span chain should still allow correlating the error back to the specific endpoint.

Test Coverage

No tests are added. A regression test for the cross-DC error proxy path would be valuable, specifically asserting that a non-2xx response from a remote DC is forwarded with the correct status code and body rather than converted to a 500.

Summary

Category	Assessment
Correctness	Fix is correct and addresses the core issue
Code quality	Clean, minimal change
Logging conventions	Follows project conventions; minor nit on ?status vs %status
Information preservation	URL no longer in the error log, but available in calling span
Test coverage	No tests added; a unit test for the error-forwarding path would be valuable

Overall this is a good, targeted fix. The main actionable suggestion is to consider changing ?status to %status in the log line for more readable output.