You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(realtime): record a session the upstream never let start as a failure - #1412
A realtime session the upstream never let start now answers and records as a failure, whichever way it was refused. Four changes, all on /v1/realtime; a session that became usable is not touched.
An upstream that accepts the upgrade and then reports an error. This is how OpenAI answers a wrong key: it completes the 101, sends an error event, then closes with 3000 (4000 for an unknown model or the beta header). That used to read as a success: the row and metrics said 200, and the client got one extra gateway server_error frame after the upstream's. Now, when the upstream's half ends having sent an error event and no other frame:
The client gets only the upstream's frames and its close code and reason. No gateway frame. A dropped connection with no Close still closes 1011, also without a gateway frame.
The row's error_class is upstream_in_band, and error_message and the access log's error read upstream reported an in-band error: <upstream text>. The first error event is the one recorded. At content_mode: full, response holds that frame as sent. provider_request_id is read from the 101's request-id headers, on the row and the access-log line. OpenAI's 101 carried no x-request-id when measured.
status_code, the access log, and the status and outcome request-metric labels come from OpenAI's event vocabulary, since no HTTP status arrives: invalid_api_key is 401; rate_limit_exceeded and insufficient_quota are 429; any other invalid_request_error is 400 (from fix: answer video failures the way chat does, translate Azure 4xx without a Content-Type, record realtime model errors as 404 #1413 on, invalid_model and model_not_found are 404); everything else, including servers with other vocabularies, is 502. A numeric code in 400-499 is used as itself; one in 500-599 is recorded as 502, like every other 5xx. The 401 and 400 lines were measured against OpenAI. The 429 lines were not.
Cooldown takes the status-list gate when there is a status, so with the default list a wrong key cools the model as upstream_auth_failure and invalid_model or a caller's beta header doesn't. With no status it takes trigger_on_transport, as upstream_in_band_error.
A client that hangs up on the error event before the upstream closes gets the same row, so a bad key doesn't flip to a success with client timing. Frames that arrive during the close wait count too: an error then still refuses the session, and a usable frame still clears it.
aisix_deployment_*: the success sample used to be taken at the 101. It now waits for the first non-error frame or the end of the session, so a refused session counts one failure and no success. A session an upstream failure ended before any frame (a dropped connection, the idle deadline) now counts as a failure too, where it used to count as a success.
The session counts as usable from its first frame that isn't an error event: on OpenAI that's session.created, and a compatible server need not open with it. Unchanged, and pinned by tests: an error event after that frame, which OpenAI sends for a bad client event while the session stays open; a usable session that later ends on an error close, which keeps its 200 and the gateway frame; and a non-1000 close with no frames at all, which keeps its 200 and the gateway frame because the client has no other explanation.
A refused handshake's body is read in full. The dial used to see only the bytes that came with the response head, so a body arriving separately fell back to upstream returned 401 Unauthorized and lost the upstream's text, code and param. Whether that happened depended on the body's size and on timing. The handshake is still tungstenite's, and its stream now comes back on refusal, so the rest is read by the head's framing (Content-Length, chunked, or to the end of the connection), up to MAX_UPSTREAM_ERROR_BODY_BYTES, within the model's stream budget, the limit a streaming HTTP route's error body has. TLS for the dial moved out of tokio-tungstenite into upstream_dial::connect_tls, with the same rustls config and server name.
Client-visible: the frame now carries the upstream's text, code and param on these refusals.
A body past the bound reads as chat reads it: its first 1 KiB as text.
A body that stalls or is cut off gets the status fallback in the frame, and the row and access log say upstream returned HTTP <status>: failed to read response body: <reason>, where they used to say upstream returned HTTP <status> during the realtime handshake. A stalled body holds the client's frame for up to one stream budget past the dial, and with no budget (timeout: 0 and no stream_timeout) for as long as the upstream stalls and the client stays. A client that leaves ends the read at once, and the reason then reads the client closed the session.
The failure row records what the upstream answered. On a refused handshake, response at content_mode: full was synthetic text. It is now the upstream's body as sent, or what arrived of it with chunked framing decoded, and metadata_only still leaves it out. upstream_response_headers now carries the exporter's listed headers, through the same filter as HTTP routes, and provider_request_id is read from the response's request-id headers, on the row and on the access-log line. These come from the same functions HTTP routes use. An empty body keeps the synthetic text; from #1413 on, a 4xx or 5xx reads there as chat does (upstream returned HTTP <status> with no error detail in the response), and a 2xx or 3xx keeps during the realtime handshake. The row also now says the upstream was reached: upstream_latency_ms is the dial's duration where it was 0, and OTLP spans carry the server address where they had none.
A dial past upstream.connect_timeout_ms is a timeout. It recorded transport and 502, so trigger_on_timeout had no effect on a realtime dial hang. It now records error_classtimeout and 504 on the row, the access log and the metrics, reads upstream request timed out after <n>ms: ..., and takes the timeout gate, so trigger_on_timeout: false keeps the model available and trigger_on_transport: false no longer does. A kernel ETIMEDOUT on the dial reads the same way, as it does on HTTP routes. What the client gets doesn't change: the fixed failed to connect to the upstream realtime endpoint frame and close 1011. HTTP routes answer a timeout with upstream request timed out after <n>ms, so this one frame stays different.
No configuration needs editing. Alerts or dashboards that count realtime dial hangs under 502, or OpenAI wrong-key sessions under 200, will see them move to 504 and 401.
realtime-refused-handshake-e2e drives all of this against scripted upstreams: head and body written separately, the measured OpenAI sequences, and chat on the same upstream as the oracle for the frame and the row. Two existing specs pinned the old behaviour and were changed: the silent dial's status in realtime-ws-e2e and the never-arrived body's row text in failure-row-metadata-only-e2e.
…lure
An upstream that completes the upgrade and then reports an error before any
other frame (OpenAI's answer to a wrong key) is now a failure row with the
upstream's text, and the client receives only the upstream's frames. A refused
handshake's body is read in full from the stream up to the HTTP routes' bound
and time limit, and its row records the real response and headers. A dial past
connect_timeout records timeout and 504 and takes the timeout cooldown gate.
No actionable comments were generated in the recent review.
Walkthrough
Realtime upstream dialing now uses the configured connector and applies a budget to the WebSocket handshake. Refused responses can be read within configured limits. The relay tracks session opening and refusal state to determine caller errors, deployment outcomes, logs, and usage events.
Changes: Realtime upstream refusal handling
Layer / File(s)
Summary
TLS dialing and handshake budget crates/aisix-gateway/src/upstream_dial.rs, crates/aisix-proxy/src/realtime.rs, tests/e2e/src/cases/realtime-ws-e2e.test.ts, tests/e2e/src/cases/realtime-refused-handshake-e2e.test.ts
A public connect_tls helper applies TLS using the deployment configuration and removes IPv6 brackets before server-name validation. Realtime dialing uses the helper, includes the WebSocket handshake in its budget, and reports dial timeouts as 504 in the updated metrics checks.
Refused response recovery and capture crates/aisix-proxy/src/realtime.rs, tests/e2e/src/cases/realtime-refused-handshake-e2e.test.ts, tests/e2e/src/cases/failure-row-metadata-only-e2e.test.ts
The relay recovers the stream after a refused handshake and reads supported response bodies within size and timeout limits. It records refusal details, provider request IDs, and configured headers. Tests cover complete, partial, malformed, oversized, stalled, and interrupted bodies.
Session outcome and telemetry crates/aisix-proxy/src/realtime.rs, tests/e2e/src/cases/realtime-refused-handshake-e2e.test.ts
The relay tracks whether a usable session opened or an error occurred first. It applies refusal status and cooldown handling, avoids duplicate error events, and records session outcomes and usage. Tests cover in-band errors, usable sessions, closes, and drops.
sequenceDiagram
participant Caller
participant RealtimeProxy
participant UpstreamDial
participant Upstream
participant Telemetry
Caller->>RealtimeProxy: Open realtime session
RealtimeProxy->>UpstreamDial: Connect and perform TLS
UpstreamDial->>Upstream: Send WebSocket upgrade
Upstream-->>UpstreamDial: Accept or refuse upgrade
UpstreamDial-->>RealtimeProxy: Handshake result and recovered stream on refusal
RealtimeProxy->>Upstream: Read bounded refusal response when available
RealtimeProxy-->>Caller: Relay session frames or refusal error
RealtimeProxy->>Telemetry: Record session outcome and response details
Loading
Possibly related PRs
api7/aisix#1330: Adds the upstream dialer interface and HTTPS-proxy TLS setup that this change extends.
api7/aisix#1411: Changes the same realtime refused-handshake path, including recorded status and cooldown handling.
api7/aisix#1410: Changes refused-handshake body reads and upstream status handling in realtime.rs.
Merge Risk: Minimal · up to 2396c
The change records realtime sessions that the upstream refuses or never lets start as failures, and bounds the dial and the refused-body read. No actionable merge-blocking issue was found. Ordinary CI and the new e2e suite are sufficient checks before merge.
Pre-merge checks: 6
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title names the main change: a realtime session that the upstream refuses before it starts is now recorded as a failure. It uses the conventional fix(realtime) prefix. The raw summary and PR objec…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
E2e Test Quality Review
The PR adds full E2E coverage for the changed realtime flows. The new suite exercises delayed, chunked, EOF, oversized, stalled, truncated, client-aborted, and silent-dial failures. It also verifies i…
Security Check
No security-check failure is introduced by this pull request. - Category 1: No listed credential is logged or returned. Refused-response headers use the existing allowlist, which excludes authorizatio…
Full details: Title check
Explanation
The title names the main change: a realtime session that the upstream refuses before it starts is now recorded as a failure. It uses the conventional fix(realtime) prefix. The raw summary and PR objectives both describe this behavior change for refused sessions and error events before the first non-error frame, so the title matches the changeset.
Full details: E2e Test Quality Review
Explanation
The PR adds full E2E coverage for the changed realtime flows. The new suite exercises delayed, chunked, EOF, oversized, stalled, truncated, client-aborted, and silent-dial failures. It also verifies in-band refusal mapping, usable-session behavior, deployment samples, cooldown, metrics, SLS rows, headers, request IDs, and metadata-only exports. Test cases use isolated model names and request IDs, so they do not depend on execution order. The implementation changes remain within realtime dialing, refusal handling, telemetry, and related expectations. Existing realtime proxy E2E coverage also exercises the TLS and HTTPS-proxy paths.
Full details: Security Check
Explanation
No security-check failure is introduced by this pull request. - Category 1: No listed credential is logged or returned. Refused-response headers use the existing allowlist, which excludes authorization, cookie, API-key, and proxy-credential headers. Provider request IDs are sanitized. Error bodies are bounded and full-content capture remains opt-in. - Category 2: No database model, migration, or plaintext secret write changed. - Category 3: No mutating endpoint or permission check changed. - Category 4: No cross-resource lookup or ownership check changed. - Category 5: TLS uses the shared deployment Rustls configuration. The new helper preserves certificate verification semantics and removes IPv6 brackets before SNI parsing. - Category 6: No shared-resource deletion, binding, or cascade operation changed. - Category 7: No environment or secret-reference resolution path changed. The changed code records upstream error details and filtered response metadata for /v1/realtime, but the inspected paths do not include API keys, bearer tokens, cookies, private keys, or authentication headers in those values.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Essentials
Run ID: 7746e4a6-8c7f-4670-a9ae-a8b80b29f6d0
Commits
Reviewing files that changed from the base of the PR and between c1e6bde and 2396cf6.
Included review availability: This review used your included allowance. Your plan provides up to 5 included reviews per hour; 0 remain after this review.
…sample
The body read ends when the client leaves, keeps what arrived, and decodes a
chunked body's framing out of the row. A refused handshake's row says the
upstream was reached. A refused session's row and line carry the upgrade's
provider request id, frames that arrive while closing still decide whether the
session was refused, and a session an upstream failure ended before any frame
counts as a deployment failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A realtime session the upstream never let start now answers and records as a failure, whichever way it was refused. Four changes, all on
/v1/realtime; a session that became usable is not touched.An upstream that accepts the upgrade and then reports an error. This is how OpenAI answers a wrong key: it completes the
101, sends anerrorevent, then closes with3000(4000for an unknown model or the beta header). That used to read as a success: the row and metrics said 200, and the client got one extra gatewayserver_errorframe after the upstream's. Now, when the upstream's half ends having sent anerrorevent and no other frame:1011, also without a gateway frame.error_classisupstream_in_band, anderror_messageand the access log'serrorreadupstream reported an in-band error: <upstream text>. The first error event is the one recorded. Atcontent_mode: full,responseholds that frame as sent.provider_request_idis read from the101's request-id headers, on the row and the access-log line. OpenAI's101carried nox-request-idwhen measured.status_code, the access log, and thestatusandoutcomerequest-metric labels come from OpenAI's event vocabulary, since no HTTP status arrives:invalid_api_keyis 401;rate_limit_exceededandinsufficient_quotaare 429; any otherinvalid_request_erroris 400 (from fix: answer video failures the way chat does, translate Azure 4xx without a Content-Type, record realtime model errors as 404 #1413 on,invalid_modelandmodel_not_foundare 404); everything else, including servers with other vocabularies, is 502. A numericcodein 400-499 is used as itself; one in 500-599 is recorded as 502, like every other 5xx. The 401 and 400 lines were measured against OpenAI. The 429 lines were not.upstream_auth_failureandinvalid_modelor a caller's beta header doesn't. With no status it takestrigger_on_transport, asupstream_in_band_error.aisix_deployment_*: the success sample used to be taken at the101. It now waits for the first non-error frame or the end of the session, so a refused session counts one failure and no success. A session an upstream failure ended before any frame (a dropped connection, the idle deadline) now counts as a failure too, where it used to count as a success.The session counts as usable from its first frame that isn't an
errorevent: on OpenAI that'ssession.created, and a compatible server need not open with it. Unchanged, and pinned by tests: anerrorevent after that frame, which OpenAI sends for a bad client event while the session stays open; a usable session that later ends on an error close, which keeps its 200 and the gateway frame; and a non-1000 close with no frames at all, which keeps its 200 and the gateway frame because the client has no other explanation.A refused handshake's body is read in full. The dial used to see only the bytes that came with the response head, so a body arriving separately fell back to
upstream returned 401 Unauthorizedand lost the upstream's text,codeandparam. Whether that happened depended on the body's size and on timing. The handshake is still tungstenite's, and its stream now comes back on refusal, so the rest is read by the head's framing (Content-Length, chunked, or to the end of the connection), up toMAX_UPSTREAM_ERROR_BODY_BYTES, within the model's stream budget, the limit a streaming HTTP route's error body has. TLS for the dial moved out of tokio-tungstenite intoupstream_dial::connect_tls, with the same rustls config and server name.codeandparamon these refusals.upstream returned HTTP <status>: failed to read response body: <reason>, where they used to sayupstream returned HTTP <status> during the realtime handshake. A stalled body holds the client's frame for up to one stream budget past the dial, and with no budget (timeout: 0and nostream_timeout) for as long as the upstream stalls and the client stays. A client that leaves ends the read at once, and the reason then readsthe client closed the session.Content-Lengthand no chunking used to fall back; it is now quoted once the connection ends.The failure row records what the upstream answered. On a refused handshake,
responseatcontent_mode: fullwas synthetic text. It is now the upstream's body as sent, or what arrived of it with chunked framing decoded, andmetadata_onlystill leaves it out.upstream_response_headersnow carries the exporter's listed headers, through the same filter as HTTP routes, andprovider_request_idis read from the response's request-id headers, on the row and on the access-log line. These come from the same functions HTTP routes use. An empty body keeps the synthetic text; from #1413 on, a 4xx or 5xx reads there as chat does (upstream returned HTTP <status> with no error detail in the response), and a 2xx or 3xx keepsduring the realtime handshake. The row also now says the upstream was reached:upstream_latency_msis the dial's duration where it was 0, and OTLP spans carry the server address where they had none.A dial past
upstream.connect_timeout_msis a timeout. It recordedtransportand 502, sotrigger_on_timeouthad no effect on a realtime dial hang. It now recordserror_classtimeoutand 504 on the row, the access log and the metrics, readsupstream request timed out after <n>ms: ..., and takes the timeout gate, sotrigger_on_timeout: falsekeeps the model available andtrigger_on_transport: falseno longer does. A kernelETIMEDOUTon the dial reads the same way, as it does on HTTP routes. What the client gets doesn't change: the fixedfailed to connect to the upstream realtime endpointframe and close1011. HTTP routes answer a timeout withupstream request timed out after <n>ms, so this one frame stays different.No configuration needs editing. Alerts or dashboards that count realtime dial hangs under 502, or OpenAI wrong-key sessions under 200, will see them move to 504 and 401.
realtime-refused-handshake-e2edrives all of this against scripted upstreams: head and body written separately, the measured OpenAI sequences, and chat on the same upstream as the oracle for the frame and the row. Two existing specs pinned the old behaviour and were changed: the silent dial's status inrealtime-ws-e2eand the never-arrived body's row text infailure-row-metadata-only-e2e.🤖 Generated with Claude Code
Summary by CodeRabbit