feat(translation): preserve raw stream events

[bbednarski9]

What

This PR adds a composed streaming-response event to Switchyard so a host can send a provider response through libsy::Algorithm::run_stream without discarding provider-specific JSON fields.

The implementation:

• keeps LlmResponseChunk provider-neutral;
• changes LlmResponseStream to carry LlmResponseStreamEvent;
• represents each stream item as normalized chunks plus optional ProviderStreamEvent { source, raw } preservation;
• exposes preserving decode and encode operations through TranslationEngine, the API used by the concrete downstream Relay consumer;
• replays raw when the source and target formats match;
• encodes normalized chunks when the target format differs;
• centralizes the replay-versus-translation decision above provider codecs;
• folds normalized chunks during aggregate conversion while preserving typed decode and upstream-stream errors; and
• covers OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages with same-format and cross-format contract tests.

The fields are intentionally accessed through constructors and accessors. Replacing normalized content creates a normalized-only event and drops preservation, preventing stale provider JSON from being replayed after an algorithm changes the semantic content.

This is intentionally limited to the streaming translation contract.

Lifecycle

sequenceDiagram
    participant Provider
    participant Host as Host / Relay
    participant Libsy as libsy run_stream
    participant Caller

    Provider-->>Host: Raw JSON stream event<br/>text delta + provider extension
    Host->>Host: Decode to LlmResponseStreamEvent<br/>preservation + normalized chunks
    Host-->>Libsy: Respond to CallLlm with event stream
    Note over Libsy: Algorithms inspect or aggregate<br/>provider-neutral normalized chunks
    Libsy-->>Host: ReturnToAgent(event stream)

    alt Caller format matches provider format
        Host-->>Caller: Replay preserved raw JSON
    else Caller format differs
        Host->>Host: Encode normalized chunks<br/>into caller format
        Host-->>Caller: Cross-format stream event
    end

Loading

Why

run_stream deliberately gives the embedding host control of provider dispatch. The host fulfills each CallLlm, but the resulting stream still passes back through libsy so an algorithm can inspect it, aggregate it, or select it as the final response.

Before this change, decoding a provider event produced only provider-neutral chunks. That is sufficient for cross-format translation, but it cannot faithfully replay a same-format stream after the libsy round trip.

For example, an OpenAI-compatible provider may emit:

{
  "id": "chatcmpl-1",
  "object": "chat.completion.chunk",
  "system_fingerprint": "fp_abc",
  "choices": [
    {
      "index": 0,
      "delta": {"content": "Hello"},
      "finish_reason": null
    }
  ]
}

Switchyard normalizes the semantic content to approximately:

LlmResponseChunk::TextDelta {
    index: 0,
    text: "Hello".into(),
}

That normalized chunk has no representation for system_fingerprint. The composed stream item carries both:

LlmResponseStreamEvent {
    preservation: Some(ProviderStreamEvent {
        source: FormatId::from(WireFormat::OpenAiChat),
        raw: provider_event,
    }),
    normalized: vec![text_delta],
}

Algorithms consume the normalized meaning. switchyard-translation alone interprets preservation: it replays the parsed raw JSON for same-format output and ignores it when encoding a different protocol.

The concrete downstream consumer is NVIDIA/NeMo-Relay#586. Its streaming adapter calls TranslationEngine::decode_stream_event for each host-dispatched provider event and TranslationEngine::encode_stream_event after ReturnToAgent; a Switchyard backend, server route, or PyO3 binding is therefore not required for this contract.

This enables library-first integrations such as NeMo Relay to:

1. run a Switchyard algorithm in process;
2. perform the actual provider call in the host;
3. return the real response stream through CallLlmRequest::respond;
4. let libsy continue through ReturnToAgent; and
5. preserve provider extensions when the final wire format is unchanged.

For cross-format routes, only fields represented by the neutral IR are translated. In the example above, OpenAI system_fingerprint is intentionally absent from an Anthropic Messages output because Anthropic has no corresponding field.

Boundaries and compatibility

• Preservation is exact for the parsed serde_json::Value, not for the original SSE bytes. Comments, SSE id fields, whitespace, and JSON key ordering are not retained.
• Provider-only fields are replayed only for same-format output. They are never injected into a different provider protocol.
• LlmResponseChunk remains provider-neutral. Preservation is composed beside normalized chunks at the LlmResponseStreamEvent boundary.
• The LlmResponseStream item type changes from LlmResponseChunk to LlmResponseStreamEvent, so downstream stream producers and consumers must wrap or unwrap normalized chunks.
• Algorithms using aggregate conversion continue to see normalized content because aggregation folds every event's normalized chunks.
• An algorithm that replaces normalized content must also drop preservation; the provided replace_normalized API enforces that behavior.

Closes: N/A — this is an integration-discovered library contract gap.

How tested

• uv run ruff check . clean — not applicable; no Python changed
• uv run mypy switchyard clean — not applicable; no Python changed
• uv run pytest tests/ green — not applicable; no Python changed
• Manual smoke (described below)

Rust validation:

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets -- -D warnings
• cargo doc --workspace --no-deps
• cargo test -p switchyard-protocol
• cargo test -p switchyard-translation
• cargo test -p switchyard-libsy
• cargo test -p switchyard-llm-client
• cargo test -p switchyard-server
• cargo test --workspace --exclude switchyard-py

The same-format helper-path test sends an OpenAI Chat stream event containing system_fingerprint through decode_stream and encode_stream and verifies exact parsed-JSON replay. Cross-format tests verify that normalized content is encoded instead.

The non-PyO3 workspace suite passes. A local cargo test --workspace attempt reached switchyard-py but could not link the macOS test artifact because the local linker did not resolve Python symbols; no Rust test failed before that link step.

Integration validation used NeMo Relay as the host-side dispatcher with no Switchyard service or switchyard-llm-client. Buffered and streaming routes were exercised across OpenAI Chat, OpenAI Responses, and Anthropic Messages caller APIs, using NVIDIA Inference Hub, Anthropic, and Ollama targets.

Checklist

• One class per file; filename = snake_case of the primary class. — not applicable; Rust-only change
• New public symbols exported from switchyard/__init__.py.__all__ if intended for downstream use. — not applicable; Rust crate API
• Unit tests added for new components / bug fixes.
• README / --help updated if customer-facing surface changed. — not required; no CLI/configuration change
• Commits signed off (Signed-off-by: Your Name <email>) per the DCO.

Notes for reviewers

Please focus on the ownership boundary between preservation and normalized, and on the LlmResponseStreamEvent host/algorithm contract before Switchyard 0.2 stabilizes. The same-format path never reconstructs an event from the normalized IR; the cross-format path never copies unknown provider fields into the target protocol.

Preservation decoding is exposed through TranslationEngine and the stream helper used by switchyard-llm-client. Both encoding surfaces delegate to the same shared preservation dispatcher. Provider codecs handle only normalized chunks, including when the engine uses a custom registered codec.

Summary by CodeRabbit

• New Features

  • Added composed streaming-event preservation so same-format events retain provider-specific data.
  • Added streaming decode and encode APIs for translating individual events between supported formats.
  • Improved cross-format streaming translation using provider-neutral normalized content.

• Bug Fixes

  • Prevented duplicate terminal output after replaying a terminal provider event.
  • Improved handling and reporting of errors occurring during streamed provider events.

[bbednarski9]

@CodeRabbit review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Walkthrough

Changes

Provider Event Streaming

Layer / File(s)	Summary
Provider event aggregation crates/protocol/src/stream.rs	Adds provider-event chunks with raw and normalized data, recursively aggregates normalized children, translates nested stream errors, and tests both behaviors.
Preserving stream translation APIs crates/switchyard-translation/src/codecs/stream.rs, crates/switchyard-translation/src/engine.rs	Adds streaming decode and encode APIs that preserve same-format events and recursively translate normalized chunks across formats.
Provider codec handling and tests crates/switchyard-translation/src/codecs/*/stream.rs, crates/switchyard-translation/tests/stream_translation.rs	Adds provider-specific passthrough or normalized encoding for Anthropic, OpenAI Chat, and OpenAI Responses, with round-trip and cross-format coverage.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Poem

I’m a rabbit with events in my ears,
Keeping raw streams safe through the years.
Normalized hops cross each codec’s lane,
Errors get names, then vanish like rain.
Same-source crumbs replay bright and clear!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately captures the main change: preserving raw stream events in translation.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

crates/switchyard-translation/tests/stream_translation.rs (1)

16-69: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add concise comments documenting the preservation contracts under test.

• crates/switchyard-translation/tests/stream_translation.rs#L16-L69: note that equality is for the parsed JSON value replayed to the same provider format.
• crates/switchyard-translation/tests/stream_translation.rs#L71-L97: note that cross-format encoding must discard provider-specific fields and use normalized events.

As per coding guidelines, Rust changes require concise comments for important test behavior.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-translation/tests/stream_translation.rs` around lines 16 -
69, Add concise comments documenting the two preservation contracts in
stream_translation.rs: at lines 16-69, clarify that same-format replay compares
the parsed JSON value emitted back to the same provider format; at lines 71-97,
clarify that cross-format encoding uses normalized events and discards
provider-specific fields.

Source: Coding guidelines

crates/protocol/src/stream.rs (1)

78-101: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Document the raw-replay versus normalized-routing invariant.

• crates/protocol/src/stream.rs#L78-L101: explain recursive error propagation through normalized children.
• crates/switchyard-translation/src/engine.rs#L298-L316: document same-format raw replay versus cross-format normalized encoding.
• crates/switchyard-translation/src/codecs/anthropic/stream.rs#L129-L137: annotate the ProviderEvent routing branch.
• crates/switchyard-translation/src/codecs/openai_chat/stream.rs#L157-L165: annotate the ProviderEvent routing branch.
• crates/switchyard-translation/src/codecs/responses/stream.rs#L157-L165: annotate the ProviderEvent routing branch.

As per coding guidelines, Rust changes require concise comments for non-obvious private helpers and complex routing logic.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/protocol/src/stream.rs` around lines 78 - 101, Document the raw-replay
versus normalized-routing invariant with concise comments: in
crates/protocol/src/stream.rs lines 78-101, explain that push_checked_chunk
recursively processes normalized ProviderEvent children and propagates their
errors; in crates/switchyard-translation/src/engine.rs lines 298-316, describe
raw replay for same-format responses versus normalized encoding across formats;
and annotate the ProviderEvent routing branches in
crates/switchyard-translation/src/codecs/anthropic/stream.rs lines 129-137,
crates/switchyard-translation/src/codecs/openai_chat/stream.rs lines 157-165,
and crates/switchyard-translation/src/codecs/responses/stream.rs lines 157-165.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/protocol/src/stream.rs`:
- Around line 103-108: The documentation overstates preservation as exact or
lossless replay; describe it as retaining and replaying parsed JSON values only.
Update the documentation in crates/protocol/src/stream.rs lines 103-108,
crates/switchyard-translation/src/codecs/stream.rs line 217, and
crates/switchyard-translation/src/engine.rs lines 247 and 264-268 to use
parsed-event, parsed-value, and structural JSON-value replay wording, while
preserving the existing API behavior.

---

Nitpick comments:
In `@crates/protocol/src/stream.rs`:
- Around line 78-101: Document the raw-replay versus normalized-routing
invariant with concise comments: in crates/protocol/src/stream.rs lines 78-101,
explain that push_checked_chunk recursively processes normalized ProviderEvent
children and propagates their errors; in
crates/switchyard-translation/src/engine.rs lines 298-316, describe raw replay
for same-format responses versus normalized encoding across formats; and
annotate the ProviderEvent routing branches in
crates/switchyard-translation/src/codecs/anthropic/stream.rs lines 129-137,
crates/switchyard-translation/src/codecs/openai_chat/stream.rs lines 157-165,
and crates/switchyard-translation/src/codecs/responses/stream.rs lines 157-165.

In `@crates/switchyard-translation/tests/stream_translation.rs`:
- Around line 16-69: Add concise comments documenting the two preservation
contracts in stream_translation.rs: at lines 16-69, clarify that same-format
replay compares the parsed JSON value emitted back to the same provider format;
at lines 71-97, clarify that cross-format encoding uses normalized events and
discards provider-specific fields.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f57ca300-512e-41f4-92bf-4be3ad0a3101

📥 Commits

Reviewing files that changed from the base of the PR and between 27a9af0 and 773a451.

📒 Files selected for processing (7)

• crates/protocol/src/stream.rs
• crates/switchyard-translation/src/codecs/anthropic/stream.rs
• crates/switchyard-translation/src/codecs/openai_chat/stream.rs
• crates/switchyard-translation/src/codecs/responses/stream.rs
• crates/switchyard-translation/src/codecs/stream.rs
• crates/switchyard-translation/src/engine.rs
• crates/switchyard-translation/tests/stream_translation.rs

[grahamking]

@bbednarski9 Can you have Claude run the switchyard-rust-review skill against it? I get a few hits.

1. Replay + finish_stream emits a duplicate terminal sequence. crates/switchyard-translation/src/engine.rs:305
2. Speculative API with zero callers. Six new public surfaces, four copies of dispatch logic, one new protocol enum variant — and no backend, server route, or PyO3 binding constructs a ProviderEvent.
3. Protocol type is no longer provider-neutral. crates/protocol/src/stream.rs:112
4. The same dispatch is written four times. engine.rs:298, anthropic/stream.rs:129, openai_chat/stream.rs:157, responses/stream.rs:157

I'm not sure if those are relevant.

And more minor and style nits.

[bbednarski9]

Addressed the three remaining review findings in separate commits:

• 5c8de7b0 centralizes preserved-event replay and cross-format dispatch above provider codecs. Both TranslationEngine and the existing built-in encoder helper use that path, and a custom-codec regression test proves preservation does not depend on built-in codec arms.
• d9d51f82 removes the unused free preserving decoder. The remaining preserving API is TranslationEngine, with NVIDIA/NeMo-Relay#586 as its concrete downstream consumer; Relay calls it from the host-owned provider streaming path, so no Switchyard backend, server route, or PyO3 caller is expected.
• 643733b9 documents ProviderEvent as an intentional host/algorithm transport envelope. source and raw are opaque preservation data interpreted only by switchyard-translation; algorithms and cross-format encoding consume the normalized children. This mirrors buffered PreservationMetadata without making provider fields semantic IR.

Validation: workspace Clippy with warnings denied passed; all workspace tests excluding switchyard-py passed. The full workspace test reached switchyard-py but the local macOS linker could not resolve Python symbols.

[bbednarski9]

Addressed the algebraic-type review in 445f999: LlmResponseChunk is provider-neutral again, and LlmResponseStream now carries LlmResponseStreamEvent { preservation, normalized }. Same-format output replays ProviderStreamEvent.raw; cross-format output encodes only normalized chunks. The shared dispatcher remains above provider codecs, aggregate/server/libsy consumers were migrated, and replacing normalized content drops stale preservation. Validation: workspace Clippy with warnings denied, workspace docs, all affected crate suites, and cargo test --workspace --exclude switchyard-py.