feat: route with Switchyard libsy in process

[bbednarski9]

Overview

Replaces the service-backed Switchyard plugin with an in-process, library-first
integration driven by libsy::Algorithm::run_stream. Relay remains responsible
for provider dispatch, authentication, retries, fallback, and observability.

The draft supports both weighted random routing and libsy's judge-backed LLM
classifier. It remains a placeholder while the focused raw-stream preservation
dependency lands and compatible Switchyard 0.2 crates are published.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

• Drives the complete run_stream lifecycle. Switchyard requests provider work,
  Relay dispatches every CallLlm, and Relay returns the real buffered response,
  stream, or provider error through CallLlmRequest::respond.
• Supports weighted, reproducibly seeded random routing.
• Adds kind = "llm_classifier" with classifier, weak, and strong semantic
  targets; solve/confidence thresholds; optional capability floor; session
  affinity; and message-hash fallback.
• Dispatches both classifier consultation and final routed calls through Relay's
  configured provider bindings. The plugin does not use
  switchyard-llm-client or switchyard-server.
• Supports OpenAI Chat and OpenAI Responses classifier targets. Weak and strong
  targets may use OpenAI Chat, OpenAI Responses, or Anthropic Messages.
• Uses switchyard-translation as the sole request, response, and stream
  translation layer. The plugin does not fall back to Relay provider codecs.
• Starts a fresh libsy run for each retry. A failed classifier consultation
  follows libsy's fail-closed strong-target behavior; only a failed final routed
  call participates in Relay retry classification.
• Emits requested, call, decision, retry, error, and fallback routing marks from
  genuine libsy lifecycle data. Classifier consultations are marked
  is_routed_call = false.
• Adds buffered, streaming, cross-protocol, retry, affinity, fail-closed,
  telemetry, process-boundary, and standalone no-service classifier coverage.
• Removes the Decision API client, service health checks, service startup
  examples, dedicated Switchyard ATOF receiver, and service-backed version-1
  configuration.
• Temporarily pins switchyard-libsy, switchyard-protocol, and
  switchyard-translation to
  aa0a31126cc7ca922a6acdf4f1ef656744698c6c, the rebased source behind
  feat(translation): preserve raw stream events NVIDIA-NeMo/Switchyard#192.

Breaking change: the Switchyard plugin no longer supports service-backed
version-1 configuration.

Current upstream dependency:

• fix(libsy): preserve classifier system instruction NVIDIA-NeMo/Switchyard#195 is merged and resolves LIBSY-GAP-005. The
  classifier now places its governing policy in LlmRequest.instructions, so
  OpenAI Chat receives a system message and OpenAI Responses receives its
  native instructions field.
• The focused Relay contract probe is active and passing for both supported
  classifier protocols.
• The gateway process E2E asserts that the provider receives the classifier
  policy with role system, followed by the task with role user, for both
  buffered and streaming classifier runs.
• Switchyard added TaskClassifierConfig.recent_turn_window after the original
  development pin. Relay explicitly sets it to None to preserve the existing
  classifier behavior; exposing it as Relay configuration is outside this
  compatibility update.
• feat(translation): preserve raw stream events NVIDIA-NeMo/Switchyard#192 remains the only source-level upstream dependency.
  After it lands, the fork pin can move to upstream main; compatible versioned
  crates will replace source pins after Switchyard publishes them.

Validation:

• cargo test -p nemo-relay-switchyard — 32 passed
• cargo test -p nemo-relay-cli --features switchyard --test switchyard_process_e2e — 2 passed
• cargo test -p nemo-relay-ffi -- --test-threads=1 — 155 passed
• cargo clippy --workspace --all-targets -- -D warnings
• cargo fmt --all
• targeted uv run pre-commit run --files ...
• uv run pre-commit run --all-files
• examples/switchyard/run-classifier-e2e.sh

Known repository-wide validation issue: just test-rust is not green in this
checkout. It reproduces the existing release/0.6 plugin-registry ordering
failure (pii_redaction is not registered, followed by poisoned shared
registries): 895 core tests pass before 77 related failures. The focused
Switchyard and serialized FFI paths are green, and this PR's previous
cross-platform CI matrix passed.

Where should the reviewer start?

Start with crates/switchyard/src/component.rs for algorithm construction and
the generic run_stream driver. Then review
classifier_run_stream_offloads_the_judge_and_routed_calls_to_relay,
classifier_prompt_is_encoded_as_a_system_instruction, and
switchyard_plugin_runs_classifier_and_routed_calls_without_a_service for the
library/Relay ownership and provider-boundary contracts.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to feat(translation): preserve raw stream events NVIDIA-NeMo/Switchyard#192.
• Relates to fix(libsy): preserve classifier system instruction NVIDIA-NeMo/Switchyard#195.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 57d74e5c-c89f-4992-8693-cc5a1db2b298

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-586.docs.buildwithfern.com/nemo/relay/dev

[github-actions]

License Diff

Compared against origin/release/0.6.

Lockfile license changes

Lockfile License Changes

Rust

Added

• switchyard-libsy 0.1.0 (Apache-2.0)

Removed

• None

Updated/Changed

opentelemetry

Before:

• opentelemetry 0.31.0 (Apache-2.0)

After:

• opentelemetry 0.31.0 (Apache-2.0)
• opentelemetry 0.32.0 (Apache-2.0)

rand

Before:

• rand 0.10.1 (Apache-2.0)
• rand 0.9.3 (Apache-2.0)

After:

• rand 0.10.1 (Apache-2.0)
• rand 0.8.7 (Apache-2.0)
• rand 0.9.3 (Apache-2.0)

rand_chacha

Before:

• rand_chacha 0.9.0 (Apache-2.0)

After:

• rand_chacha 0.3.1 (Apache-2.0)
• rand_chacha 0.9.0 (Apache-2.0)

rand_core

Before:

• rand_core 0.10.1 (Apache-2.0)
• rand_core 0.9.5 (Apache-2.0)

After:

• rand_core 0.10.1 (Apache-2.0)
• rand_core 0.6.4 (Apache-2.0)
• rand_core 0.9.5 (Apache-2.0)

Node

Added

• None

Removed

• None

Updated/Changed

• None

Python

Added

• None

Removed

• None

Updated/Changed

• None

Status output

[license-diff] selected languages: rust, node, python
[license-diff] generating current inventory
[license-diff] current: generating Rust inventory
[license-diff] current: Rust inventory complete (431 packages)
[license-diff] current: generating Node inventory
[license-diff] current: Node inventory complete (367 packages)
[license-diff] current: generating Python inventory
[license-diff] current: Python inventory complete (105 packages)
[license-diff] current inventory complete
[license-diff] checking out base ref origin/release/0.6 into a temporary worktree
[license-diff] base: generating Rust inventory
[license-diff] base: Rust inventory complete (426 packages)
[license-diff] base: generating Node inventory
[license-diff] base: Node inventory complete (367 packages)
[license-diff] base: generating Python inventory
[license-diff] base: Python inventory complete (105 packages)
[license-diff] base inventory complete
[license-diff] removing temporary base worktree
[license-diff] comparing inventories
[license-diff] rendering Markdown output
[license-diff] done

[bbednarski9]

Superseded by the generic native C ABI v2 host/SDK in #594 and the external Switchyard run_stream plugin in NVIDIA-NeMo/Switchyard#220. Parity is now proven with fake-provider process E2E plus a live Inference Hub smoke across two models (8 buffered, 4 streaming); the resulting Phoenix trace contains 12 switchyard.routing.requested and 12 switchyard.routing.decision events: http://10.193.162.160:6006/projects/UHJvamVjdDoxOA==/traces. This PR remains the behavioral reference for the static implementation.