Skip to content

feat(plugin): add NeMo Relay dynamic integration - #220

Draft
bbednarski9 wants to merge 11 commits into
NVIDIA-NeMo:mainfrom
bbednarski9:feat/nemo-relay-dynamic-plugin
Draft

feat(plugin): add NeMo Relay dynamic integration#220
bbednarski9 wants to merge 11 commits into
NVIDIA-NeMo:mainfrom
bbednarski9:feat/nemo-relay-dynamic-plugin

Conversation

@bbednarski9

@bbednarski9 bbednarski9 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

What

Adds switchyard-nemo-relay-plugin, an external NeMo Relay native dynamic
plugin that exports only Relay native C ABI v2 and drives Switchyard algorithms
through libsy::Algorithm::run_stream.

The plugin:

  • decodes inbound OpenAI Chat, OpenAI Responses, and Anthropic Messages
    requests with switchyard-translation;
  • supports weighted, reproducible random routing and the LLM-classifier router;
  • resolves every libsy CallLlm step to a semantic target and asks Relay to
    perform the actual provider dispatch;
  • feeds buffered responses, preserved provider stream events, and structured
    provider failures back through libsy until ReturnToAgent;
  • preserves Relay-owned retry, trusted fallback, cancellation, and
    first-stream-item commitment behavior;
  • emits routing requested, decision, retry, error, and fallback marks from real
    libsy steps;
  • ships a v2 manifest, configuration schema, bundle builder, fake-provider
    process E2E, and operator documentation.

It also makes run_stream executor-neutral and lazy while retaining its
optional run observer. libsy no longer requires its caller to provide a Tokio
runtime merely to poll an algorithm stream, which is required when the caller
is a host-managed dynamic-plugin callback.

The crate is a source/build unit (cdylib + rlib, publish = false). The
operator-facing artifact is a release bundle containing the shared library,
materialized manifest, schema, license notices, and SHA-256 digest.

Why

NeMo Relay must remain responsible for provider authentication, transport,
retries, fallback, and observability, while Switchyard algorithms may transform
requests, request one or more LLM calls, inspect their results, and select the
final response. A model-selection-only API cannot express classifiers or future
ensemble algorithms.

Relay native C ABI v2 provides the typed, C-safe provider-dispatch contract
needed to implement that lifecycle without linking the full Relay runtime,
using switchyard-llm-client, or running switchyard-server.

This PR is stacked on and depends on #192. PR #192 is what lets same-protocol
provider stream events survive the required Relay -> libsy -> Relay round
trip without losing unknown fields. Feature changes specific to this PR are
introduced by commits 017bea20 and 2efcb3c4; merge commit 9725b24a syncs
the branch with current main and reconciles the observer-capable run_stream
signature without changing #192.

Related: NVIDIA/NeMo-Relay#594

How tested

  • cargo test -p switchyard-libsy
  • cargo test -p switchyard-translation --test stream_translation
  • cargo test -p switchyard-nemo-relay-plugin
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo fmt --all --check
  • RUSTDOCFLAGS="-D warnings" cargo doc -p switchyard-libsy -p switchyard-nemo-relay-plugin --no-deps
  • Ruff and mypy clean for the new bundle/E2E Python helpers
  • Manual process smoke: packaged the dylib, validated/added/enabled it with
    the Relay CLI, and exercised buffered plus streaming OpenAI Chat, OpenAI
    Responses, and Anthropic Messages routes
  • Manual random-router smoke: all three target protocols, cross-protocol
    routes, unknown-field replay, 12 concurrent runs, retry-once, retry
    exhaustion, and exactly-once fallback
  • Manual LLM-classifier smoke: classifier CallLlm, selected-target
    CallLlm, and final response for buffered and streaming requests
  • Revalidated after syncing current main: 201 libsy tests, 5 plugin tests,
    focused Clippy, and the cross-repository Relay process E2E pass. The E2E
    reports 12 concurrent random calls across all three protocols, 27 routing
    decisions, 2 LLM-classifier decisions, 2 retry attempts, exactly 1 fallback
    call, and successful buffered plus raw-stream preservation.
  • Live Inference Hub smoke: eight buffered and four streaming calls routed
    across nvidia/meta/llama-3.2-1b-instruct and
    nvidia/mistralai/mistral-7b-instruct-v0.3
  • Phoenix verification: one complete 14-span OpenInference trace contains
    12 switchyard.routing.requested events, 12
    switchyard.routing.decision events, and both semantic targets

Live trace:
http://10.193.162.160:6006/projects/UHJvamVjdDoxOA==/traces

The repository-wide Rust workspace run reaches an existing macOS
switchyard-py link failure for unresolved Python symbols. All touched Rust
crates and the cross-repository process E2E pass independently.

Checklist

  • One class per file; filename = snake_case of the primary class (not
    applicable to the Rust crate; Python helpers follow snake_case).
  • New public symbols exported from switchyard/__init__.py.__all__ if
    intended for downstream use (no Python public symbols added).
  • Unit tests added for new components / bug fixes.
  • README / --help updated if customer-facing surface changed.
  • Commits signed off (Signed-off-by: Your Name <email>) per the DCO.

Notes for reviewers

Please review the integration boundary rather than the already-reviewed #192
stream-preservation implementation:

caller wire request
  -> switchyard-translation decode
  -> libsy run_stream
  -> CallLlm
  -> Relay ABI v2 provider dispatch
  -> switchyard-translation decode response/event
  -> CallLlmRequest::respond
  -> ReturnToAgent
  -> switchyard-translation encode to caller

Intentional constraints:

  • v2-only: no Relay native C ABI v1 compatibility mode;
  • no Relay codecs, private dispatch headers, Relay runtime dependency,
    Switchyard LLM client, or Switchyard service;
  • no decision-only/observe-only API or synthesized decision telemetry;
  • translation rejects lossy required cross-protocol features rather than
    weakening the request;
  • the Relay SDK is pinned to the ABI-v2 feature commit until
    nemo-relay-plugin publishes its first compatible version;
  • release automation is deferred until the SDK dependency can move from the
    development commit to that published version.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
…replay

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
…al event

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Keep preservation on TranslationEngine, the API consumed by NVIDIA/NeMo-Relay#586, and remove the unused built-in convenience decoder.

Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant