feat(plugin): add NeMo Relay dynamic integration - #220
Draft
bbednarski9 wants to merge 11 commits into
Draft
Conversation
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
…replay Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
…al event Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Keep preservation on TranslationEngine, the API consumed by NVIDIA/NeMo-Relay#586, and remove the unused built-in convenience decoder. Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
switchyard-nemo-relay-plugin, an external NeMo Relay native dynamicplugin that exports only Relay native C ABI v2 and drives Switchyard algorithms
through
libsy::Algorithm::run_stream.The plugin:
requests with
switchyard-translation;CallLlmstep to a semantic target and asks Relay toperform the actual provider dispatch;
provider failures back through libsy until
ReturnToAgent;first-stream-item commitment behavior;
libsy steps;
process E2E, and operator documentation.
It also makes
run_streamexecutor-neutral and lazy while retaining itsoptional run observer. libsy no longer requires its caller to provide a Tokio
runtime merely to poll an algorithm stream, which is required when the caller
is a host-managed dynamic-plugin callback.
The crate is a source/build unit (
cdylib+rlib,publish = false). Theoperator-facing artifact is a release bundle containing the shared library,
materialized manifest, schema, license notices, and SHA-256 digest.
Why
NeMo Relay must remain responsible for provider authentication, transport,
retries, fallback, and observability, while Switchyard algorithms may transform
requests, request one or more LLM calls, inspect their results, and select the
final response. A model-selection-only API cannot express classifiers or future
ensemble algorithms.
Relay native C ABI v2 provides the typed, C-safe provider-dispatch contract
needed to implement that lifecycle without linking the full Relay runtime,
using
switchyard-llm-client, or runningswitchyard-server.This PR is stacked on and depends on #192. PR #192 is what lets same-protocol
provider stream events survive the required
Relay -> libsy -> Relayroundtrip without losing unknown fields. Feature changes specific to this PR are
introduced by commits
017bea20and2efcb3c4; merge commit9725b24asyncsthe branch with current
mainand reconciles the observer-capablerun_streamsignature without changing #192.
Related: NVIDIA/NeMo-Relay#594
How tested
cargo test -p switchyard-libsycargo test -p switchyard-translation --test stream_translationcargo test -p switchyard-nemo-relay-plugincargo clippy --workspace --all-targets -- -D warningscargo fmt --all --checkRUSTDOCFLAGS="-D warnings" cargo doc -p switchyard-libsy -p switchyard-nemo-relay-plugin --no-depsthe Relay CLI, and exercised buffered plus streaming OpenAI Chat, OpenAI
Responses, and Anthropic Messages routes
routes, unknown-field replay, 12 concurrent runs, retry-once, retry
exhaustion, and exactly-once fallback
CallLlm, selected-targetCallLlm, and final response for buffered and streaming requestsmain: 201 libsy tests, 5 plugin tests,focused Clippy, and the cross-repository Relay process E2E pass. The E2E
reports 12 concurrent random calls across all three protocols, 27 routing
decisions, 2 LLM-classifier decisions, 2 retry attempts, exactly 1 fallback
call, and successful buffered plus raw-stream preservation.
across
nvidia/meta/llama-3.2-1b-instructandnvidia/mistralai/mistral-7b-instruct-v0.312
switchyard.routing.requestedevents, 12switchyard.routing.decisionevents, and both semantic targetsLive trace:
http://10.193.162.160:6006/projects/UHJvamVjdDoxOA==/traces
The repository-wide Rust workspace run reaches an existing macOS
switchyard-pylink failure for unresolved Python symbols. All touched Rustcrates and the cross-repository process E2E pass independently.
Checklist
snake_caseof the primary class (notapplicable to the Rust crate; Python helpers follow
snake_case).switchyard/__init__.py.__all__ifintended for downstream use (no Python public symbols added).
--helpupdated if customer-facing surface changed.Signed-off-by: Your Name <email>) per the DCO.Notes for reviewers
Please review the integration boundary rather than the already-reviewed #192
stream-preservation implementation:
Intentional constraints:
Switchyard LLM client, or Switchyard service;
weakening the request;
nemo-relay-pluginpublishes its first compatible version;development commit to that published version.