diff --git a/README.md b/README.md index a30261864..767f98ad0 100644 --- a/README.md +++ b/README.md @@ -241,12 +241,25 @@ setup, Switchyard uses the server's `OPENROUTER_API_KEY` for upstream requests. Do not use the placeholder with `forward_auth = true` or a gateway that requires a real client credential. -Codex CLI and other OpenAI clients use the OpenAI variables instead: +For Codex CLI, add this provider to `~/.codex/config.toml`: + +```toml +[model_providers.switchyard] +name = "Switchyard" +base_url = "http://localhost:4000/v1" +wire_api = "responses" +requires_openai_auth = false +``` + +Then select the provider and route: ```bash -export OPENAI_BASE_URL="http://localhost:4000/v1" +codex --model switchyard -c 'model_provider="switchyard"' ``` +No Codex API key is needed for this local setup. Switchyard uses the server's +`OPENROUTER_API_KEY` for upstream requests. + ## Routing Algorithms Start with **Auto**. Choose Task or Execution when you want more control over diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.md index bb3b38caf..34ec8a989 100644 --- a/crates/libsy-llm-client/README.md +++ b/crates/libsy-llm-client/README.md @@ -47,8 +47,10 @@ Within this workspace: ```toml [dependencies] +futures-util = "0.3" +switchyard-libsy = { path = "../libsy" } switchyard-llm-client = { path = "../libsy-llm-client" } -switchyard-protocol = { path = "../libsy-protocol" } +switchyard-protocol = { path = "../protocol" } switchyard-translation = { path = "../switchyard-translation" } # for WireFormat ``` @@ -153,7 +155,7 @@ ordered fallbacks for the client to try. `ClientRouter::single` is the single-pr ```rust use std::sync::Arc; -use switchyard_libsy::Algorithm; +use switchyard_libsy::{Algorithm, RuntimeModels}; use switchyard_llm_client::{ClientRouter, TranslatingLlmClient}; use switchyard_protocol::Request; @@ -161,10 +163,11 @@ async fn route( algorithm: Arc, client: Arc, request: Request, + models: Arc, ) -> switchyard_libsy::Result { let clients = ClientRouter::single(client); let (selected_model, _response) = - switchyard_llm_client::run(algorithm, clients, request, None).await?; + switchyard_llm_client::run(algorithm, clients, request, models, None).await?; Ok(selected_model.to_string()) } ``` @@ -285,8 +288,8 @@ after a stream has started does not count another attempt. | `InvalidResponse { source }` | the upstream response could not be decoded | | `Other(source)` | a client-specific failure outside the shared categories | -[`switchyard_protocol::Request`]: ../libsy-protocol -[`switchyard_protocol::Response`]: ../libsy-protocol +[`switchyard_protocol::Request`]: ../protocol +[`switchyard_protocol::Response`]: ../protocol [`libsy-proxy`]: ../libsy-proxy [`Backend`]: src/backend.rs [`HttpBackendConfig`]: src/backend.rs diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md index c1b1af8af..d57d70dde 100644 --- a/docs/routing_algorithms/llm_classifier_routing.md +++ b/docs/routing_algorithms/llm_classifier_routing.md @@ -270,48 +270,45 @@ enable `message_hash_fallback` to key on the first user-message text under ### Responses continuations by ID -A Responses API client can continue without resending the conversation history: -`previous_response_id` refers to a stored response, and `conversation` refers to -a provider's conversation. When answer targets use different `[llm_clients]` -entries, Switchyard records the model that served each stored Responses ID and -conversation ID. A request with a recorded ID returns to that exact model -without a judge call or fallback to another provider. This applies to every -`classify_trigger` and to `stage_router` and `composite` routes. Classifier-only -clients do not count; when answer targets share one client, routing proceeds as -usual. - -Switchyard tracks buffered and streamed Responses replies, including answers -returned by `/v1/decision`. It does not record Chat Completions or Anthropic -Messages response IDs as Responses continuation IDs. A provider's `store` value -takes precedence over the request value. If that value is `false`, Switchyard -skips the response ID but still records the conversation ID. +A Responses API client can continue without resending the conversation history. +Switchyard handles state differently depending on the target API: + +- **Native Responses targets:** The provider stores the conversation. When answer + targets use different `[llm_clients]` entries, Switchyard records response and + conversation IDs with the model that served them, but no transcript. Classifier-only + clients do not count. When answer targets share one client, no local record is needed. + The provider's `store` value takes precedence over the request value. If it is + `false`, Switchyard skips the response ID but still records the conversation ID. +- **Chat Completions or Anthropic Messages targets:** For incoming Responses + requests, Switchyard retains request and reply messages in memory under the + response ID. It restores that history on a later `previous_response_id` request. + This also applies when answer targets share one client. Request `store: false` + prevents retaining the new response and history, but does not delete earlier + records. This path supports `previous_response_id`, not provider conversation IDs. + +A request with a recorded ID returns to the model that served it without a judge +call or fallback to another provider. This applies to every `classify_trigger` +and to `stage_router` and `composite` routes. Tracking covers buffered replies and +completed streams, including answers returned by `/v1/decision`. Ordinary Chat +Completions and Anthropic Messages requests do not create Responses history. Each route keeps up to 65,536 distinct ID-to-model records per process. This limit counts response and conversation IDs recorded over the process's lifetime, not tokens or simultaneous requests. Records do not expire, and Switchyard does -not remove older records to make room. The map stores copies of IDs and model -names, with no transcript storage. - -If recording new IDs would exceed the limit, a buffered reply returns HTTP 503 -with code `response_state_limit_exceeded`. If an ID is already recorded for a -different model, it returns HTTP 409 with code `response_state_conflict`. Neither -failure changes existing records. Switchyard checks both IDs before adding -either from a response or stream event. - -These checks run after the provider has returned a response or stream event, so -the error does not mean the provider did no work. A stream that has already sent -HTTP headers emits a framed error and stops; it cannot change the HTTP status. -Switchyard logs these errors. For `/v1/responses`, buffered and streamed failures -increment `/v1/stats` `total_errors`. Other routes continue independently. - -An existing ID can still be used at capacity if the reply adds no new IDs. For -example, `store: false` can suppress a new response ID, but it does not suppress -conversation-ID tracking. A stored follow-up normally creates a new response ID -and therefore fails when the map is full. - -A restart removes all records, and each replica has its own map. Unknown IDs use -normal routing and can still fail at the selected provider. To keep dynamic -routing across turns, send the full history instead of an ID. +not remove older records to make room. Cross-format records also retain message +history, so this ID limit is not a memory limit. Memory use depends on the retained +messages as well as the number of IDs. + +At capacity, Switchyard logs a warning and returns the reply without retaining +new IDs or history. Existing records remain usable, but a later continuation from +an unrecorded ID may fail. Conflicting native Responses IDs return HTTP 409 with +code `response_state_conflict`. If streaming headers have already been sent, the +stream emits an error instead of changing the HTTP status. + +A restart removes all local records and history, and each replica has its own +state. Unknown IDs use normal routing and can still fail at the selected provider. +Send the full history instead of an ID to avoid relying on local continuation +state and to keep dynamic routing across turns. ## Run the route