Skip to content

Add a model fallback chain#529

Open
hanna-paasivirta wants to merge 1 commit into
chat-model-envfrom
chat-model-fallback
Open

Add a model fallback chain#529
hanna-paasivirta wants to merge 1 commit into
chat-model-envfrom
chat-model-fallback

Conversation

@hanna-paasivirta

@hanna-paasivirta hanna-paasivirta commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Short Description

If a chat service's model is unavailable, it now falls back to another model instead of failing the request. Each fallback is reported to Sentry, because it is meant to be temporary.

Closes #535

Implementation Details

  • Two small wrappers in services/models.py wrap each chat call, one for normal calls and one for streaming.
  • The chain is Opus, then the previous Opus, then Sonnet, starting from the service's preferred model.
  • Fallback triggers when a model is removed (404) or the provider is down or overloaded (500, 502, 503, 529). The SDK already retries transient errors before this point.
  • Streaming only falls back before the first token reaches the user. After that an error surfaces, so we never re-stream a duplicate answer.
  • Normal errors like bad request, auth, and rate limit are not fallbacks and surface immediately.

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

  • Code generation (copilot but not intellisense)
  • Learning or fact checking
  • Strategy / design
  • Optimisation / refactoring
  • Translation / spellchecking / doc gen
  • Other
  • I have not used AI

You can read more details in our Responsible AI Policy

@hanna-paasivirta

Copy link
Copy Markdown
Contributor Author

Editing this to set models per service then retest. See #528 (comment)

@hanna-paasivirta hanna-paasivirta marked this pull request as ready for review June 15, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant