Skip to content

[ENHANCEMENT] allow for backup model confgurations #1657

Description

@baboon25

Problem (one or two sentences)

High Demand models can experience API Request Failures due to different kind of rate limits or other reasons for temporal unavailability.
This causes Zoo to get stuck on requests for considerable amounts if time.

Context (who is affected and when)

as described above

Desired behavior (conceptual, not technical)

Configuration Profiles would allow for one or multiple configured models that are used in certain conditions i.e. response code: 429
Alternatively API Configuration under Modes could provide multiple configs. This would it make it easier to

  1. isolate this behavior to certain modes
  2. provide different LLM Provide in cases the reason for the unavailability is tied to a certain provider

Constraints / preferences (optional)

No response

Request checklist

  • I've searched existing Issues and Discussions for duplicates
  • This describes a specific problem with clear context and impact

Zoo Code Task Links (optional)

No response

Acceptance criteria (optional)

No response

Proposed approach (optional)

No response

Trade-offs / risks (optional)

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions