Problem (one or two sentences)
High Demand models can experience API Request Failures due to different kind of rate limits or other reasons for temporal unavailability.
This causes Zoo to get stuck on requests for considerable amounts if time.
Context (who is affected and when)
as described above
Desired behavior (conceptual, not technical)
Configuration Profiles would allow for one or multiple configured models that are used in certain conditions i.e. response code: 429
Alternatively API Configuration under Modes could provide multiple configs. This would it make it easier to
- isolate this behavior to certain modes
- provide different LLM Provide in cases the reason for the unavailability is tied to a certain provider
Constraints / preferences (optional)
No response
Request checklist
Zoo Code Task Links (optional)
No response
Acceptance criteria (optional)
No response
Proposed approach (optional)
No response
Trade-offs / risks (optional)
No response
Problem (one or two sentences)
High Demand models can experience API Request Failures due to different kind of rate limits or other reasons for temporal unavailability.
This causes Zoo to get stuck on requests for considerable amounts if time.
Context (who is affected and when)
as described above
Desired behavior (conceptual, not technical)
Configuration Profiles would allow for one or multiple configured models that are used in certain conditions i.e. response code:
429Alternatively API Configuration under Modes could provide multiple configs. This would it make it easier to
Constraints / preferences (optional)
No response
Request checklist
Zoo Code Task Links (optional)
No response
Acceptance criteria (optional)
No response
Proposed approach (optional)
No response
Trade-offs / risks (optional)
No response