feat: add llm request throttling

[uipreliga]

This pull request introduces configurable concurrency control for LLM (Large Language Model) API requests, allowing users to limit the number of simultaneous LLM requests to prevent system overload. The main change is the addition of a throttling mechanism using an asyncio semaphore, which can be configured via a new CLI option. This improves robustness and flexibility for users running evaluations and interacting with LLM services.

LLM Throttling and Concurrency Control:

• Added a new module llm_throttle.py that provides get_llm_semaphore and set_llm_concurrency functions to manage and configure the concurrency limit for LLM API requests using an asyncio semaphore. The default limit is 20 concurrent requests, and the semaphore is managed per event loop to avoid cross-loop issues.
• Updated the UiPathLlmChatService and related LLM gateway service methods to use the semaphore, ensuring all LLM API calls are throttled according to the configured concurrency limit. [1] [2] [3] [4]
• Exposed get_llm_semaphore and set_llm_concurrency in the platform.chat package’s public API for external configurability. [1] [2]

CLI Improvements:

• Added a new CLI option --max-llm-concurrency (default: 20) to the eval command, allowing users to set the maximum number of concurrent LLM requests when running evaluations. The value is passed to set_llm_concurrency before any LLM calls are made. [1] [2] [3]

Versioning:

• Bumped the package version from 2.4.3 to 2.4.4 to reflect the new functionality.