Repository navigation
Description
Semantic Kernel currently invokes one AI service per call via service IDs, and large-input workloads are handled ad hoc inside connectors (e.g. the 100-input Gemini embedding chunking in PR #14700). Peer framework pydantic-ai provides a shared batch model: encode requests, group them by credential, create/status/cancel provider batch jobs, and report per-request results including unrun requests.
Proposal: add an opt-in batch request API for SK Python connectors (BatchRequest/Batch/result types plus batch_encode/create/status/cancel on a model-service interface), implemented first for one provider, so future connectors get consistent grouping, polling and error behavior. No public signatures change for existing non-batch paths; this needs design agreement before implementation per the contribution policy.
Users
SK users indexing document sets or running large offline evaluation/scoring workloads who today must chunk inputs and poll provider batch jobs themselves, per connector.
Expected Effect
A shared batch abstraction lets connectors (OpenAI, Azure OpenAI, Google, Anthropic) support provider batch endpoints with consistent grouping, polling and error semantics instead of ad-hoc per-connector chunking like PR #14700.
Scope
Python first (where connector activity is concentrated): a new batch module under semantic_kernel/connectors or services plus opt-in implementation on one connector (e.g. OpenAI chat or Google embeddings). .NET parity and additional providers stay out of scope until the design is agreed.
Validation
cd python && uv run pytest tests/unit/connectors -q plus new tests: encode produces provider payloads; create groups only same-credential requests; status polling stops at ended; unrun/canceled requests surface typed errors; >100-input embedding calls split like #14700. Expected: new tests fail on base, pass with change.
Risks
New public API surface in a compatibility-bound project; provider batch semantics differ (Google inline vs OpenAI file-based jobs), so a wrong abstraction could force breaking changes later.
Effort
Design discussion plus ~1-2 weeks: shared batch types and codec-equivalent helpers, one connector implementation, unit tests mirroring pydantic-ai's grouped/polling/error coverage.
Comparison
pydantic-ai implements this as provider-independent codecs shared by anthropic/_batch.py, _openai_batch.py and _google_batch.py with BatchJobError, group isolation by credential hash and byte-limit prechecks; SK has only ordered service selection (MultipleModelTests) and per-connector workarounds.
Maintainer direction requested before implementation.
AI-assisted contribution.
Activity
[-]Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/-][+].Net: Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/+][-].Net: Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/-][+]Python: Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/+][-]Python: Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/-][+]New Feature: Python — provider-independent batch request API for model services (encode/create/status/cancel/collect)[/+]