Skip to content

New Feature: Python — provider-independent batch request API for model services (encode/create/status/cancel/collect) #14746

Open

Description

Semantic Kernel currently invokes one AI service per call via service IDs, and large-input workloads are handled ad hoc inside connectors (e.g. the 100-input Gemini embedding chunking in PR #14700). Peer framework pydantic-ai provides a shared batch model: encode requests, group them by credential, create/status/cancel provider batch jobs, and report per-request results including unrun requests.

Proposal: add an opt-in batch request API for SK Python connectors (BatchRequest/Batch/result types plus batch_encode/create/status/cancel on a model-service interface), implemented first for one provider, so future connectors get consistent grouping, polling and error behavior. No public signatures change for existing non-batch paths; this needs design agreement before implementation per the contribution policy.

Users

SK users indexing document sets or running large offline evaluation/scoring workloads who today must chunk inputs and poll provider batch jobs themselves, per connector.

Expected Effect

A shared batch abstraction lets connectors (OpenAI, Azure OpenAI, Google, Anthropic) support provider batch endpoints with consistent grouping, polling and error semantics instead of ad-hoc per-connector chunking like PR #14700.

Scope

Python first (where connector activity is concentrated): a new batch module under semantic_kernel/connectors or services plus opt-in implementation on one connector (e.g. OpenAI chat or Google embeddings). .NET parity and additional providers stay out of scope until the design is agreed.

Validation

cd python && uv run pytest tests/unit/connectors -q plus new tests: encode produces provider payloads; create groups only same-credential requests; status polling stops at ended; unrun/canceled requests surface typed errors; >100-input embedding calls split like #14700. Expected: new tests fail on base, pass with change.

Risks

New public API surface in a compatibility-bound project; provider batch semantics differ (Google inline vs OpenAI file-based jobs), so a wrong abstraction could force breaking changes later.

Effort

Design discussion plus ~1-2 weeks: shared batch types and codec-equivalent helpers, one connector implementation, unit tests mirroring pydantic-ai's grouped/polling/error coverage.

Comparison

pydantic-ai implements this as provider-independent codecs shared by anthropic/_batch.py, _openai_batch.py and _google_batch.py with BatchJobError, group isolation by credential hash and byte-limit prechecks; SK has only ordered service selection (MultipleModelTests) and per-connector workarounds.

Maintainer direction requested before implementation.

AI-assisted contribution.

Activity

  1. added
    .NETIssue or Pull requests regarding .NET code
    pythonPull requests for the Python Semantic Kernel
    on Nov 8, 2026
  2. changed the title [-]Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/-] [+].Net: Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/+] on Nov 8, 2026
  3. changed the title [-].Net: Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/-] [+]Python: Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/+] on Nov 8, 2026
  4. changed the title [-]Python: Add a provider-independent batch request API for model services (encode/create/status/cancel/collect)[/-] [+]New Feature: Python — provider-independent batch request API for model services (encode/create/status/cancel/collect)[/+] on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    .NETIssue or Pull requests regarding .NET codepythonPull requests for the Python Semantic Kerneltriage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions