Skip to content

docs: add MLflow AI Gateway documentation page#11025

Draft
PattaraS wants to merge 1 commit intodeepset-ai:mainfrom
PattaraS:add-mlflow-gateway-integration
Draft

docs: add MLflow AI Gateway documentation page#11025
PattaraS wants to merge 1 commit intodeepset-ai:mainfrom
PattaraS:add-mlflow-gateway-integration

Conversation

@PattaraS
Copy link
Copy Markdown

@PattaraS PattaraS commented Apr 2, 2026

Summary

  • Adds docs-website/docs/pipeline-components/generators/mlflowgatewaychatgenerator.mdx — a documentation page showing how to use OpenAIChatGenerator with MLflow AI Gateway
  • Updates sidebars.js to include the new page in the generators section

What is MLflow AI Gateway?

MLflow AI Gateway is a database-backed LLM proxy built into the MLflow tracking server (MLflow ≥ 3.0). It exposes an OpenAI-compatible REST API (/gateway/openai/v1) that routes to OpenAI, Anthropic, Gemini, Mistral, Bedrock, Ollama, and more — configured through a UI with encrypted secrets management, fallback/retry, traffic splitting, and built-in usage tracing.

How It Works with Haystack

No extra packages needed — just use the existing OpenAIChatGenerator with api_base_url:

generator = OpenAIChatGenerator(
    model="my-endpoint-name",
    api_key=Secret.from_token("unused"),
    api_base_url="http://localhost:5000/gateway/openai/v1",
)

Page Contents

  • Overview table (mandatory/output variables)
  • Prerequisites: starting MLflow server and creating a gateway endpoint (UI, REST API)
  • On its own usage example
  • In a pipeline example
  • Streaming example
  • mlflow.deployments Python client alternative

Test Plan

  • Page renders correctly in the docs site
  • MLflow AI Gateway appears in the generators sidebar

This pull request was AI-assisted by Claude.

Adds a documentation page showing how to use OpenAIChatGenerator with
MLflow AI Gateway's OpenAI-compatible endpoint via the api_base_url
parameter. Includes standalone, pipeline, and streaming examples.
@PattaraS PattaraS requested a review from a team as a code owner April 2, 2026 09:17
@PattaraS PattaraS requested review from bogdankostic and removed request for a team April 2, 2026 09:17
@vercel
Copy link
Copy Markdown

vercel bot commented Apr 2, 2026

@PattaraS is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@PattaraS PattaraS marked this pull request as draft April 2, 2026 09:17
@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants