litellm-runner

End-to-end regression test suite for BerriAI/litellm.

This repo is tests only — the pytest runbooks that exercise a running LiteLLM gateway. Everything else (building the gateway image, the vcluster it runs in, the schedule, metrics, and Slack/webhook notifications) lives in litellm-ops under apps/base/litellm/e2e/.

How it fits together

litellm-ops (infra + reporting)              this repo (tests)
───────────────────────────────             ─────────────────
build litellm @latest main → ECR
  └─ litellm-e2e vcluster (EKS hub)
       ├─ LiteLLM gateway on the ECR image  ◄── pytest tests/ runs against it
       ├─ isolated Postgres + Redis              via LITELLM_BASE_URL + _MASTER_KEY
       └─ scheduled CronJob
            ├─ run the suite
            ├─ compare vs baseline
            └─ Slack / webhook on regressions

litellm-ops owns running the suite and reporting the result; this repo owns what is tested.

Running the suite

The suite only needs a reachable gateway — point it at one with two env vars:

export LITELLM_BASE_URL=http://localhost:4000     # gateway under test
export LITELLM_MASTER_KEY=sk-...                  # its master key

pip install -r tests/requirements.txt
pytest tests/ -m spend -v        # or -m budgets / translation / caching / ...

For a real run, port-forward the gateway in the litellm-ops litellm-e2e vcluster and set LITELLM_BASE_URL to it.

Layout

Path	What
tests/	the suite, one directory per subsystem
tests/conftest.py	toolkit: admin/temp_key/temp_team/temp_user, chat, wait_for
tests/pytest.ini	category markers (budgets, spend, translation, ...)
tests/COVERAGE.md	what each category covers and the isolation rules

Extending coverage

Add a category: a new marker in tests/pytest.ini, a tests/<feature>/ directory of test_*.py runbooks built on the conftest toolkit, and set pytestmark = pytest.mark.<feature>. See tests/COVERAGE.md for the design rules. The litellm-ops rig rolls each directory up into its own litellm_e2e_runbook_success{runbook=<dir>} Grafana series.