⚡ PromptDiff

Fast, CI-Native Regression Testing for LLM Prompts ("Git Diff for Prompts")

Catch silent quality regressions, schema breakages, latency spikes, and cost inflation before merging prompt changes.

🌐 Live Interactive Demo • 📖 Docs & API • 🚀 Quickstart • 🔍 Core Workflow • 🩺 Environment Doctor • 🐍 Python SDK • 📚 Recipe Catalog • 🧪 Pytest Integration • 📦 Installation

🌐 Try it Live in Your Browser: Test the interactive prompt diff playground, token cost calculator, and AST mutation visualizer without installing anything: latryee.github.io/promptdiff

---

📑 Table of Contents

• 💡 Why PromptDiff?
• ⚖️ Honest Comparison: PromptDiff vs Alternatives
• 🚀 Quickstart in 30 Seconds
• 🩺 Environment Doctor
• 🔍 How It Works: Pull Request Quality Gate
• 📚 Curated Recipe Catalog
• 🧪 Pytest Plugin Integration
• 🐍 Python SDK
• 📦 Installation & Modular Extras
• 🧩 Advanced & Extended Modules
• 🛡️ Production Engineering Standards & Quality Assurance
• 🏛️ Architecture Deep-Dive
• 🔒 Data Privacy & Local Storage Disclosure
• 🤝 Community & Contributing
• 📈 Star History
• 📄 License

---

💡 Why PromptDiff?

Modifying system prompts or switching models often leads to unexpected side effects: broken JSON formatting, subtle hallucinations, increased latency, or ballooning token costs.

promptdiff brings standard software regression testing to prompt engineering:

• CLI & CI/CD First: Run lightweight local evaluations in seconds or gate pull requests in GitHub Actions.
• Deterministic Caching: SHA-256 keyed SQLite disk cache ensures identical runs cost $0 and execute in milliseconds.
• Accurate Token & Cost Gating: Model pricing registry with local tokenizers calculates exact financial and latency deltas.
• Hardened Subprocess Sandbox: Isolated code execution runner with resource limits and exploit-tested AST/memory sandboxing.
• Rich Reports: Standalone, zero-dependency interactive HTML reports and automated sticky PR comments (Explore Live Demo).

---

⚖️ Honest Comparison: PromptDiff vs Alternatives

Selecting the right evaluation tool depends heavily on your team's workflow, runtime stack, and data sovereignty requirements:

Feature / Dimension	PromptDiff	promptfoo	LangSmith	Braintrust
Primary Focus	Local-first regression CI/CD & prompt version diffing	LLM red-teaming, security & multi-provider CLI evals	Production tracing, debug sessions & SaaS observability	Enterprise eval platform, proxy logging & collaboration
Runtime & Language	Pure Python 3.10+ (zero heavy dependencies)	Node.js / TypeScript	Hosted SaaS (Python / TS SDKs)	Hosted SaaS / Enterprise on-prem
Data Privacy	100% Local / On-prem (SQLite on local disk; zero telemetry exfiltration)	Local / Self-hosted	Cloud SaaS (prompts & traces sent to vendor servers)	Cloud SaaS / Enterprise Private Cloud
CI/CD Quality Gate	Native promptdiff test & Pytest plugin (exit code 1 on regression)	Native CLI runner & GitHub Actions	Webhook / CI SDK assertions	CI integration via CLI / SDK
Cost & Latency Diffing	Deterministic offline token & pricing delta engine	Basic cost approximations	Cloud dashboard cost tracking	Cloud dashboard cost analytics
Sandboxed Code Execution	Isolated OS subprocess (-I -s -B, memory & CPU limits)	Node VM sandbox	Cloud worker execution	Cloud execution sandbox
Automated Prompt Optimization	Reflexive meta-prompting & MCTS compiler	Optional external scripts	Playground prompt engineering	Automated AI prompt tuner
Full Distributed Tracing	⚠️ Telemetry logs only (OpenTelemetry / MLflow exportable)	⚠️ Eval-focused only	✅ Full distributed waterfall traces	✅ Distributed trace logging & proxy
Pricing Model	100% Free & Open Source (MIT)	Open Source (MIT) with Enterprise tier	Proprietary SaaS (Usage-based subscription)	Commercial SaaS / Enterprise license

When to choose which:

• Choose PromptDiff if you are a Python/MLOps team that treats prompts as code in Git, wants pytest-native integration, requires 100% data sovereignty without external cloud dependencies, and needs fast PR regression gates.
• Choose promptfoo if you have a Node.js/TypeScript stack, want a rich browser-based red-teaming workspace, or need pre-packaged adversarial jailbreak test suites.
• Choose LangSmith if your primary requirement is distributed production trace visualization across multi-agent LangChain graphs.
• Choose Braintrust if you want an enterprise-managed centralized cloud evaluation platform with web-based team playground collaboration.

---

🚀 Quickstart in 30 Seconds

Note

PyPI Package Name: Published on PyPI as promptdiff-eval due to a legacy package name collision, while the CLI binary and import name remain promptdiff.

# 1. Install promptdiff core (lightweight, zero heavy ML dependencies)
pip install promptdiff-eval

# 2. Scaffold a starter evaluation project
promptdiff init my-evals
cd my-evals

# 3. Run regression tests offline (Zero API keys required)
promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs testcases.jsonl \
  --mock \
  --eval "latency,cost,similarity" \
  --assert "cost_delta <= 15%, latency_delta <= 20%" \
  --export-html report.html

---

🩺 Environment Doctor

Diagnose local environment readiness, LLM API keys, optional packages (tiktoken, sentence-transformers, mlflow, wandb), and disk cache engine with a single command:

promptdiff doctor

---

🔍 How It Works: Pull Request Quality Gate

Integrate promptdiff directly into your CI/CD pipeline to block regressions before merging to main:

promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs datasets/testcases.jsonl \
  --model gpt-4o \
  --eval "json_validity,latency,cost,similarity,llm_judge,faithfulness,security" \
  --assert "cost_delta <= 10%, latency_delta <= 15%, similarity >= 0.75, faithfulness >= 0.85" \
  --fail-on-regression \
  --export-markdown report.md

Exit Code	CI Status	Category	Description
0	PASSED	SUCCESS	Quality assertions satisfied; no regressions detected. Safe to merge.
1	REGRESSION	REGRESSION_DETECTED	Regression threshold violated (e.g. cost jump, latency spike, schema break). CI fails.
2	CONFIG_ERROR	CONFIGURATION_ERROR	Malformed dataset, invalid prompt syntax, or bad assertion expression.
3	PROVIDER_ERROR	PROVIDER_ERROR	Provider API authentication failure, rate limit, quota exhaustion, or timeout.
4	INTERNAL_ERROR	INTERNAL_ERROR	Unexpected unhandled runtime exception or critical internal error.

⚙️ Production CI Flags

promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs datasets/large_testcases.jsonl \
  --tags "critical,edge-case" \
  --limit 200 \
  --timeout 45.0 \
  --redact \
  --experiment-id "release-2026-q3" \
  --fail-on-regression

• --tags: Filter test cases by comma-separated tags (only executes matching cases).
• --limit: Stream and evaluate only the first $N$ matching test cases (ideal for smoke tests).
• --timeout: Enforce per-testcase execution timeout in seconds (default: 30.0s).
• --redact: Mask API keys (OpenAI, Anthropic, Gemini, AWS), Bearer/JWT tokens, and PII (emails, SSNs) from outputs, logs, and exported reports.
• --experiment-id: Track evaluations with a custom run ID for multi-run database telemetry.

🤖 Standalone Pull Request Commenter (scripts/pr_commenter.py)

For non-composite CI environments (Jenkins, GitLab CI, Buildkite, or custom GitHub Actions steps), use the standalone PR commenting script:

# Run regression evaluation exporting report JSON
promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
  --inputs datasets/testcases.jsonl \
  --mock \
  --export-json report.json

# Post or update sticky Markdown evaluation comment on PR
python scripts/pr_commenter.py \
  --report report.json \
  --repo "$GITHUB_REPOSITORY" \
  --pr "$PR_NUMBER" \
  --token "$GITHUB_TOKEN"

---

📚 Curated Recipe Catalog

Pull ready-to-use prompt templates, test suites, and tailored evaluators for your specific use case:

# List all domain recipes
promptdiff recipe list

# Pull a specific starter kit
promptdiff recipe pull rag-qa           # RAG Grounding & Faithfulness
promptdiff recipe pull json-extractor   # Strict Structured Output & Schema AST
promptdiff recipe pull sql-gen          # Natural Language to SQL
promptdiff recipe pull security-guard   # Prompt Injection & Extraction Defense

---

🧪 Pytest Plugin Integration

PromptDiff includes an official, native pytest plugin (pytest-promptdiff) for seamless test automation:

1. Declarative Test Marker (@pytest.mark.promptdiff)

Define regression tests declaratively on test functions. Quality assertions are automatically verified by the plugin:

import pytest
from promptdiff.core.models import TestCase


@pytest.mark.promptdiff(
    v1="prompts/support_v1.txt",
    v2="prompts/support_v2.txt",
    dataset=[
        TestCase(id="tc1", vars={"query": "How do I reset my password?"}),
        TestCase(id="tc2", vars={"query": "Request billing refund"}),
    ],
    model="gpt-4o",
    eval="json_validity,latency,cost,similarity",
    assertions=["cost_delta <= 15%", "latency_delta <= 20%"],
    mock=True,
)
def test_support_prompt_regression(report):
    # 'report' fixture is automatically injected with the DiffReport
    assert report.verdict.passed
    assert report.verdict.cost_delta_pct < 15.0

2. Programmatic Fixture (prompt_diff)

For dynamic setups requiring conditional test generation:

# tests/test_prompts.py
import pytest
from promptdiff.core.models import TestCase


@pytest.mark.asyncio
async def test_support_prompt_regression(prompt_diff):
    report = await prompt_diff.compare(
        v1="prompts/support_v1.txt",
        v2="prompts/support_v2.txt",
        test_cases=[
            TestCase(id="tc1", vars={"query": "How do I reset my password?"}),
            TestCase(id="tc2", vars={"query": "Request billing refund"}),
        ],
        model="gpt-4o",
        mock=True,
    )
    assert report.verdict.passed, f"Regression detected: {report.verdict.failed_assertions}"

Run with standard pytest:

pytest tests/test_prompts.py

---

🐍 Python SDK

Use promptdiff programmatically inside Python applications or evaluation scripts:

import promptdiff
from promptdiff.core.models import TestCase

# Run regression evaluation
report = promptdiff.compare(
    v1="prompts/support_v1.txt",
    v2="prompts/support_v2.txt",
    dataset=[
        TestCase(id="tc1", vars={"query": "Reset password"}),
        TestCase(id="tc2", vars={"query": "Billing question"}),
    ],
    model="gpt-4o",
    mock=True,
    assertions=["cost_delta <= 15%", "latency_delta <= 20%"],
)

print(f"Passed: {report.verdict.passed}")
print(f"Cost Delta: {report.verdict.cost_delta_pct:.1f}%")

# Compress prompt tokens while maintaining quality
shrunk = promptdiff.shrink(
    prompt="Please kindly act as an AI and answer: {{query}}",
    dataset=[TestCase(id="1", vars={"query": "Help"})],
    mock=True,
)
print(f"Compressed Prompt: {shrunk.compressed_prompt}")

---

📦 Installation & Modular Extras

PromptDiff is built with a slim, featherweight core and modular extras so you only install what you need:

# Core CLI & CI runner (typer, rich, pydantic, httpx, jinja2, pyyaml, tenacity, numpy)
pip install promptdiff-eval

# Semantic dense embedding similarity (sentence-transformers)
pip install "promptdiff-eval[semantic]"

# Interactive split-screen Terminal UI (Textual)
pip install "promptdiff-eval[tui]"

# Streamlit telemetry web dashboard
pip install "promptdiff-eval[ui]"

# All optional components
pip install "promptdiff-eval[all]"

---

🧩 Advanced & Extended Modules

Command / Tool	Extra Required	Description
promptdiff cache-impact	Core	KV-cache prefix breakpoint analyzer & monthly financial cash loss forecaster.
promptdiff replay-traces	Core	Production OpenTelemetry & Langfuse shadow replayer with automated PII masking.
promptdiff arena	Core	Evaluate $N$ prompt versions with Bayesian Bradley-Terry & ELO skill ratings.
promptdiff studio	Core	Launch zero-dependency local-first visual diff web studio & playground.
promptdiff mcts	Core	Active Monte Carlo Tree Search prompt optimizer with Pareto frontier.
promptdiff redteam	Core	Multi-turn TAP adversarial red-teaming (steganography & CVSS risk matrix).
promptdiff cascade	Core	Confidence-aware model cascade router & enterprise ROI forecaster.
promptdiff check	Core	Static linting & token cost analysis for prompt templates.
promptdiff serve	Core	Launch FastAPI REST API server & playground (pip install fastapi uvicorn).
promptdiff diff	Core	Instant side-by-side terminal syntax diff without calling model APIs.
promptdiff pricing	Core	Query token pricing and cost calculations for 30+ providers.
promptdiff fuzz	Core	Red-teaming security fuzzer scanning 20 distinct adversarial injection vectors.
promptdiff tui	[tui]	Launch interactive split-screen terminal workspace (pip install promptdiff-eval[tui]).
promptdiff ui	[ui]	Launch Streamlit web dashboard for interactive telemetry (pip install promptdiff-eval[ui]).
promptdiff optimize	Core	Reflective auto-prompt optimizer (DSPy style) using meta-model feedback.
promptdiff shrink	Core	Token compressor pruning boilerplate fluff while preserving 100% output quality.
promptdiff cache-sim	Core	Prefix caching hit rate analyzer and ROI forecaster.
promptdiff history	Core	Benchmark prompt quality and cost evolution across Git revisions.

---

🛡️ Production Engineering Standards & Quality Assurance

PromptDiff is built to enterprise MLOps standards with zero tolerance for unverified code or silent regressions:

Dimension	Quality Standard	Verification
Comprehensive Test Suite	390+ unit, integration, and security tests (see CI badge above for live count)	pytest passing on Linux, macOS, and Windows
Test Coverage	92%+ branch & statement coverage (see CI badge above)	Automated threshold enforcement in CI (--cov-fail-under=85)
Isolated Code Sandbox	Subprocess execution with resource limits (RLIMIT_AS, RLIMIT_CPU)	Exploit-tested AST/memory barriers & strict timeout handling
Strict Type Safety	100% type-annotated codebase (PEP 561 compliant py.typed)	mypy --strict promptdiff (0 errors across 121 source files)
Code Formatting & Linting	Automated style checking & import order	ruff check . & ruff format --check . in pre-commit
Internal Dogfooding	We dogfood PromptDiff on our own prompts — see .promptdiff-self-test/, prompt evolution from v1 to the optimizer-compressed version	Verified via offline mock regression tests in CI
Cryptographic Provenance	HMAC-SHA256 zero-width prompt steganography	Constant-time tamper detection (hmac.compare_digest)
Schema Drift Protection	Automated drift protection against JSON schema divergence	DiffReport.model_json_schema() verified in CI pipeline

---

🏛️ Architecture Deep-Dive

Detailed system design documentation, architectural diagrams, mathematical formulations, and core engineering decisions are available in:

👉 Technical Architecture & System Design Deep-Dive (PORTFOLIO.md)

---

🔒 Data Privacy & Local Storage Disclosure

PromptDiff operates under an absolute local-first, zero-telemetry exfiltration guarantee:

• Local Persistence Only: Evaluation runs and token metrics are written to local SQLite storage (.promptdiff/telemetry.db). No prompt contents, outputs, or traces are ever sent to external cloud servers.
• Automated Retention Management: Automatically delete historical records older than $N$ days with --db-retention-days <N> or run promptdiff db prune --days 14.
• Sensitive Data & PII Redaction: Pass --redact (or redact=True in Python SDK) to automatically mask API keys (OpenAI, Anthropic, Gemini, AWS, Hugging Face), Bearer/JWT tokens, and sensitive PII (emails, SSNs, credit cards) in terminal outputs, logs, and exported artifacts.
• Ephemeral Storage: Run with --db-path ":memory:" for zero disk persistence.
• Complete security documentation and disclosure SLAs are available in SECURITY.md.

---

🤝 Community & Contributing

• Contributing Guide
• Code of Conduct
• Security Policy
• Changelog

---

📈 Star History

---

📄 License

Distributed under the MIT License. See LICENSE for more information.