diff --git a/.devin/promptfoo/SKILL.md b/.devin/promptfoo/SKILL.md new file mode 100644 index 0000000..23772f1 --- /dev/null +++ b/.devin/promptfoo/SKILL.md @@ -0,0 +1,63 @@ +--- +name: promptfoo +description: Context-efficient reference for building, configuring, and optimizing promptfoo evals, providers, and assertions locally. Use this skill before writing or debugging promptfoo-related configs and custom scripts. +--- + +# Promptfoo Knowledge Skill + +Use this skill when the task involves creating, editing, or debugging promptfoo configuration (`promptfooconfig.yaml`), providers, custom evaluation scripts, or CLI workflows. + +## When to Run + +- Setting up a new promptfoo eval or project +- Configuring providers (built-in, local, or custom) +- Writing or debugging custom JavaScript/TypeScript/Python providers +- Writing or debugging custom assertions or transforms +- Optimizing prompts or comparing models +- Troubleshooting failed evals, provider errors, or OOM/timeouts +- Converting between CLI, YAML, and Node-package usage + +## What Promptfoo Is + +promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. It runs prompts against one or more providers (LLM APIs), compares outputs, and scores them with assertions and metrics. The core idea is **test-driven prompt engineering**: define expectations first, then iterate. + +## Mental Model + +``` +prompts × providers × tests = matrix of outputs + | + v + assertions & metrics +``` + +- **Prompts**: text, chat JSON, or dynamically generated templates +- **Providers**: LLM targets (OpenAI, Anthropic, local Ollama, custom JS/Python, HTTP, etc.) +- **Tests**: input variables + assertions that define pass/fail criteria +- **Config**: a single `promptfooconfig.yaml` binds them together + +## Phase Map + +Read phases in order when learning the tool; jump directly to a phase when the task is focused. + +| Phase | File | Covers | +|-------|------|--------| +| 0 | `SKILL.md` | Index, when-to-use, mental model (you are here) | +| 1 | `phases/01-overview.md` | Philosophy, workflow, key concepts | +| 2 | `phases/02-quickstart-workflow.md` | Install, init, eval, view, .env | +| 3 | `phases/03-config-reference.md` | `promptfooconfig.yaml` structure and properties | +| 4 | `phases/04-prompts.md` | Text, file, chat, dynamic, Nunjucks | +| 5 | `phases/05-providers.md` | Built-in providers, syntax, configuration, env vars | +| 6 | `phases/06-assertions-metrics.md` | Deterministic, model-graded, custom, derived, named | +| 7 | `phases/07-custom-providers.md` | JS/TS/Python provider interfaces and return contracts | +| 8 | `phases/08-evaluation-scripts.md` | Custom asserts, transforms, Node package API | +| 9 | `phases/09-cli-commands.md` | CLI commands, environment variables, CI behavior | +| 10 | `phases/10-debugging-scenarios.md` | Troubleshooting, logs, OOM, stuck evals, custom-provider debugging | +| QR | `quick-reference.md` | Compact cheat sheet | + +## Quick Decision Guide + +- **New eval?** Start at `phases/02-quickstart-workflow.md`, then `phases/03-config-reference.md`. +- **Need a custom model/API?** Read `phases/07-custom-providers.md`. +- **Need a custom scorer?** Read `phases/08-evaluation-scripts.md`. +- **Something is broken?** Read `phases/10-debugging-scenarios.md` first. +- **Need a property list?** Read `phases/03-config-reference.md` or `quick-reference.md`. diff --git a/.devin/promptfoo/phases/01-overview.md b/.devin/promptfoo/phases/01-overview.md new file mode 100644 index 0000000..cca1fa1 --- /dev/null +++ b/.devin/promptfoo/phases/01-overview.md @@ -0,0 +1,78 @@ +# Phase 1: Overview and Philosophy + +## What promptfoo Does + +promptfoo evaluates LLM outputs systematically. You define prompts, run them against one or more models/providers, and check results against assertions. It supports: + +- Declarative YAML configuration (`promptfooconfig.yaml`) +- 60+ built-in providers plus custom JavaScript, Python, HTTP, and WebSocket providers +- Deterministic and model-graded assertions +- Caching, concurrency, live reload, and web-based result comparison +- CLI, Node library, and CI/CD integration + +## Core Philosophy + +**Test-driven prompt engineering beats trial-and-error.** + +The recommended workflow is a tight loop: + +1. **Define test cases** — identify core use cases and failure modes +2. **Configure evaluation** — specify prompts, providers, and assertions +3. **Run evaluation** — `promptfoo eval` +4. **Analyze results** — web UI, JSON/CSV exports, or assertion summaries +5. **Iterate** — expand tests as you find more examples and failure modes + +## Key Concepts + +### Prompts + +The instruction sent to the model. Can be plain text, chat-formatted JSON, file-based, or generated by a function. Variables are substituted with `{{varName}}`. + +### Providers (also called Targets) + +The model or API that produces the output. Examples: + +- `openai:chat:gpt-5-mini` +- `anthropic:messages:claude-opus-4-6` +- `ollama:chat:llama3.3` +- `file://custom_provider.js` + +`providers` and `targets` are interchangeable keys in the config. + +### Tests + +A single test case feeds one set of input variables into all prompts and providers. Each test has: + +- `vars`: variable values substituted into prompts +- `assert`: array of assertions that must pass +- `options`: per-test overrides (provider, repeat, transform, etc.) +- `metadata`: key/value pairs used for filtering or reporting + +### Assertions + +Checks run on each output. Two broad categories: + +- **Deterministic**: `equals`, `contains`, `regex`, `is-json`, `javascript`, `python`, `latency`, `cost`, etc. +- **Model-assisted**: `llm-rubric`, `similar`, `factuality`, `classifier`, `moderation`, etc. + +### Metrics + +Named aggregates of assertion results. Use `metric` on assertions to group them, and `derivedMetrics` to compute composite scores (e.g., F1). + +### Config File + +`promptfooconfig.yaml` is the single source of truth for an evaluation. It contains prompts, providers, tests, defaults, env overrides, and output settings. + +## When to Use Each Interface + +| Use Case | Interface | +|----------|-----------| +| Day-to-day evals, manual comparison | CLI (`promptfoo eval`, `promptfoo view`) | +| Automated CI/CD, unit tests | CLI with exit codes or Node package | +| Embedded in an application | Node package (`promptfoo.evaluate`) | +| Custom API/model not built-in | Custom provider (JS/TS/Python) | +| Custom scoring logic | Custom assertion (JS/Python) or derived metric | + +## Next Step + +If you are starting from scratch, go to `phases/02-quickstart-workflow.md`. If you are editing an existing config, go to `phases/03-config-reference.md`. diff --git a/.devin/promptfoo/phases/02-quickstart-workflow.md b/.devin/promptfoo/phases/02-quickstart-workflow.md new file mode 100644 index 0000000..45b57e4 --- /dev/null +++ b/.devin/promptfoo/phases/02-quickstart-workflow.md @@ -0,0 +1,135 @@ +# Phase 2: Quickstart Workflow + +## Installation + +Requires Node.js `^20.20.0` or `>=22.22.0`. Node.js 24 LTS is recommended. + +```bash +# npm (global) +npm install -g promptfoo + +# npx (no install) +npx promptfoo@latest + +# Homebrew (Mac/Linux) +brew install promptfoo + +# As a library +npm install promptfoo +``` + +Verify: + +```bash +promptfoo --version +# or +npx promptfoo@latest --version +``` + +## First Eval + +### 1. Initialize + +```bash +promptfoo init +# or with an example +npx promptfoo@latest init --example getting-started +``` + +This creates `promptfooconfig.yaml` (and optionally `README.md` and prompt files). + +### 2. Configure + +A minimal config: + +```yaml +prompts: + - 'Translate the following to {{language}}: {{input}}' + +providers: + - openai:chat:gpt-5-mini + - anthropic:messages:claude-opus-4-6 + +tests: + - vars: + language: French + input: Hello world + assert: + - type: icontains + value: 'Bonjour' + - vars: + language: Spanish + input: Where is the library? + assert: + - type: icontains + value: 'biblioteca' +``` + +### 3. Run + +```bash +# Read promptfooconfig.yaml in current directory +promptfoo eval + +# Use a different config +promptfoo eval -c my-config.yaml +``` + +### 4. View Results + +```bash +# Interactive web UI +promptfoo view + +# Or export +promptfoo eval --output results.json +``` + +## Common CLI Variations + +```bash +# Override providers on the fly +promptfoo eval -r openai:gpt-5-mini -r ollama:chat:llama3.3 + +# Override prompts +promptfoo eval -p file://prompts/*.txt + +# Override tests +promptfoo eval -t file://tests.yaml + +# Concurrency and repeat +promptfoo eval -j 4 --repeat 3 + +# Disable cache for fresh calls +promptfoo eval --no-cache + +# Watch for changes +promptfoo eval -w +``` + +## Environment Variables + +promptfoo loads `.env` from the current working directory. Typical keys: + +```bash +OPENAI_API_KEY=sk-... +ANTHROPIC_API_KEY=sk-... +OLLAMA_BASE_URL=http://localhost:11434 +``` + +You can also pass keys inline in the provider `config` (not recommended for committed configs). + +## Web UI Setup + +Alternative to `init`: + +```bash +promptfoo eval setup +``` + +This opens a browser-based flow to create prompts, providers, and test cases. + +## Next Step + +- To understand the full config schema, read `phases/03-config-reference.md`. +- To learn about prompts specifically, read `phases/04-prompts.md`. diff --git a/.devin/promptfoo/phases/03-config-reference.md b/.devin/promptfoo/phases/03-config-reference.md new file mode 100644 index 0000000..b8f2eb2 --- /dev/null +++ b/.devin/promptfoo/phases/03-config-reference.md @@ -0,0 +1,222 @@ +# Phase 3: Configuration Reference + +The `promptfooconfig.yaml` file is the central contract. It defines what to evaluate, how to evaluate it, and how to report results. + +## Top-Level Config (`TestSuiteConfig` / `UnifiedConfig`) + +```yaml +description: Optional human-readable description +tags: + env: ci + team: platform + +providers: ... # or targets: ... +prompts: ... +tests: ... +scenarios: ... + +defaultTest: ... +outputPath: results.json +sharing: false +writeLatestResults: true + +env: + LOG_LEVEL: debug + +derivedMetrics: ... +extensions: ... +metadata: ... +redteam: ... +tracing: ... + +commandLineOptions: ... +evaluateOptions: ... +``` + +## Key Top-Level Properties + +| Property | Type | Purpose | +|----------|------|---------| +| `description` | string | Human-readable eval description | +| `tags` | `Record` | Run metadata, filterable | +| `providers` / `targets` | string \| object \| array | LLM APIs to test | +| `prompts` | string \| array \| record | Prompts to run | +| `tests` | string \| array \| generator | Test cases | +| `scenarios` | array | Grouped test configurations | +| `defaultTest` | object \| `file://` | Defaults applied to every test | +| `outputPath` | string \| string[] | File(s) to write results | +| `sharing` | boolean \| object | Enable result sharing | +| `writeLatestResults` | boolean | Persist to `promptfoo` storage for web viewer | +| `env` | object | Environment variable overrides | +| `derivedMetrics` | array | Composite metrics | +| `extensions` | string[] | Extension hook files | +| `metadata` | object | Arbitrary config metadata | +| `redteam` | object | Red-team configuration | +| `tracing` | object | OpenTelemetry tracing config | +| `commandLineOptions` | object | Default CLI flags | +| `evaluateOptions` | object | Runtime options (cache, concurrency, etc.) | + +## Test Case Properties + +Each test case is an object in the `tests` array or generated from a file. + +```yaml +tests: + - description: Optional description + vars: + language: French + input: Hello world + assert: + - type: contains + value: Bonjour + options: + provider: openai:gpt-5-mini + repeat: 2 + transform: output.toUpperCase() + metadata: + category: translation +``` + +| Property | Type | Purpose | +|----------|------|---------| +| `description` | string | Test name, used in UI and filters | +| `vars` | object | Variables substituted into prompts | +| `assert` | array | Assertions for this test | +| `options` | object | Per-test overrides | +| `metadata` | object | Key/value pairs for filtering/reporting | +| `threshold` | number | Minimum pass score (0-1) | +| `providers` | string[] | Limit which providers run this test | + +### `options` Object + +| Property | Purpose | +|----------|---------| +| `provider` | Provider used for model-graded assertions | +| `transform` | JavaScript/Python transform applied to output before grading | +| `repeat` | Run this test N times | +| `disableDefaultAsserts` | Skip `defaultTest.assert` for this test | +| `response_format` / `temperature` / etc. | Provider-specific overrides | + +## `defaultTest` + +Apply defaults to all tests: + +```yaml +defaultTest: + vars: + audience: developer + assert: + - type: llm-rubric + value: Does not describe itself as an AI or chatbot + options: + provider: openai:gpt-5-mini +``` + +Use `options.disableDefaultAsserts: true` on a specific test to override inherited assertions. + +## Assertion Properties + +```yaml +assert: + - type: contains + value: expected text + threshold: 0.5 + weight: 2 + metric: Tone + config: + caseSensitive: false +``` + +| Property | Purpose | +|----------|---------| +| `type` | Assertion type (required) | +| `value` | Expected value or criteria | +| `threshold` | Score required to pass (varies by type) | +| `weight` | Relative weight in aggregate scoring | +| `metric` | Named metric bucket | +| `provider` | Grader provider for model-graded assertions | +| `rubricPrompt` | Override rubric prompt for model-graded assertions | +| `config` | Assertion-specific config | + +## Assertion Sets + +Group assertions and require a subset or all to pass: + +```yaml +assert: + - type: assert-set + threshold: 0.5 + assert: + - type: cost + threshold: 0.001 + - type: latency + threshold: 200 +``` + +## Named and Derived Metrics + +Named metrics group assertions: + +```yaml +tests: + - assert: + - type: equals + value: Yarr + metric: Tone + - type: icontains + value: grub + metric: Tone + - type: is-json + metric: Consistency +``` + +Derived metrics compute composite scores: + +```yaml +derivedMetrics: + - name: f1_score + value: '2 * precision * recall / (precision + recall)' +``` + +`value` can be a mathematical expression or a JavaScript function. `__count` is available for averages. + +## External File Imports + +Load any list from a file: + +```yaml +prompts: + - file://prompts/*.txt + +providers: + - file://providers.yaml + +tests: + - file://tests.yaml + - file://tests.csv + +defaultTest: file://default.yaml +``` + +CSV/XLSX files are auto-mapped from column headers. Use `__expected` columns for inline assertions. + +## Multiple Configs + +Run multiple configs together: + +```bash +promptfoo eval -c config1.yaml -c config2.yaml +promptfoo eval -c my_configs/* +``` + +## Config Validation + +```bash +promptfoo validate config +promptfoo validate target +``` + +## Next Step + +- Prompt formats: `phases/04-prompts.md` +- Provider configuration: `phases/05-providers.md` diff --git a/.devin/promptfoo/phases/04-prompts.md b/.devin/promptfoo/phases/04-prompts.md new file mode 100644 index 0000000..8dc6364 --- /dev/null +++ b/.devin/promptfoo/phases/04-prompts.md @@ -0,0 +1,199 @@ +# Phase 4: Prompt Configuration + +promptfoo supports several ways to define prompts, from simple strings to dynamic functions. + +## Text Prompts + +```yaml +prompts: + - 'Translate the following to {{language}}: {{input}}' + - 'Summarize this article: {{article}}' +``` + +Variables are substituted with `{{varName}}`. Multiline strings are supported with YAML `|`: + +```yaml +prompts: + - | + You are a helpful assistant. + User: {{input}} + Assistant: +``` + +## File-Based Prompts + +```yaml +prompts: + - file://prompts/customer_service.txt + - file://prompts/technical_support.md + - file://prompts/*.txt +``` + +Supported formats: + +- `.txt` — plain text +- `.md` — Markdown (content used as-is) +- `.j2` — Jinja2 templates (not Nunjucks, but supported) +- `.csv` — multiple prompts with `prompt` and `label` columns + +Multiple prompts in one file are separated by `---`: + +```text +Translate to French: {{text}} +--- +Translate to Spanish: {{text}} +``` + +## Chat Format (JSON) + +Use a JSON array of messages for chat models: + +```json +[ + { "role": "system", "content": "You are a helpful coding assistant." }, + { "role": "user", "content": "Write a function to {{task}}" } +] +``` + +Reference it from config: + +```yaml +prompts: + - file://chat_prompt.json +``` + +Multi-turn conversations include `assistant` messages: + +```json +[ + { "role": "system", "content": "You are a tutoring assistant." }, + { "role": "user", "content": "What is recursion?" }, + { "role": "assistant", "content": "Recursion is a function that calls itself." }, + { "role": "user", "content": "Show me an example in {{language}}." } +] +``` + +## Dynamic Prompts (Functions) + +Generate prompts with JavaScript or Python based on variables and provider. + +### JavaScript + +```javascript +// generate_prompt.js +module.exports = async function ({ vars, provider }) { + const complexity = vars.complexity || 'medium'; + if (complexity === 'simple') { + return `Explain ${vars.topic} in simple terms.`; + } + return `Provide a detailed explanation of ${vars.topic} with examples.`; +}; +``` + +```yaml +prompts: + - file://generate_prompt.js +``` + +### Python + +```python +# generate_prompt.py +def create_prompt(context): + vars = context['vars'] + if vars.get('technical_audience'): + return f"Provide a technical analysis of {vars['topic']}" + return f"Explain {vars['topic']} for beginners" +``` + +```yaml +prompts: + - file://generate_prompt.py:create_prompt +``` + +### Returning Provider Config + +A dynamic prompt can also return per-call provider config: + +```javascript +module.exports = async function ({ vars }) { + return { + prompt: `Analyze ${vars.topic}`, + config: { + temperature: vars.creative ? 0.9 : 0.3, + max_tokens: vars.detailed ? 1000 : 200, + }, + }; +}; +``` + +## Nunjucks Templates + +promptfoo uses Nunjucks for variable substitution. Supported features: + +- Variables: `{{var}}` +- Conditionals: `{% if premium %}...{% endif %}` +- Loops: `{% for item in items %}...{% endfor %}` +- Filters: `{{name | upper}}` +- Custom filters via `nunjucksFilters` + +### Custom Filters + +```javascript +// uppercase_first.js +module.exports = function (str) { + return str.charAt(0).toUpperCase() + str.slice(1); +}; +``` + +```yaml +nunjucksFilters: + uppercaseFirst: ./uppercase_first.js + +prompts: + - 'Dear {{ name | uppercaseFirst }}, {{ message }}' +``` + +### Object Variables in Templates + +Objects passed as vars are stringified by default. To access a property, use dot notation or the `json` filter: + +```yaml +# vars: user: { name: Alice } +prompts: + - 'Hello {{ user.name }}' + - 'User data: {{ user | json }}' +``` + +If you see `[object Object]`, you are using an object variable without property access or a filter. + +## Prompt Labels and IDs + +Label prompts for easier filtering and reporting: + +```yaml +prompts: + - id: file://customer_prompt.txt + label: Customer Service + - id: file://technical_prompt.txt + label: Technical Support +``` + +## Default Prompt + +If no `prompts` are specified, promptfoo uses `{{prompt}}` as a passthrough. This is useful when the input itself is the prompt. + +## Executable Scripts + +You can also use an executable script as a prompt: + +```yaml +prompts: + - exec: python generate_prompt.py +``` + +Use sparingly; dynamic prompt functions are usually cleaner. + +## Next Step + +- Providers: `phases/05-providers.md` diff --git a/.devin/promptfoo/phases/05-providers.md b/.devin/promptfoo/phases/05-providers.md new file mode 100644 index 0000000..f4b2913 --- /dev/null +++ b/.devin/promptfoo/phases/05-providers.md @@ -0,0 +1,181 @@ +# Phase 5: Providers + +Providers are the interfaces to LLMs and AI services. `providers` and `targets` are interchangeable in config. + +## Provider Syntax + +### Simple String + +```yaml +providers: + - openai:chat:gpt-5-mini + - anthropic:messages:claude-opus-4-6 + - ollama:chat:llama3.3 +``` + +### Object with Config + +```yaml +providers: + - id: openai:chat:gpt-5-mini + label: fast-model + config: + temperature: 0.7 + max_tokens: 150 +``` + +### File-Based + +```yaml +providers: + - file://provider.yaml + - file://providers.json +``` + +A single provider file looks like: + +```yaml +id: openai:chat:gpt-5-mini +label: Foo bar +config: + temperature: 0.9 +``` + +A multi-provider file is an array of such objects. + +## Common Configuration Options + +Many providers support these fields in `config`: + +| Option | Description | +|--------|-------------| +| `temperature` | Randomness (0.0-1.0) | +| `max_tokens` / `max_completion_tokens` / `max_output_tokens` | Token limits | +| `top_p` | Nucleus sampling | +| `frequency_penalty` | Penalize frequent tokens | +| `presence_penalty` | Penalize new tokens | +| `stop` | Stop sequences | +| `seed` | Deterministic seed | +| `apiKey` / `apiKeyEnvar` | Auth inline or via env var | +| `apiBaseUrl` / `apiHost` | Custom endpoint | +| `headers` | Extra HTTP headers | +| `maxRetries` | Retry count | +| `passthrough` | Provider-specific extra fields | + +## Authentication Patterns + +Preferred: environment variables. + +```bash +export OPENAI_API_KEY=sk-... +export ANTHROPIC_API_KEY=sk-... +export OLLAMA_BASE_URL=http://localhost:11434 +``` + +Fallback: inline config. + +```yaml +providers: + - id: openai:chat:gpt-5-mini + config: + apiKey: sk-... +``` + +## Built-in Provider Examples + +### OpenAI + +```yaml +providers: + - id: openai:chat:gpt-5-mini + config: + temperature: 0 + max_completion_tokens: 128 + apiKey: sk-abc123 +``` + +Common OpenAI config also includes: + +- `response_format` (`json_object`, `json_schema`) +- `tools`, `tool_choice`, `function_call` +- `reasoning` (for o-series) +- `inputCost`, `outputCost`, `audioInputCost`, `audioOutputCost` +- `organization`, `store`, `metadata`, `user` + +### Anthropic + +```yaml +providers: + - id: anthropic:messages:claude-opus-4-6 + config: + max_tokens: 1000 + temperature: 0.7 +``` + +Supports tool calling, images/vision, prompt caching, citations, PDFs, extended thinking, and structured outputs. + +### Ollama (Local) + +```yaml +providers: + - id: ollama:chat:llama3.3 + config: + temperature: 0.7 + num_predict: 1024 + passthrough: + keep_alive: '5m' + format: 'json' +``` + +Environment variables: + +- `OLLAMA_BASE_URL` (default `http://localhost:11434`) +- `OLLAMA_API_KEY` +- `REQUEST_TIMEOUT_MS` + +Use `ollama:completion:` for the `/api/generate` endpoint and `ollama:chat:` for `/api/chat`. Use `ollama:embeddings:` for similarity assertions. + +## Custom Integration Types + +| Type | Syntax | Use Case | +|------|--------|----------| +| File-based config | `file://provider.yaml` | Reusable provider definitions | +| JavaScript provider | `file://custom_provider.js` | Custom JS/TS logic | +| Python provider | `id: file://custom_provider.py` | Custom Python logic | +| HTTP/HTTPS | `id: https://api.example.com/v1/chat` | Generic REST API | +| WebSocket | `id: ws://example.com/ws` | Streaming/real-time API | +| Exec script | `exec: python chain.py` | One-off command | +| MCP | `mcp://...` | Model Context Protocol servers | + +## Provider-Specific Test Filtering + +Run a test only on certain providers: + +```yaml +providers: + - id: openai:gpt-5-mini + label: fast-model + - id: openai:gpt-5.6 + label: smart-model + +tests: + - vars: + question: 'What is 2+2?' + providers: + - fast-model + - vars: + question: 'Explain quantum entanglement' + providers: + - smart-model +``` + +Matching supports exact labels, `provider:model`, or wildcards like `openai:*`. + +## Model Context Protocol (MCP) + +promptfoo supports MCP servers for tool use and agentic capabilities. Configure MCP providers under the provider or target section. See the upstream docs for server-specific setup. + +## Next Step + +- Custom providers: `phases/07-custom-providers.md` +- Assertions: `phases/06-assertions-metrics.md` diff --git a/.devin/promptfoo/phases/06-assertions-metrics.md b/.devin/promptfoo/phases/06-assertions-metrics.md new file mode 100644 index 0000000..613f687 --- /dev/null +++ b/.devin/promptfoo/phases/06-assertions-metrics.md @@ -0,0 +1,207 @@ +# Phase 6: Assertions and Metrics + +Assertions validate LLM outputs. They are grouped into deterministic (programmatic) and model-assisted (LLM/ML-based) types. + +## Using Assertions + +Add an `assert` array to a test case: + +```yaml +tests: + - vars: + question: 'What is 2+2?' + assert: + - type: equals + value: '4' + - type: contains + value: 'four' +``` + +Every assertion can be negated by prepending `not-`: + +```yaml +assert: + - type: not-equals + value: 'I cannot help with that' + - type: not-contains + value: 'error' +``` + +## Deterministic Assertions + +| Type | What it checks | +|------|----------------| +| `equals` | Exact match | +| `contains` / `icontains` | Substring match (case-sensitive / insensitive) | +| `contains-any` / `contains-all` / `icontains-any` / `icontains-all` | Multiple substrings | +| `regex` | Regular expression match | +| `starts-with` | Prefix match | +| `is-json` / `contains-json` | Valid JSON, optionally with schema | +| `is-html` / `contains-html` | HTML presence/validity | +| `is-sql` / `contains-sql` | SQL presence/validity | +| `is-xml` / `contains-xml` | XML presence/validity | +| `is-refusal` / `not-is-refusal` | Refusal detection | +| `javascript` | Custom JS function | +| `python` | Custom Python function | +| `webhook` | External HTTP endpoint | +| `latency` | Response time below threshold (ms) | +| `cost` | Cost below threshold | +| `finish-reason` | Stop reason (stop, length, content_filter, tool_calls) | +| `levenshtein` | Edit distance | +| `rouge-n`, `bleu`, `gleu`, `meteor` | Text similarity metrics | +| `perplexity` / `perplexity-score` | Model confidence (requires logprobs) | +| `trace-span-count`, `trace-span-duration`, `trace-error-spans` | Trace assertions | +| `trajectory:tool-used`, `trajectory:tool-args-match`, `trajectory:tool-sequence`, `trajectory:step-count` | Agent trajectory | +| `skill-used` | Skill usage | +| `classifier` | Classification model | +| `select-best` | Select best output | +| `word-count` | Word count bounds | +| `assert-set` | Group of assertions | + +### Examples + +```yaml +assert: + - type: contains-json + value: file://schema.json + + - type: regex + value: '\d{4}' + + - type: latency + threshold: 5000 + + - type: cost + threshold: 0.001 + + - type: finish-reason + value: stop +``` + +## Model-Assisted Assertions + +| Type | Purpose | +|------|---------| +| `llm-rubric` | LLM judges output against a rubric | +| `model-graded-closedqa` | Closed QA evaluation | +| `factuality` | Factual consistency | +| `similar` | Embedding cosine similarity | +| `classifier` | ML classifier | +| `moderation` | Content moderation | +| `g-eval` | G-Eval scoring | +| `answer-relevance` | Relevance of answer to question | +| `context-faithfulness` | Output supported by context (RAG) | +| `context-recall` | Ground truth appears in context | +| `context-relevance` | Context relevant to query | +| `conversation-relevance` | Conversation relevance | +| `trajectory:goal-success` | Agent achieved goal | +| `pi` | Pi Labs preference scoring | +| `max-score` | Max score across graders | + +### Example: LLM Rubric + +```yaml +assert: + - type: llm-rubric + value: Is not apologetic and provides a clear, concise answer + provider: openai:gpt-5-mini + threshold: 0.8 +``` + +### Example: Similarity + +```yaml +assert: + - type: similar + value: 'Hello world' + threshold: 0.8 + provider: huggingface:sentence-similarity:sentence-transformers/all-MiniLM-L6-v2 +``` + +## Overriding the Grader + +Set the LLM used for model-graded assertions at three levels: + +1. CLI: `promptfoo eval --grader openai:gpt-5.6` +2. Per-test or suite: `defaultTest.options.provider` +3. Per-assertion: `assertion.provider` + +```yaml +defaultTest: + options: + provider: openai:gpt-5-mini + +# or +tests: + - assert: + - type: llm-rubric + value: Is spoken like a pirate + provider: + id: openai:gpt-5.6 + config: + temperature: 0 +``` + +## Assertion Sets + +Require all or a fraction of assertions to pass: + +```yaml +assert: + - type: assert-set + threshold: 0.5 + assert: + - type: cost + threshold: 0.001 + - type: latency + threshold: 200 +``` + +## Named Metrics + +Tag assertions to aggregate them: + +```yaml +tests: + - assert: + - type: equals + value: Yarr + metric: Tone + - type: icontains + value: grub + metric: Tone + - type: is-json + metric: Consistency +``` + +## Derived Metrics + +Compute composite scores after evaluation: + +```yaml +derivedMetrics: + - name: f1_score + value: '2 * precision * recall / (precision + recall)' + + - name: avg_tone + value: 'Tone / __count' +``` + +`value` can be a mathematical expression or a JavaScript function. Errors are logged at debug level and do not fail the eval. + +## CSV Inline Assertions + +In CSV/XLSX test files, use `__expected` columns: + +```csv +input,__expected +"Hello world","contains: Hello" +"Calculate 5 * 6","equals: 30" +"What's the weather?","llm-rubric: Provides weather information" +``` + +Multiple assertions use `__expected1`, `__expected2`, etc. + +## Next Step + +- Custom assertion scripts: `phases/08-evaluation-scripts.md` diff --git a/.devin/promptfoo/phases/07-custom-providers.md b/.devin/promptfoo/phases/07-custom-providers.md new file mode 100644 index 0000000..8d92b1a --- /dev/null +++ b/.devin/promptfoo/phases/07-custom-providers.md @@ -0,0 +1,288 @@ +# Phase 7: Custom Providers + +Custom providers let you integrate any API or custom logic into promptfoo. Supported formats: JavaScript, TypeScript, Python, HTTP/HTTPS, WebSocket, and exec scripts. + +## JavaScript / TypeScript Provider + +A custom provider must implement at minimum an `id()` method and a `callApi()` method. + +### Minimal Example + +```javascript +// echoProvider.mjs +export default class EchoProvider { + id = () => 'echo'; + + callApi = async (prompt, context, options) => { + return { + output: `Echo: ${prompt}`, + }; + }; +} +``` + +```yaml +providers: + - id: file://echoProvider.mjs +``` + +### With Constructor + +```javascript +const promptfoo = require('promptfoo').default; + +module.exports = class OpenAIProvider { + constructor(options) { + this.providerId = options.id || 'openai-custom'; + this.config = options.config; + } + + id() { + return this.providerId; + } + + async callApi(prompt, context, options) { + const { data } = await promptfoo.cache.fetchWithCache( + 'https://api.openai.com/v1/chat/completions', + { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + Authorization: `Bearer ${process.env.OPENAI_API_KEY}`, + }, + body: JSON.stringify({ + model: this.config?.model || 'gpt-5-mini', + messages: [{ role: 'user', content: prompt }], + max_completion_tokens: this.config?.max_tokens || 1024, + temperature: this.config?.temperature || 1, + }), + }, + ); + + return { + output: data.choices[0].message.content, + tokenUsage: data.usage, + }; + } +}; +``` + +### ProviderResponse Contract + +`callApi` must return an object with these fields: + +```javascript +{ + output: 'Model response text or structured data', + error: 'Error message if applicable', // optional + prompt: 'The actual prompt sent to the LLM', // optional, shown in UI + tokenUsage: { total: 100, prompt: 50, completion: 50 }, // optional + cost: 0.002, // optional + cached: false, // optional + latencyMs: 150, // optional + conversationEnded: false, // optional (multi-turn redteam) + conversationEndReason: 'thread_closed', // optional + metadata: {}, // optional +} +``` + +### Context Parameter + +`context` contains test case information and utilities: + +```javascript +{ + vars: {}, // Test case variables + prompt: {}, // Prompt template (raw, label, config) + test: { // Full test case + vars: {}, + metadata: { + pluginId: '...', // Redteam plugin + strategyId: '...', // Redteam strategy + }, + }, + originalProvider: {}, // Original provider when overridden + logger: {}, // Winston logger +} +``` + +### TypeScript Example + +```typescript +import promptfoo from 'promptfoo'; +import type { ApiProvider, ProviderOptions, ProviderResponse, CallApiContextParams } from 'promptfoo'; + +export default class TypedProvider implements ApiProvider { + protected providerId: string; + public config: Record; + + constructor(options: ProviderOptions) { + this.providerId = options.id || 'typed-provider'; + this.config = options.config || {}; + } + + id(): string { + return this.providerId; + } + + async callApi(prompt: string, context?: CallApiContextParams): Promise { + const username = (context?.vars?.username as string) || 'anonymous'; + return { + output: `Hello, ${username}! You said: "${prompt}"`, + tokenUsage: { + total: prompt.length, + prompt: prompt.length, + completion: 0, + }, + }; + } +} +``` + +### Additional Capabilities + +JavaScript providers can also implement: + +- `callEmbeddingApi` for embeddings +- `callClassificationApi` for classification +- Multimodal content handling (images, audio) + +## Python Provider + +### Minimal Example + +```python +# echo_provider.py +def call_api(prompt, options, context): + config = options.get('config', {}) + prefix = config.get('prefix', 'Tell me about: ') + return { + "output": f"{prefix}{prompt}", + } +``` + +```yaml +providers: + - id: 'file://echo_provider.py' +``` + +### Function Interface + +Your Python script must implement one or more of these functions: + +```python +def call_api(prompt: str, options: dict, context: dict) -> dict: + pass + +def call_embedding_api(prompt: str, options: dict) -> dict: + pass + +def call_classification_api(prompt: str, options: dict) -> dict: + pass +``` + +Async versions (`async def`) are also supported. + +### Parameters + +- `prompt`: string, or JSON-encoded chat messages `[{"role": "user", "content": "..."}]` +- `options`: `{ "id": "file://...", "config": { ... } }` plus `basePath` +- `context` (only for `call_api`): `{ "vars": {}, "prompt": {}, "test": { "metadata": {} } }` + +### Return Format + +```python +{ + "output": "Your response here", + "tokenUsage": {"total": 150, "prompt": 50, "completion": 100}, + "cost": 0.0025, + "cached": False, + "latencyMs": 150, + "conversationEnded": False, + "conversationEndReason": "thread_closed", + "error": "Description of what went wrong", +} +``` + +### Configuration + +```yaml +providers: + - id: file://./my_provider.py + label: 'My Custom API' + config: + model: 'gpt-5' + temperature: 0.7 + max_tokens: 2000 +``` + +You can also set a per-provider Python executable: + +```yaml +providers: + - id: 'file://my_provider.py' + config: + pythonExecutable: /path/to/specific/python +``` + +## HTTP/HTTPS Provider + +```yaml +providers: + - id: https://api.example.com/v1/chat/completions + config: + headers: + Authorization: 'Bearer your_api_key' + method: 'POST' + body: + model: 'my-model' + messages: '[{"role":"user","content":"{{prompt}}"}]' +``` + +## WebSocket Provider + +```yaml +providers: + - id: ws://example.com/ws + config: + messageTemplate: '{"prompt": "{{prompt}}"}' +``` + +## Exec Provider + +```yaml +providers: + - 'exec: python chain.py' +``` + +## Caching in Custom Providers + +Use the built-in cache for HTTP calls: + +```javascript +const { data, cached } = await promptfoo.cache.fetchWithCache( + url, + { method: 'POST', body: JSON.stringify(payload) }, + 5000, // timeout ms +); +``` + +## Multiple Instances + +Reuse the same provider file with different config/labels: + +```yaml +providers: + - id: file:///path/to/provider.js + label: high-temperature + config: + temperature: 0.9 + - id: file:///path/to/provider.js + label: low-temperature + config: + temperature: 0.1 +``` + +## Next Step + +- Custom assertions and transforms: `phases/08-evaluation-scripts.md` diff --git a/.devin/promptfoo/phases/08-evaluation-scripts.md b/.devin/promptfoo/phases/08-evaluation-scripts.md new file mode 100644 index 0000000..1cfe2a0 --- /dev/null +++ b/.devin/promptfoo/phases/08-evaluation-scripts.md @@ -0,0 +1,252 @@ +# Phase 8: Evaluation Scripts and Node Package + +Custom scripts let you implement custom assertions, transforms, and dynamic prompts. You can also use promptfoo as a Node library. + +## Custom JavaScript Assertions + +### Inline + +```yaml +assert: + - type: javascript + value: output.toLowerCase().includes('bananas') +``` + +### Multiline + +```yaml +assert: + - type: javascript + value: | + if (output === 'Expected output') { + return { pass: true, score: 0.5 }; + } + return { pass: false, score: 0, reason: 'Assertion failed' }; +``` + +### External File + +```yaml +assert: + - type: javascript + value: file://relative/path/to/script.js + config: + maximumOutputSize: 10 +``` + +Specify a named function: + +```yaml +assert: + - type: javascript + value: file://relative/path/to/script.js:customFunction +``` + +The file must export a function taking `(output, context)`: + +```javascript +module.exports = (output, context) => { + return output.length <= context.config.maximumOutputSize; +}; +``` + +### Return Types + +- `boolean` — pass/fail +- `number` — score (0 = fail, higher is better) +- `GradingResult` — full result object + +```javascript +return { + pass: true, + score: 0.75, + reason: 'Looks good to me', + componentResults: [ + { pass: true, score: 0.5, reason: 'Contains banana' }, + { pass: true, score: 0.5, reason: 'Contains yellow' }, + ], +}; +``` + +### Context for JavaScript Assertions + +```javascript +{ + prompt: string | undefined; + vars: Record; + test: AtomicTestCase; + logProbs: number[] | undefined; + config?: Record; + provider: ApiProvider | undefined; + providerResponse: ProviderResponse | undefined; + trace?: TraceData; + metadata?: Record; +} +``` + +## Custom Python Assertions + +### Inline + +```yaml +assert: + - type: python + value: 'len(output) > 10' +``` + +### External File + +```yaml +assert: + - type: python + value: file://relative/path/to/script.py + config: + outputLengthLimit: 10 +``` + +Named function: + +```yaml +assert: + - type: python + value: file://relative/path/to/script.py:custom_assert +``` + +Default function name is `get_assert`. + +```python +from typing import Dict, Union + +def get_assert(output: str, context) -> Union[bool, float, Dict[str, any]]: + return { + 'pass': True, + 'score': 0.6, + 'reason': 'Looks good to me', + } +``` + +### GradingResult Mapping + +Python snake_case keys are mapped to camelCase: + +- `pass_` / `'pass'` → `pass` +- `named_scores` → `namedScores` +- `component_results` → `componentResults` +- `tokens_used` → `tokensUsed` + +### Context for Python Assertions + +Same shape as JavaScript, available as `context['vars']`, `context['prompt']`, `context['providerResponse']`, etc. + +## Transforms + +Transforms modify the output before it is graded. They can be defined at test, assertion, or global level. + +```yaml +tests: + - vars: + question: 'What is 2+2?' + options: + transform: output.toLowerCase() + assert: + - type: contains + value: 'four' +``` + +Transforms can be JavaScript/Python expressions or functions: + +```yaml +tests: + - options: + transform: file://transform.js +``` + +Execution order: global → test → assertion. + +## Node Package API + +Install: + +```bash +npm install promptfoo +``` + +Basic usage: + +```javascript +import promptfoo from 'promptfoo'; + +const evalRecord = await promptfoo.evaluate( + { + prompts: ['Rephrase in French: {{body}}'], + providers: ['openai:gpt-5-mini'], + tests: [ + { vars: { body: 'Hello world' } }, + { vars: { body: "I'm hungry" } }, + ], + writeLatestResults: true, + }, + { maxConcurrency: 2 }, +); + +const results = await evalRecord.toEvaluateSummary(); +console.log(results); +``` + +### Function-Based Providers and Assertions + +```javascript +await promptfoo.evaluate({ + prompts: [ + 'Rephrase in French: {{body}}', + (vars) => `Rephrase like a pirate: ${vars.body}`, + ], + providers: [ + 'openai:gpt-5-mini', + (prompt, context) => { + return { output: '' }; + }, + ], + tests: [ + { + vars: { body: 'Hello world' }, + assert: [ + { + type: 'javascript', + value: (output) => ({ + pass: output.includes("J'ai faim"), + score: output.includes("J'ai faim") ? 1 : 0, + reason: 'Output contained substring', + }), + }, + ], + }, + ], +}); +``` + +### Loading Providers Programmatically + +```javascript +import { loadApiProvider } from 'promptfoo'; + +const provider = await loadApiProvider('openai:o3-mini'); +const providerWithOptions = await loadApiProvider('azure:chat:test', { + options: { apiHost: 'test-host', apiKey: 'test-key' }, +}); +``` + +## Extension Hooks + +Register hooks for lifecycle events: + +```yaml +extensions: + - file://hooks.js:afterAll +``` + +Common hooks: `beforeAll`, `afterAll`, `beforeEach`, `afterEach`, `beforeEachTest`, `afterEachTest`. Useful for setup/teardown and custom reporting. + +## Next Step + +- CLI commands and environment variables: `phases/09-cli-commands.md` diff --git a/.devin/promptfoo/phases/09-cli-commands.md b/.devin/promptfoo/phases/09-cli-commands.md new file mode 100644 index 0000000..87a9ee8 --- /dev/null +++ b/.devin/promptfoo/phases/09-cli-commands.md @@ -0,0 +1,243 @@ +# Phase 9: CLI Commands and Environment Variables + +## Common CLI Options + +Most commands support: + +- `--env-file, --env-path ` — load `.env` files +- `-v, --verbose` — verbose output (note: for `eval`, `-v` is `--vars`; use `--verbose` or `LOG_LEVEL=debug`) +- `--help` + +## `promptfoo eval` + +Run an evaluation. Reads `promptfooconfig.yaml` by default. + +```bash +promptfoo eval +promptfoo eval -c my-config.yaml +``` + +### Key Flags + +| Flag | Purpose | +|------|---------| +| `-c, --config ` | Config file(s) | +| `-p, --prompts ` | Override prompts | +| `-r, --providers ` | Override providers | +| `-t, --tests ` | Override test file | +| `-a, --assertions ` | Override assertions | +| `-o, --output ` | Output file(s) | +| `-j, --max-concurrency ` | Parallelism | +| `--delay ` | Delay between requests | +| `--repeat ` | Repeat each test N times | +| `--no-cache` | Disable caching | +| `--no-write` | Do not write latest results | +| `--no-table` | Skip table output | +| `--no-progress-bar` | Quiet progress | +| `-w, --watch` | Watch files and rerun | +| `--resume [evalId]` | Resume a previous eval | +| `--retry-errors` | Retry failed/error cases | +| `--grader ` | Grader for model-graded assertions | +| `--tag ` | Run metadata tags | +| `--var ` | Override a single variable | +| `--vars ` | Load variables from file | +| `--filter-pattern ` | Filter tests by description | +| `--filter-providers ` | Filter providers | +| `--filter-prompts ` | Filter prompts | +| `--filter-metadata ` | Filter tests by metadata | +| `--filter-range ` | Run a stable index slice | +| `--filter-sample ` | Random sample of N tests | +| `--filter-sample-seed ` | Seed for random sample | +| `--filter-failing ` | Re-run tests that failed in a previous eval | +| `--filter-errors-only ` | Re-run only errors | +| `--suggest-prompts ` | Generate prompt suggestions | + +### Exit Codes + +- `0` — all tests passed +- `100` — at least one failure or pass rate below `PROMPTFOO_PASS_RATE_THRESHOLD` +- `1` — any other error +- Override failed-test exit code with `PROMPTFOO_FAILED_TEST_EXIT_CODE` + +## `promptfoo view` + +Start the interactive web UI: + +```bash +promptfoo view +promptfoo view -p 3000 +``` + +If you changed `PROMPTFOO_CONFIG_DIR`, pass the directory: + +```bash +promptfoo view /path/to/config-dir +``` + +## `promptfoo optimize` + +Improve one prompt against one provider: + +```bash +promptfoo optimize +promptfoo optimize -c path/to/config.yaml --prompt-index 1 --provider-index 0 +promptfoo optimize --validation-split 0.2 +``` + +Without `--validation-split`, optimization may overfit to the configured test set. + +## `promptfoo validate` + +```bash +promptfoo validate config +promptfoo validate target +``` + +## `promptfoo init` + +Create a new project or example: + +```bash +promptfoo init +promptfoo init my-project --example getting-started +``` + +## `promptfoo generate dataset` + +Generate synthetic test cases: + +```bash +promptfoo generate dataset -w +promptfoo generate dataset -c my_config.yaml -o new_tests.yaml -i 'European cities only' +``` + +## `promptfoo generate assertions` + +Generate additional assertions: + +```bash +promptfoo generate assertions -w +promptfoo generate assertions -c my_config.yaml -o new_asserts.yaml -i 'pronunciation checks' +``` + +## `promptfoo cache` + +```bash +promptfoo cache clear +``` + +## `promptfoo logs` + +```bash +promptfoo logs --list +promptfoo logs +``` + +## `promptfoo debug` + +Display debug information for troubleshooting: + +```bash +promptfoo debug -c my-config.yaml +``` + +## `promptfoo share [id]` + +Create a shareable URL for the most recent eval or a specific ID. + +## `promptfoo list` + +```bash +promptfoo list evals +promptfoo list prompts +promptfoo list datasets +``` + +## `promptfoo export` + +```bash +promptfoo export eval +promptfoo export logs +``` + +## Environment Variables + +promptfoo reads `.env` from the working directory. + +### General + +| Variable | Purpose | Default | +|----------|---------|---------| +| `PROMPTFOO_CACHE_ENABLED` | Enable cache | `true` | +| `PROMPTFOO_CACHE_TYPE` | `disk` or `memory` | `disk` | +| `PROMPTFOO_CACHE_PATH` | Cache directory | `~/.promptfoo/cache` | +| `PROMPTFOO_CACHE_TTL` | Cache TTL seconds | `1209600` | +| `PROMPTFOO_CONFIG_DIR` | Config/data directory | `~/.promptfoo` | +| `PROMPTFOO_LOG_DIR` | Log directory | `~/.promptfoo/logs` | +| `PROMPTFOO_EVAL_TIMEOUT_MS` | Per-request timeout | — | +| `PROMPTFOO_MAX_EVAL_TIME_MS` | Total eval timeout | — | +| `PROMPTFOO_PASS_RATE_THRESHOLD` | Minimum pass rate | — | +| `PROMPTFOO_FAILED_TEST_EXIT_CODE` | Override exit code 100 | — | +| `PROMPTFOO_MAX_EVAL_TIME_MS` | Total eval time limit | — | +| `PROMPTFOO_REQUIRE_JSON_PROMPTS` | Require JSON prompts | `false` | +| `PROMPTFOO_DISABLE_UPDATE` | Disable update checks | `false` | +| `PROMPTFOO_DISABLE_RUNTIME_WARNINGS` | Disable runtime warnings | `false` | +| `PROMPTFOO_DISABLE_REMOTE_GENERATION` | Disable remote generation | `false` | +| `PROMPTFOO_DISABLE_TEMPLATING` | Disable Nunjucks | `false` | +| `PROMPTFOO_DISABLE_VAR_EXPANSION` | Disable var expansion | `false` | +| `PROMPTFOO_DISABLE_JSON_AUTOESCAPE` | Disable JSON escaping | `false` | +| `PROMPTFOO_DISABLE_OBJECT_STRINGIFY` | Disable object stringify | `false` | +| `PROMPTFOO_DISABLE_ERROR_LOG` | Disable error log | `false` | +| `PROMPTFOO_DISABLE_DEBUG_LOG` | Disable debug log | `false` | + +### Memory/Output Stripping + +| Variable | Purpose | +|----------|---------| +| `PROMPTFOO_STRIP_PROMPT_TEXT` | Strip prompt text from outputs | +| `PROMPTFOO_STRIP_RESPONSE_OUTPUT` | Strip model outputs | +| `PROMPTFOO_STRIP_TEST_VARS` | Strip test variables | +| `PROMPTFOO_STRIP_GRADING_RESULT` | Strip grading results | +| `PROMPTFOO_STRIP_METADATA` | Strip metadata | + +### Python Integration + +| Variable | Purpose | +|----------|---------| +| `PROMPTFOO_PYTHON` | Path to Python executable | +| `PROMPTFOO_PYTHON_DEBUG_ENABLED` | Enable `pdb` debugging | + +### Provider-Specific + +| Variable | Provider | +|----------|----------| +| `OPENAI_API_KEY` | OpenAI | +| `OPENAI_BASE_URL` | OpenAI custom endpoint | +| `ANTHROPIC_API_KEY` | Anthropic | +| `OLLAMA_BASE_URL` | Ollama | +| `OLLAMA_API_KEY` | Ollama | +| `REQUEST_TIMEOUT_MS` | Ollama | +| `WITHPI_API_KEY` | Pi scorer | + +## Command-Line Options in Config + +Set defaults in `promptfooconfig.yaml`: + +```yaml +commandLineOptions: + maxConcurrency: 10 + repeat: 3 + delay: 1000 + verbose: true + grader: openai:gpt-5-mini + cache: false + filterPattern: 'auth.*' + var: + temperature: '0.7' +``` + +CLI flags override these defaults. + +## Next Step + +- Debugging: `phases/10-debugging-scenarios.md` diff --git a/.devin/promptfoo/phases/10-debugging-scenarios.md b/.devin/promptfoo/phases/10-debugging-scenarios.md new file mode 100644 index 0000000..75916ee --- /dev/null +++ b/.devin/promptfoo/phases/10-debugging-scenarios.md @@ -0,0 +1,220 @@ +# Phase 10: Debugging and Troubleshooting + +## Log Files and Debugging + +Default log directory: `~/.promptfoo/logs` + +- View from CLI: `promptfoo logs --list` and `promptfoo logs ` +- Custom directory: `PROMPTFOO_LOG_DIR=./logs promptfoo eval` +- Export logs: `promptfoo export logs` +- Enable debug logs: `LOG_LEVEL=debug npx promptfoo eval` or `npx promptfoo eval --verbose` +- Live debug toggle: `promptfoo eval` with verbose mode to see real-time details + +## Scenario: Out of Memory + +For large evals, reduce memory pressure: + +```bash +# Required: do not use --no-write +# Avoid table rendering +promptfoo eval --no-table --output results.jsonl +``` + +Strip heavy data: + +```bash +export PROMPTFOO_STRIP_PROMPT_TEXT=true +export PROMPTFOO_STRIP_RESPONSE_OUTPUT=true +export PROMPTFOO_STRIP_TEST_VARS=true +export PROMPTFOO_STRIP_GRADING_RESULT=true +export PROMPTFOO_STRIP_METADATA=true +``` + +Increase Node.js heap: + +```bash +NODE_OPTIONS="--max-old-space-size=8192" npx promptfoo eval +``` + +## Scenario: Stuck or Slow Evals + +Set timeouts: + +```bash +export PROMPTFOO_EVAL_TIMEOUT_MS=30000 # per request +export PROMPTFOO_MAX_EVAL_TIME_MS=300000 # total eval +``` + +Or in config: + +```yaml +env: + PROMPTFOO_EVAL_TIMEOUT_MS: 30000 + PROMPTFOO_MAX_EVAL_TIME_MS: 300000 +``` + +Reduce concurrency for rate-limited providers: + +```bash +promptfoo eval -j 1 --delay 1000 +``` + +Disable cache only when fresh calls are required: + +```bash +promptfoo eval --no-cache +``` + +## Scenario: Custom Python Provider Not Found + +Symptoms: `spawn py -3 ENOENT` or `Python 3 not found` + +Fixes: + +1. Set the Python path: + + ```bash + export PROMPTFOO_PYTHON=/usr/local/bin/python3 + ``` + +2. Or set per-provider: + + ```yaml + providers: + - id: 'file://my_provider.py' + config: + pythonExecutable: /path/to/python + ``` + +3. Test that Python is reachable: + + ```bash + python -c "import sys; print(sys.executable)" + ``` + +## Scenario: Debugging Python Providers/Assertions + +Enable debug output: + +```bash +LOG_LEVEL=debug npx promptfoo eval +``` + +Use `pdb`: + +```bash +export PROMPTFOO_PYTHON_DEBUG_ENABLED=true +``` + +Add breakpoints in your Python code: + +```python +import pdb + +def call_api(prompt, options, context): + pdb.set_trace() + # ... +``` + +## Scenario: Custom JavaScript Provider Not Loading + +- Ensure the file exports a default class or function +- Check that the path is correct relative to the config file +- Use `LOG_LEVEL=debug` to see load errors +- Verify Node.js version is `^20.20.0` or `>=22.22.0` + +## Scenario: Provider Authentication Errors + +- Confirm the relevant API key env var is set +- For inline keys, verify the provider `config.apiKey` value +- Check `apiBaseUrl` / `apiHost` if using a proxy or custom endpoint +- Use `promptfoo debug -c config.yaml` to inspect resolved config + +## Scenario: Object Template Handling + +If you see `[object Object]` in prompts or outputs, you are passing an object variable without property access or a filter. + +Fix: + +```yaml +# Instead of {{user}} +# Use {{user.name}} or {{user | json}} +``` + +## Scenario: Network / Proxy Issues + +Configure proxy: + +```bash +export HTTP_PROXY=http://proxy.example.com:8080 +export HTTPS_PROXY=http://proxy.example.com:8080 +``` + +Custom CA certificates: + +```bash +export NODE_EXTRA_CA_CERTS=/path/to/cert.pem +``` + +## Scenario: Ollama Connection Refused + +Common causes: + +- Ollama not listening on `0.0.0.0` for remote connections +- IPv4/IPv6 mismatch with `localhost` + +Fixes: + +```bash +# Use explicit IPv4 +export OLLAMA_BASE_URL=http://127.0.0.1:11434 + +# Or start Ollama on all interfaces +export OLLAMA_HOST=0.0.0.0:11434 +ollama serve +``` + +## Scenario: Assertion Failures Are Hard to Understand + +- Use `promptfoo view` to inspect the actual prompt, output, and assertion result +- Return full `GradingResult` objects from custom assertions with a `reason` field +- Add `description` fields to test cases for easier filtering + +## Scenario: Caching Surprises Results + +- `--no-cache` ensures fresh calls +- `--repeat` uses separate cache entries; add `--no-cache` if every repeat must call the provider +- Clear cache: `promptfoo cache clear` + +## Scenario: Runtime Warnings / Update Checks + +Disable if needed: + +```bash +export PROMPTFOO_DISABLE_RUNTIME_WARNINGS=true +export PROMPTFOO_DISABLE_UPDATE=true +``` + +## Validation Before Debugging + +Run config validation before a full eval: + +```bash +promptfoo validate config +promptfoo validate target +``` + +## Debugging Checklist + +1. Check logs at `~/.promptfoo/logs` or `PROMPTFOO_LOG_DIR` +2. Run with `LOG_LEVEL=debug` or `--verbose` +3. Validate config with `promptfoo validate config` +4. Isolate the failing test with `--filter-pattern` or `--filter-range` +5. Test the provider directly (custom providers can be unit-tested independently) +6. Disable cache if stale results are suspected +7. Check timeouts and concurrency for stuck evals +8. Inspect output stripping options for OOM + +## Next Step + +- Quick reference: `quick-reference.md` diff --git a/.devin/promptfoo/quick-reference.md b/.devin/promptfoo/quick-reference.md new file mode 100644 index 0000000..59d9878 --- /dev/null +++ b/.devin/promptfoo/quick-reference.md @@ -0,0 +1,224 @@ +# Quick Reference: Promptfoo + +Compact lookup for common properties, commands, and patterns. + +## Config File Skeleton + +```yaml +description: My eval + +prompts: + - 'Translate to {{language}}: {{input}}' + +providers: + - openai:chat:gpt-5-mini + - anthropic:messages:claude-opus-4-6 + +tests: + - vars: + language: French + input: Hello world + assert: + - type: icontains + value: 'Bonjour' +``` + +## Common CLI Commands + +```bash +promptfoo init +promptfoo eval +promptfoo eval -c my.yaml -j 4 --no-cache +promptfoo view +promptfoo optimize +promptfoo validate config +promptfoo cache clear +promptfoo logs --list +promptfoo debug -c my.yaml +``` + +## Provider Syntax + +```yaml +# Simple string +providers: + - openai:chat:gpt-5-mini + - anthropic:messages:claude-opus-4-6 + - ollama:chat:llama3.3 + +# Object with config +providers: + - id: openai:chat:gpt-5-mini + label: fast + config: + temperature: 0.7 + max_tokens: 150 + +# Custom provider +providers: + - file://custom_provider.js + - id: file://custom_provider.py + label: my-python-provider +``` + +## Common Provider Config + +| Option | Description | +|--------|-------------| +| `temperature` | Randomness (0-1) | +| `max_tokens` / `max_completion_tokens` / `max_output_tokens` | Token limits | +| `top_p` | Nucleus sampling | +| `frequency_penalty` / `presence_penalty` | Penalties | +| `stop` | Stop sequences | +| `seed` | Deterministic seed | +| `apiKey` / `apiKeyEnvar` | Auth | +| `apiBaseUrl` / `apiHost` | Custom endpoint | +| `headers` | Extra HTTP headers | +| `maxRetries` | Retries | +| `passthrough` | Provider-specific extra fields | + +## Prompt Formats + +```yaml +prompts: + # Text + - 'Translate: {{text}}' + # File + - file://prompt.txt + # Chat JSON + - file://chat.json + # Dynamic JS + - file://generate_prompt.js + # Dynamic Python + - file://generate_prompt.py:fn_name +``` + +## Assertion Types + +### Deterministic + +`equals`, `contains`, `icontains`, `contains-any`, `contains-all`, `icontains-any`, `icontains-all`, `regex`, `starts-with`, `is-json`, `contains-json`, `is-html`, `contains-html`, `is-sql`, `contains-sql`, `is-xml`, `contains-xml`, `is-refusal`, `javascript`, `python`, `webhook`, `latency`, `cost`, `finish-reason`, `levenshtein`, `rouge-n`, `bleu`, `gleu`, `meteor`, `perplexity`, `perplexity-score`, `classifier`, `select-best`, `word-count`, `assert-set`. + +Negate any with `not-` prefix (e.g., `not-contains`). + +### Model-Assisted + +`llm-rubric`, `model-graded-closedqa`, `factuality`, `similar`, `classifier`, `moderation`, `g-eval`, `answer-relevance`, `context-faithfulness`, `context-recall`, `context-relevance`, `conversation-relevance`, `trajectory:goal-success`, `pi`, `max-score`. + +### Assertion Examples + +```yaml +assert: + - type: contains + value: 'expected text' + - type: is-json + value: file://schema.json + - type: latency + threshold: 5000 + - type: llm-rubric + value: Is clear and concise + provider: openai:gpt-5-mini + - type: similar + value: 'expected meaning' + threshold: 0.8 +``` + +## Test Case Properties + +```yaml +tests: + - description: Name + vars: + key: value + assert: [...] + options: + provider: openai:gpt-5-mini + repeat: 2 + transform: output.toLowerCase() + metadata: + category: x + threshold: 0.8 + providers: + - fast-model +``` + +## `defaultTest` + +```yaml +defaultTest: + vars: + audience: developer + assert: + - type: llm-rubric + value: Does not describe itself as an AI + options: + provider: openai:gpt-5-mini +``` + +## Derived Metrics + +```yaml +derivedMetrics: + - name: f1_score + value: '2 * precision * recall / (precision + recall)' +``` + +## Environment Variables + +```bash +OPENAI_API_KEY=... +ANTHROPIC_API_KEY=... +OLLAMA_BASE_URL=http://localhost:11434 +PROMPTFOO_CACHE_ENABLED=true +PROMPTFOO_LOG_DIR=./logs +PROMPTFOO_EVAL_TIMEOUT_MS=30000 +PROMPTFOO_MAX_EVAL_TIME_MS=300000 +PROMPTFOO_PYTHON=/usr/local/bin/python3 +PROMPTFOO_PYTHON_DEBUG_ENABLED=true +LOG_LEVEL=debug +``` + +## Custom JS Provider Contract + +```javascript +export default class MyProvider { + id = () => 'my-provider'; + callApi = async (prompt, context, options) => ({ + output: '...', + tokenUsage: { total: 100, prompt: 50, completion: 50 }, + cost: 0.002, + error: undefined, + }); +} +``` + +## Custom Python Provider Contract + +```python +def call_api(prompt, options, context): + return { + "output": "...", + "tokenUsage": {"total": 100, "prompt": 50, "completion": 50}, + "cost": 0.002, + } +``` + +## Custom Assertion Return + +```javascript +return { + pass: true, + score: 0.75, + reason: 'Looks good', + componentResults: [...], +}; +``` + +```python +return { + 'pass': True, + 'score': 0.75, + 'reason': 'Looks good', + 'component_results': [...], +} +```