Sample code for: AI Agent Evaluation: Build an Eval Harness in Python

agent-evaluation/README.md

@@ -0,0 +1,40 @@
+# AI Agent Evaluation: Build an Eval Harness in Python
+
+This folder contains the sample code for the Real Python tutorial
+[AI Agent Evaluation: Build an Eval Harness in Python](https://realpython.com/agent-evaluation/).
+
+It holds a small helpdesk agent for internal Python packages (`agent.py`,
+`tools.py`) and an eval harness (`agent_evals.py`, `models.py`) that scores
+each run's final answer, trajectory, and tool selection.
+
+- The top-level folder holds the finished project.
+- The `follow_along/` folder holds the starter project. The functions you
+  write in the tutorial are stubs there.
+
+## Setup
+
+```console
+$ python -m venv .venv
+$ source .venv/bin/activate
+(.venv) $ python -m pip install -r requirements.txt
+(.venv) $ export OPENAI_API_KEY="sk-..."
+```
+
+The agent uses `gpt-5.4-mini-2026-03-17`. Set the `AGENT_MODEL` environment
+variable to try another model.
+
+## Usage
+
+```console
+(.venv) $ python agent_evals.py run --prompt prompts/helpdesk_v1.txt
+(.venv) $ python agent_evals.py report runs/helpdesk_v1.jsonl
+(.venv) $ python agent_evals.py compare \
+    runs/helpdesk_v1.jsonl runs/helpdesk_v2_docs_first.jsonl
+```
+
+The optional DeepEval example needs its own requirements file:
+
+```console
+(.venv) $ python -m pip install -r requirements-deepeval.txt
+(.venv) $ python deepeval_tools.py
+```

agent-evaluation/agent.py

@@ -0,0 +1,48 @@
+"""A small helpdesk agent for internal Python packages."""
+
+import json
+import os
+import sys
+from pathlib import Path
+
+from openai import OpenAI
+
+from tools import TOOLS, call_tool
+
+MODEL = os.environ.get("AGENT_MODEL", "gpt-5.4-mini-2026-03-17")
+MAX_TURNS = 6
+client = OpenAI()
+
+
+def run_agent(question: str, instructions: str) -> list[dict]:
+    """Run the agent loop and return the full message history."""
+    messages = [
+        {"role": "system", "content": instructions},
+        {"role": "user", "content": question},
+    ]
+    for _ in range(MAX_TURNS):
+        response = client.chat.completions.create(
+            model=MODEL,
+            messages=messages,
+            tools=TOOLS,
+        )
+        message = response.choices[0].message
+        messages.append(message.to_dict())
+        if not message.tool_calls:
+            break
+        for call in message.tool_calls:
+            arguments = json.loads(call.function.arguments)
+            messages.append(
+                {
+                    "role": "tool",
+                    "tool_call_id": call.id,
+                    "content": call_tool(call.function.name, arguments),
+                }
+            )
+    return messages
+
+
+if __name__ == "__main__":
+    prompt = Path("prompts/helpdesk_v1.txt").read_text()
+    history = run_agent(sys.argv[1], prompt)
+    print(history[-1]["content"])