Skip to content

Sample code for: AI Agent Evaluation: Build an Eval Harness in Python - #928

Open
lpozo wants to merge 1 commit into
masterfrom
agent-evaluation
Open

lpozo wants to merge 1 commit into
masterfrom
agent-evaluation

Conversation

@lpozo

@lpozo lpozo commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Where to put new files:

  • New files should go into a top-level subfolder, named after the article slug. For example: my-awesome-article

How to merge your changes:

  1. Make sure the CI code style tests all pass (+ run the automatic code formatter if necessary).
  2. Find an RP Team member on Slack and ask them to review & approve your PR.
  3. Once the PR has one positive ("approved") review, GitHub lets you merge the PR.
  4. 🎉

Code examples for the AI Agent Evaluation: Build an Eval Harness in Python tutorial. Files live in the agent-evaluation/ top-level folder, named after the article slug.

  • The top-level folder holds the finished project: a tool-calling help desk agent (agent.py, tools.py), the eval harness (agent_evals.py, models.py), test cases, prompts, and the optional DeepEval example.
  • follow_along/ holds the starter project, with stubs for the functions the reader writes.
  • runs/ isn't shipped, because readers generate their own runs with the tutorial's commands.

Generated from the tutorial markdown with the post-materials skill, then verified locally on Python 3.15 with openai==3.40.0, pydantic==2.14.1, and deepeval==4.3.2 against gpt-5.4-mini-2026-03-17.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant