feat: Add judge evaluation support to agent graphs

[jsonbailey]

Summary

• Adds a new Evaluator class that coordinates per-node judge evaluation; evaluate() returns an asyncio.Task so evaluation fires immediately and is awaited before the graph run returns
• AIAgentConfig (and AICompletionConfig) now carry a pre-built Evaluator as a kw_only dataclass field, constructed eagerly in client._build_evaluator()
• LangGraph: LangGraphAgentGraphRunner stores per-node eval tasks in _pending_eval_tasks during node execution; LangChainCallbackHandler.flush() (now async) awaits them and calls track_judge_result via the same AIConfigTracker used for that node's LLM metrics
• OpenAI: OpenAIAgentGraphRunner fires judge evaluation at handoff and final-segment points, tracked via the node's config tracker
• Evaluator.noop() provides a null-object default so nodes without a judgeConfiguration require no special handling

Test plan

• All existing unit tests pass (make test — 248 tests across 3 packages)
• Lint passes (make lint)
• Manual e2e: run langgraph-multi-agent-example or chat-judge-example via hello-python-ai pointing at this worktree and verify judge events appear in the LD events stream

Closes AIC-2267

🤖 Generated with Claude Code

---

Note

Medium Risk
Introduces asynchronous judge-evaluation execution and wires results into both ManagedModel and agent-graph runners, changing result types and tracker flushing behavior. Risk is moderate due to new concurrency/task handling and API surface changes around create_tracker and evaluations fields.

Overview
Adds a new Evaluator abstraction and threads it through AICompletionConfig/AIAgentConfig so judge evaluations can be kicked off per invocation and tracked automatically.

Updates ManagedModel.invoke() to start evaluation via the config’s evaluator, attach a completion callback to emit track_judge_result, and changes ModelResponse.evaluations to carry an asyncio.Task (while AgentGraphResult now includes collected judge results).

Extends LangGraph execution to schedule per-node evaluation tasks during node invocation, store them per-run using a ContextVar, and make LDMetricsCallbackHandler.flush() async so it can await tasks, track successful judge results per node, and return all results.

Refactors judge initialization in LDAIClient to build evaluators eagerly (including new default_ai_provider plumbing), removes async judge setup from create_model(), and tightens AgentGraphDefinition.create_tracker to be required; OpenAI agent-graph tracking is aligned to always use a graph tracker and now returns token usage in LDAIMetrics.

^{Reviewed by Cursor Bugbot for commit e2f5b93. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2a15009. Configure here.}