Skip to content

feat: add Vitest workspace verifier adapter#1480

Merged
christso merged 3 commits into
mainfrom
feat/sdk-deterministic-grader-dx
Jun 22, 2026
Merged

feat: add Vitest workspace verifier adapter#1480
christso merged 3 commits into
mainfrom
feat/sdk-deterministic-grader-dx

Conversation

@christso

@christso christso commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

AgentV deterministic workspace graders can now use normal Vitest verifier files and have each it(...) outcome mapped into AgentV code-grader assertions. The common path no longer needs a per-eval verify-*.mjs or defineVitestWorkspaceGrader() wrapper: eval YAML can call AgentV's built-in adapter directly with agentv eval vitest <verifier.test.ts>.

This keeps framework-specific behavior out of core evaluator types while moving the stdin/stdout, workspace path, Vitest JSON reporter, score aggregation, and hidden-verifier copy/cleanup plumbing into AgentV-owned adapter glue.

What Changed

  • Added SDK Vitest adapter helpers: runVitestWorkspaceGrader(), defineVitestWorkspaceGrader(), and vitestReportToCodeGraderResult().
  • Added agentv eval vitest as a code-grader protocol adapter for plain Vitest verifier files.
  • Added support for verifier files that live beside the eval instead of inside the prepared workspace by copying them into a temporary workspace-local directory before running Vitest.
  • Kept direct workspace file helpers as a lower-level SDK option for tiny one-off checks or custom score shaping.
  • Updated examples/features/vitest-workspace-grader/ so the only grader file is graders/welcome-banner.test.ts; the eval calls the built-in adapter directly.
  • Updated public docs and SDK README to recommend Vitest verifiers first, then lower-level code graders/helpers when custom behavior is needed.

Verification

  • bun --filter @agentv/sdk test
  • bun --filter @agentv/sdk typecheck
  • bun --filter @agentv/sdk lint
  • bun --filter @agentv/sdk build
  • bun --filter agentv test
  • bun --filter agentv typecheck
  • bun --filter agentv lint
  • bun --filter agentv build
  • bun run build
  • bun run validate:examples
  • bun apps/cli/src/cli.ts eval examples/features/vitest-workspace-grader/evals/dataset.eval.yaml --target mock_agent passes at 100%; JSONL shows 4 Vitest tests mapped to 4 AgentV assertions.

Tracker

Related: EntityProcess/agentv-beads#11. Beads worktree redirect was repaired separately; notes were pushed for av-bv4.16, av-bv4.17, and av-bv4.19. av-bv4.18 remains the dogfood follow-up for moving the next-evals welcome-banner case to the built-in Vitest path once this AgentV branch is merged/released or otherwise available to that repo.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 22, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 23ce574
Status: ✅  Deploy successful!
Preview URL: https://25f57dd0.agentv.pages.dev
Branch Preview URL: https://feat-sdk-deterministic-grade.agentv.pages.dev

View logs

@christso christso changed the title feat(sdk): add Vitest workspace grader adapter feat: add Vitest workspace verifier adapter Jun 22, 2026
@christso christso marked this pull request as ready for review June 22, 2026 09:26
@christso christso merged commit d896af6 into main Jun 22, 2026
10 checks passed
@christso christso deleted the feat/sdk-deterministic-grader-dx branch June 22, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant