feat(project): add project add evaluator code-based

[jariy17]

Command structure

agentcore project add evaluator          add a custom evaluator to the current project
├── llm-as-a-judge                       existing — LLM prompted to score a session
└── code-based                           NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name>   NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]

add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in

Options:
  --name <name>                        the name of the evaluator
  --level <level>                      what to score: SESSION, TRACE, or TOOL_CALL
  --metric <metric>                    3P metric to scaffold as <library.Metric>,
                                       e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
  --model <model>                      judge model for the 3P metric,
                                       e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
  --lambda-arn <lambda-arn>            ARN of an existing Lambda that scores a session
  --timeout-seconds <timeout-seconds>  Lambda timeout in seconds (1-300)
  --description <description>          a description of what this evaluator measures
  --kms-key-arn <kms-key-arn>          customer-managed KMS key ARN to encrypt the evaluator
  --tags <tags>                        tags to apply (JSON object of key/value strings)
  -h, --help                           display help for command

Commits

1. c4430c03 feat — the command + 3 scaffold templates + remove evaluator
2. baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
3. e7bc3675 fix — validate --metric class + require a Bedrock --model
4. 9bd79980 fix — echo the inferred mode + caveats at add time
5. 32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
6. 853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

• bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
• Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

• Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
• Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.24%. Comparing base (d304147) to head (ef9f5ca).

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2144      +/-   ##
============================================
+ Coverage     97.22%   97.24%   +0.01%     
============================================
  Files           507      509       +2     
  Lines         33809    34009     +200     
============================================
+ Hits          32872    33073     +201     
+ Misses          937      936       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

Can we use make this helper function and use zod here.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[jariy17]

I didn't want add an abstraction to generate common assets like this one due time constraints. We can look for this in the future.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)