feat: Third-party eval metrics adapter (DeepEval + Autoevals) with strands-evals mappers by stone-coding · Pull Request #568 · aws/bedrock-agentcore-sdk-python

stone-coding · 2026-07-06T17:07:35Z

Summary

Add DeepEvalAdapter and AutoevalsAdapter that integrate third-party evaluation metrics with
AgentCore's code-based evaluator framework.

Span mapping via strands-evals: Uses detect_otel_mapper() from strands-agents-evals for
auto-detection of Strands, OpenInference LangChain, and OpenTelemetry LangChain span formats.
Bridge function converts Session → SpanMapResult for adapter consumption.
custom_mapper parameter lets customers bypass built-in mappers for unsupported frameworks:
- DeepEval: custom_mapper: Callable[[EvaluatorInput], LLMTestCase]
- Autoevals: custom_mapper: Callable[[EvaluatorInput], Dict[str, Any]]
tools_called + expected_tools extraction from execute_tool spans — enables
ToolCorrectnessMetric and ArgumentCorrectnessMetric
assertions from reference_inputs mapped to LLMTestCase.context for HallucinationMetric
reference_inputs full pipeline: expectedResponse → expected_output,
expectedTrajectory → expected_tools, assertions → context
Never raises unhandled exceptions — all error paths return valid EvaluatorOutput with
structured errorCode/errorMessage
Pinned dependency: strands-agents-evals>=1.0.0,<2.0.0

Test plan

38 unit tests passing (pytest tests/.../third_party/ -v)
21 E2E tests passing across:
- DeepEval: 12 metrics (RAG, Agentic, Custom/GEval, Contextual, Non-LLM, Safety + OpenInference +
  OpenTelemetry)
- Autoevals: 6 metrics (LLM-Judge, LLM-Scorer, Deterministic + OpenInference + OpenTelemetry)
- Custom mapper: 3 tests (Google ADK, OpenAI Agents, Autoevals path)
E2E validated against live AgentCore service (evaluate() API → Lambda → metric → score)

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

…leTurnParams deprecation

…d EvaluatorInput support

… layer, simplify per TJ/Irene feedback

…ion, add validate_fields to DeepEvalAdapter

…xpected_response_text property

- Rename span_parsers → span_mappers, simplify to Strands-only - Rename field_mapper → customer_mapper across all adapters - customer_mapper now returns native types directly: - DeepEval: EvaluatorInput → LLMTestCase - Autoevals: EvaluatorInput → Dict[str, Any] (eval kwargs) - Refactor BaseAdapter: remove intermediate execute(), add _run() pattern - 83 unit tests passing

chianhw · 2026-07-08T18:56:48Z

+            customer_mapper=lambda ev: {
+                "input": ev.session_spans[0]["attributes"]["question"],
+                "output": ev.session_spans[0]["attributes"]["answer"],
+                "expected": "the expected answer",
+            },


Do we need to accept aws lambda?

Replaced lambda examples with named functions in docstrings

chianhw · 2026-07-08T20:16:27Z

+    def __init__(
+        self,
+        scorer: Any,
+        customer_mapper: Optional[Callable[[EvaluatorInput], Dict[str, Any]]] = None,


We have not updated this type Dict[str, Any]?

Regarding naming, "Custom mapper" would be a better choice for describing an arbitrary mapper provided by a customer?

1.The type Dict[str, Any] is correct — Autoevals has no typed
input class like DeepEval's LLMTestCase. Scorers take plain
kwargs (input, output, expected) and the dict gets unpacked
as metric.eval(**kwargs). Different scorers need different
keys, so a generic dict is the right contract.

chianhw · 2026-07-08T21:00:07Z

+        self,
+        scorer: Any,
+        customer_mapper: Optional[Callable[[EvaluatorInput], Dict[str, Any]]] = None,
+        threshold: float = 0.5,


label: Optional[str] = None label in EvaluatorOutput is optional.
We don't need to have a default value to generate a label when a metric doesn't have a label and the user doesn't provide any threshold. Can we set default as None?

Changed threshold default to None. When no threshold is set,
no Pass/Fail judgment is made. Note: EvaluatorOutput's
validator currently requires a label for success responses —
I'm returning the score as the label string when threshold is
None. Let me know if you'd rather relax the validator to
allow label=None.

chianhw · 2026-07-08T21:00:37Z

+                mapping when provided. Expected keys: input, output, expected (optional).
+            threshold: Score threshold for Pass/Fail determination. Defaults to 0.5.
+        """
+        self.scorer = scorer


this naming is not aligned with DeepEval adaptor. Looks like you haven't updated AutoevalsAdapter.
If you haven't completed end to end tests for AutoevalsAdapter, you should not include it in your PR.

Working on E2E test plan now. Considering how to strcuture parameterized Lambda handlers and how to obtains span from strands, openinference, opentelemetry, and customer mappers.

chianhw · 2026-07-08T21:00:43Z

+            customer_mapper=lambda ev: LLMTestCase(
+                input=ev.session_spans[0]["attributes"]["user_query"],
+                actual_output=ev.session_spans[0]["attributes"]["response"],
+            ),


using lambda? same as above

chianhw · 2026-07-08T21:00:49Z

+    def __init__(
+        self,
+        metric: BaseMetric,
+        customer_mapper: Optional[Callable[[EvaluatorInput], LLMTestCase]] = None,


naming? same as above

chianhw · 2026-07-08T21:00:52Z

+        self.metric = metric
+        self.customer_mapper = customer_mapper
+        if model is not None:
+            self.metric.model = model


Where do we use self.metric.model?

chianhw · 2026-07-08T21:16:32Z

+    Raises:
+        ValueError: If no mapper can extract data from the spans.
+    """
+    result = _strands_mapper.map(session_spans)


We can check the span scope name and then decide which mapper to use.

Applied to all three mappers (Strands, OpenInference LangChain, OpenTelemetry LangChain).

chianhw · 2026-07-13T23:38:29Z

+            attributes = span.get("attributes", {})
+            if attributes.get("gen_ai.operation.name") == "invoke_agent":
+                return span
+        return None


Does this mapper only take the first agent span? For strands, the OtelSpanMapper in OpenTelemetryMapper package use the last agent span. That is, the agent span with the latest end_time per trace is the one retained.

Fixed, now takes the last agent span to match server-side OtelSpanMapper behavior. Applied to all three mappers (Strands, OpenInference LangChain, OpenTelemetry LangChain).

chianhw · 2026-07-14T05:13:09Z

+                expected_output=result.expected_output,
+                context=result.context,
+                retrieval_context=result.retrieval_context,
+            )


why don't we have tools_called and expected_tools?
Do we want to support ToolCorrectnessMetric, ArgumentCorrectnessMetric in DeepEval?

Added both. tools_called is now extracted from execute_tool spans (name + input_parameters + output) and mapped to LLMTestCase.tools_called. expected_tools comes from reference_inputs[0].expectedTrajectory.toolNames → LLMTestCase.expected_tools. This enables ToolCorrectnessMetric and ArgumentCorrectnessMetric.

chianhw · 2026-07-14T06:07:38Z

+    actual_output: Optional[str] = None
+    retrieval_context: Optional[List[str]] = None
+    context: Optional[List[str]] = None
+    system_prompt: Optional[str] = None


Are we using system_prompt for any metric?

Currently no DeepEval or Autoevals metric consumes system_prompt, LLMTestCase doesn't have this field. I include it in SpanMapResult for two reasons: (1) alignment with the server-side AgentInvocationSpan which also extracts it, and (2) forward-compatibility, customers using custom_mapper can access it from the spans if a future metric needs it. Happy to remove if you feel it adds unnecessary surface area.

No, I agree with you. We should keep it.

chianhw · 2026-07-14T06:08:03Z

+            expected = getattr(ref, "expected_response_text", None)
+            if expected:
+                result.expected_output = expected
+        return result


We only take the data for expected_output here, but why don't we use the expected_trajectory field and assertions in reference_inputs?

updated. Now all three fields from reference_inputs are extracted:

expectedResponse.text → SpanMapResult.expected_output → LLMTestCase.expected_output

expectedTrajectory.toolNames → SpanMapResult.expected_tools → LLMTestCase.expected_tools (enables ToolCorrectnessMetric)

assertions[].text → SpanMapResult.assertions → LLMTestCase.context (enables HallucinationMetric to check agent response against ground-truth claims)
For the Autoevals adapter, assertions are joined as a fallback for the expected kwarg when no explicit expectedResponse is provided.

chianhw · 2026-07-14T06:08:29Z

+            input=input_text,
+            actual_output=actual_output,
+            retrieval_context=tool_outputs if tool_outputs else None,
+            context=tool_outputs if tool_outputs else None,


Why do we use tool_output as context?

DeepEval uses context and retrieval_context for different metrics: HallucinationMetric requires context (ground-truth facts to check against), while FaithfulnessMetric requires retrieval_context (retrieved documents to verify grounding). Tool outputs serve both purposes — they're what the agent retrieved AND the factual basis for its response. Setting both ensures both metrics work without requiring a custom_mapper. Note: when assertions from reference_inputs are present, they override context (explicit ground truth takes
precedence over tool outputs). This matches the server-side pattern — the AgentInvocationSpan exposes tool results, and the service's built-in metrics use them as both retrieval context and grounding truth.

…21/21 passing) - Add tools_called + expected_tools extraction from execute_tool spans - Add assertions extraction from reference_inputs - Add OpenInference + OpenTelemetry LangChain mapper with auto-detection registry - Fix: use last agent span (not first) matching server-side behavior - Fix: traceId-based sibling span lookup for grandchild execute_tool spans - Fix: OTel adot_v18 body extraction with "parts" format support - Add context from retrieval_context to Autoevals adapter - Add 21 E2E tests covering DeepEval (12), Autoevals (6), Custom mapper (3) - All E2E tests pass against live AgentCore service

- Replace custom Strands/OpenInference/OpenTelemetry mappers with strands-agents-evals library (detect_otel_mapper + CloudWatchSessionMapper + LangChainOtelSessionMapper + OpenInferenceSessionMapper) - Add _session_to_span_map_result bridge (Session → SpanMapResult) - Add _detect_mapper for amazon.opentelemetry.distro scope edge case - Pin strands-agents-evals>=1.0.0,<2.0.0 in pyproject.toml - Delete span_mappers/strands/, span_mappers/langgraph/, span_mappers/base.py - Clean common.py to only SpanMapResult + _try_parse_json - Update unit test fixtures to CloudWatch split format - Update OTel fixture to adot_v17 (opentelemetry.instrumentation.langchain scope) - 38 unit tests + 21 E2E tests passing

haomiao037 added 14 commits June 30, 2026 11:59

Add DeepEvalHandler integration with unit tests

77d16a9

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

Fix span extraction to use real AgentCore _eval_log_records structure

402ea78

Set context field from tool messages for HallucinationMetric support

e9ef47d

Use metric.success for label instead of manual threshold comparison

f97827e

Add model override and timeout enforcement to DeepEvalHandler

9d256ae

Add model override, timeout enforcement, use metric.success, fix Sing…

c142d50

…leTurnParams deprecation

Fix _get_required_params to handle GEval unmappable typing params

3ccc98c

Add .deepeval/ to gitignore

a884f91

Move model override to init to avoid per-call mutation

6ac198c

Refactor to BaseAdapter framework with DeepEval/Autoevals adapters an…

8d415e5

…d EvaluatorInput support

Major refactor: move to custom_code_based_evaluators, add span parser…

8627ab0

… layer, simplify per TJ/Irene feedback

Fix review items: add reference_inputs to model, tighten error detect…

9a9b6a7

…ion, add validate_fields to DeepEvalAdapter

Adapt to upstream ReferenceInput model: remove duplicate field, use e…

0499d4b

…xpected_response_text property

stone-coding requested a review from a team July 6, 2026 17:07

stone-coding requested a deployment to manual-approval July 6, 2026 17:07 — with GitHub Actions Waiting

chianhw reviewed Jul 8, 2026

View reviewed changes

chianhw reviewed Jul 13, 2026

View reviewed changes

chianhw reviewed Jul 14, 2026

View reviewed changes

haomiao037 added 3 commits July 15, 2026 13:03

chore: Remove e2e_tests from public repo (internal testing only)

2ca8a5a

stone-coding requested a deployment to manual-approval July 16, 2026 01:50 — with GitHub Actions Waiting

stone-coding changed the title ~~feat: Revision third-party eval metrics adapter (DeepEval + Autoevals)~~ feat: Third-party eval metrics adapter (DeepEval + Autoevals) with strands-evals mappers Jul 16, 2026

Uh oh!

Conversation

stone-coding commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stone-coding commented Jul 6, 2026 •

edited

Loading