Python: [Feature]: EvalPort import/export support (to_openeval() / from_openeval())

[adhabnr-ux]

Description

Overview

We maintain EvalPort (v1.0.0), an open standard (Apache 2.0) for portable LLM evaluation datasets — test cases, graders, eval suites, and result sets in a single JSON format, with converters and SDKs (TypeScript + Python) so evals aren't locked to one framework or vendor.

We originally proposed this for AutoGen (microsoft/autogen#8005), and a contributor built a draft adapter there (microsoft/autogen#8009). Since AutoGen has since moved to maintenance mode (bug fixes / security / docs only, no new features), we published that work as a standalone package instead: https://github.com/adhabnr-ux/evalport/tree/main/adapters/autogen-openeval-adapter. Since Agent Framework is AutoGen's actively developed successor, we'd like to propose native EvalPort support here instead of another bolt-on package.

What We're Proposing

Add to_openeval() / from_openeval() conversion functions (roughly 50-100 lines using our SDK) so Agent Framework eval results can be exported to EvalPort's JSON format, and EvalPort suites can be imported as Agent Framework eval tasks. That gets round-tripping with DeepEval, Promptfoo, Inspect AI, and any other EvalPort-compatible tool for free.

Happy to align on the exact module location / package boundaries with the team, same as we did on the AutoGen PR.

Resources

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Python SDK: pip install evalport-sdk (imports as openeval for backward compatibility)
npm SDK: npm install evalport-sdk
Reference adapter with the same shape (AutoGen): https://github.com/adhabnr-ux/evalport/tree/main/adapters/autogen-openeval-adapter

We'd love to collaborate — happy to open a draft PR ourselves, or review one if someone on the team wants to take it on.

Code Sample

from openeval.types import OPENEVAL_VERSION

def to_openeval(agent_framework_eval_result) -> dict:
    """Export Agent Framework eval results to an EvalPort suite."""
    test_cases = []
    for result in agent_framework_eval_result.results:
        test_cases.append({
            "id": result.task_id,
            "input": result.task_description,
            "expected_output": result.expected_output,
            "graders": ["gr_output_match"],
        })
    return {
        "version": OPENEVAL_VERSION,
        "id": f"agent_framework_eval_{agent_framework_eval_result.run_id}",
        "test_cases": test_cases,
        "graders": [{"id": "gr_output_match", "type": "exact_match", "params": {"ignore_case": True}}],
    }

def from_openeval(suite: dict) -> list:
    """Import an EvalPort suite into Agent Framework eval tasks."""
    return [
        {"task_id": tc["id"], "description": tc["input"], "expected_output": tc.get("expected_output", "")}
        for tc in suite.get("test_cases", [])
    ]

Language/SDK

Both