feat(evals): add OpenEval dataset import and export helpers (fix #3549) - #3619
feat(evals): add OpenEval dataset import and export helpers (fix #3549)#3619SparshGarg999 wants to merge 3 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f4151d77f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from .openeval import ( | ||
| OpenEvalItem as OpenEvalItem, | ||
| from_openeval as from_openeval, | ||
| to_openeval as to_openeval, | ||
| ) |
There was a problem hiding this comment.
This replacement leaves openai.types.evals exporting only the new OpenEval helpers, so documented and in-repo imports such as from openai.types.evals import RunCreateResponse in tests/api_resources/evals/test_runs.py now fail for users of the evals runs API. Please add the new helpers without removing the generated eval response/param exports.
AGENTS.md reference: AGENTS.md:L5-L8
Useful? React with 👍 / 👎.
| from .openeval import ( | ||
| OpenEvalItem as OpenEvalItem, | ||
| from_openeval as from_openeval, | ||
| to_openeval as to_openeval, | ||
| ) |
There was a problem hiding this comment.
This replacement leaves openai.types.evals exporting only the new OpenEval helpers, so documented and in-repo imports such as from openai.types.evals import RunCreateResponse in tests/api_resources/evals/test_runs.py now fail for users of the evals runs API. Please add the new helpers without removing the generated eval response/param exports.
Useful? React with 👍 / 👎.
| Convert an OpenEval dataset item into OpenAI Chat Completion messages format. | ||
| """ | ||
| messages: List[Dict[str, Any]] = [] | ||
| for msg in item.get("input", []): |
There was a problem hiding this comment.
OpenEval/EvalPort JSONL rows commonly use scalar input values such as {'input': 'What is 2+2?'}, but this loop assumes every input entry is a message dict. In that valid case Python iterates the prompt characters and msg.get(...) raises AttributeError, so the import helper cannot read standard single-prompt datasets; please branch on str or turn arrays before treating entries as dicts.
Useful? React with 👍 / 👎.
| {"role": str(msg.get("role", "user")), "content": str(msg.get("content", ""))} | ||
| for msg in messages | ||
| ] | ||
| item: OpenEvalItem = {"input": input_messages} |
There was a problem hiding this comment.
When callers use to_openeval() to write a dataset, the returned object starts as only {'input': ...} and the function has no way to add graders, which are required on OpenEval/EvalPort TestCase rows. The resulting JSONL cannot be consumed or validated as OpenEval unless callers mutate every item afterward, so the helper should accept and emit grader references or export a suite with them.
Useful? React with 👍 / 👎.
| result: Dict[str, Any] = {"messages": messages} | ||
| if item.get("id"): | ||
| result["id"] = item["id"] | ||
| if item.get("expected_output"): |
There was a problem hiding this comment.
When a dataset case expects an empty string, for example asserting that the model should produce no text, this truthiness check drops expected_output entirely. Downstream graders then see no reference output rather than ''; match the export path and preserve values that are present but falsy.
Useful? React with 👍 / 👎.
| Export OpenAI Chat Completion messages and metadata into OpenEval dataset item format. | ||
| """ | ||
| input_messages: List[OpenEvalMessage] = [ | ||
| {"role": str(msg.get("role", "user")), "content": str(msg.get("content", ""))} |
There was a problem hiding this comment.
When exporting multimodal Chat Completions messages, content can be a list of content-part dictionaries, but this casts it through str(...) and writes a Python repr instead of preserving the structured JSON. Those exported eval rows cannot be round-tripped back into Chat Completions or consumed by tools expecting real content parts, so keep non-string content as structured data rather than stringifying it.
Useful? React with 👍 / 👎.
|
Hey @SparshGarg999 — great to see someone build against the real spec here, thank you for picking up #3549. One thing worth flagging before this lands: the
None of this needs a rewrite, just a schema-alignment pass on
Want me to sketch a corrected |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e806e37fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Export OpenAI Chat Completion messages and metadata into OpenEval dataset item format. | ||
| """ | ||
| input_messages: List[OpenEvalMessage] = [ | ||
| {"role": str(msg.get("role", "user")), "content": str(msg.get("content", ""))} |
There was a problem hiding this comment.
When the chat history contains tool/function-call messages, this export silently keeps only role and content. For example, a {"role": "tool", "tool_call_id": "call_1", ...} message round-trips through to_openeval()/from_openeval() without tool_call_id, but the SDK's ChatCompletionToolMessageParam requires that field, so exported agent/tool eval datasets produce messages that cannot be sent back to Chat Completions. Please preserve the remaining Chat Completion message fields or reject unsupported roles instead of stripping them.
Useful? React with 👍 / 👎.
| Export OpenAI Chat Completion messages and metadata into OpenEval dataset item format. | ||
| """ | ||
| input_messages: List[OpenEvalMessage] = [ | ||
| {"role": str(msg.get("role", "user")), "content": str(msg.get("content", ""))} |
There was a problem hiding this comment.
For any normal chat message list, to_openeval() writes input as an array of {role, content} objects. The OpenEval/EvalPort TestCase schema only accepts input as a single string or an array of strings, so even the new test_to_openeval_conversion fixture would fail validation before graders are considered. Export the prompt text in the standard shape, or use a documented extension, rather than Chat Completions message objects.
Useful? React with 👍 / 👎.
|
Hi @adhabnr-ux, Thank you so much for the detailed feedback and for catching these schema discrepancies! That's super helpful. I would really appreciate it if you could sketch a diff for In the meantime, I'll update Thanks again for your guidance! |
|
Here's a concrete sketch. I don't have write access to your branch, so this is a full replacement for # File generated for OpenEval dataset import/export support.
from __future__ import annotations
import uuid
from typing import Any, Dict, List, Optional, Union
from typing_extensions import Required, TypedDict
__all__ = ["OpenEvalItem", "OpenEvalGrader", "from_openeval", "to_openeval"]
class OpenEvalMessage(TypedDict, total=False):
role: str
content: str
class OpenEvalGrader(TypedDict, total=False):
"""Inline grader object -- see spec/schemas/grader.json. Most callers
will instead pass a bare grader-id string in OpenEvalItem['graders']."""
id: Required[str]
type: Required[str]
params: Dict[str, Any]
class OpenEvalItem(TypedDict, total=False):
# id and graders are REQUIRED by spec/schemas/testcase.json
# ("required": ["id", "input", "graders"]) -- not optional.
id: Required[str]
input: Required[Union[str, List[str]]]
graders: Required[List[Union[str, OpenEvalGrader]]]
expected_output: Optional[str]
metadata: Optional[Dict[str, Any]]
def from_openeval(item: OpenEvalItem) -> Dict[str, Any]:
"""
Convert a spec-valid OpenEval TestCase into OpenAI Chat Completion
messages format.
``item["input"]`` is a string or an array of strings per the EvalPort
TestCase schema, not an array of {role, content} objects -- that chat
shape is OpenAI's own native format, produced by to_openeval() below,
not something to_openeval()'s *caller* is expected to hand you.
If this item carries the original chat messages this adapter itself
exported (see to_openeval()), they're restored losslessly from
``metadata["openai"]["messages"]``. Otherwise -- e.g. a hand-authored
TestCase, or one from a different tool -- every input string is
reconstructed as a single "user" message, the same fallback every other
adapter in the EvalPort ecosystem uses for a grader/tool type it doesn't
recognize.
"""
metadata = item.get("metadata") or {}
openai_meta = metadata.get("openai") or {}
if "messages" in openai_meta:
messages: List[Dict[str, Any]] = list(openai_meta["messages"])
else:
raw_input = item.get("input", "")
input_strings = [raw_input] if isinstance(raw_input, str) else list(raw_input)
messages = [{"role": "user", "content": s} for s in input_strings]
result: Dict[str, Any] = {"messages": messages}
if item.get("id"):
result["id"] = item["id"]
if item.get("expected_output"):
result["expected_output"] = item["expected_output"]
if metadata:
result["metadata"] = metadata
return result
def to_openeval(
messages: List[Dict[str, Any]],
graders: List[Union[str, OpenEvalGrader]],
id: Optional[str] = None,
expected_output: Optional[str] = None,
metadata: Optional[Dict[str, Any]] = None,
) -> OpenEvalItem:
"""
Export OpenAI Chat Completion messages into a spec-valid OpenEval
TestCase (spec/schemas/testcase.json).
Args:
messages: OpenAI chat messages ({role, content} dicts).
graders: REQUIRED -- TestCase.graders must have >=1 entry per spec.
Pass grader-id strings (referencing graders already defined on
the enclosing Suite) or inline grader objects.
id: TestCase.id is required by spec; if omitted, a uuid4 is
generated so the output always validates.
expected_output, metadata: as before.
``input`` is built as one string per message ("{role}: {content}"),
satisfying the schema's string-or-array-of-strings requirement. The
*original* messages are additionally preserved verbatim under
metadata["openai"]["messages"] so from_openeval() can reconstruct the
exact role/content structure on import instead of collapsing everything
to "user" turns -- "openai.*" following the same reserved-namespace
convention every adapter in the EvalPort ecosystem uses ("openeval.*"
is the only prefix EvalPort itself reserves).
"""
input_strings = [
f"{msg.get('role', 'user')}: {msg.get('content', '')}" for msg in messages
]
merged_metadata: Dict[str, Any] = dict(metadata or {})
merged_metadata["openai"] = {
**merged_metadata.get("openai", {}),
"messages": [
{"role": str(m.get("role", "user")), "content": str(m.get("content", ""))}
for m in messages
],
}
item: OpenEvalItem = {
"id": id or str(uuid.uuid4()),
"input": input_strings if len(input_strings) != 1 else input_strings[0],
"graders": graders,
"metadata": merged_metadata,
}
if expected_output is not None:
item["expected_output"] = expected_output
return itemWhat changed and why, against
I did not add JSON Schema validation as a runtime dependency here ( |
|
One more piece to close the loop on, since Codex's review flagged it as P1 and my last sketch only touched Here's the fix — the original generated content with the OpenEval import appended, not swapped in: # File generated from our OpenAPI spec by Castiron. See CONTRIBUTING.md for details.
from __future__ import annotations
from .eval_api_error import EvalAPIError as EvalAPIError
from .run_list_params import RunListParams as RunListParams
from .run_create_params import RunCreateParams as RunCreateParams
from .run_list_response import RunListResponse as RunListResponse
from .run_cancel_response import RunCancelResponse as RunCancelResponse
from .run_create_response import RunCreateResponse as RunCreateResponse
from .run_delete_response import RunDeleteResponse as RunDeleteResponse
from .run_retrieve_response import RunRetrieveResponse as RunRetrieveResponse
from .create_eval_jsonl_run_data_source import CreateEvalJSONLRunDataSource as CreateEvalJSONLRunDataSource
from .create_eval_completions_run_data_source import (
CreateEvalCompletionsRunDataSource as CreateEvalCompletionsRunDataSource,
)
from .create_eval_jsonl_run_data_source_param import (
CreateEvalJSONLRunDataSourceParam as CreateEvalJSONLRunDataSourceParam,
)
from .create_eval_completions_run_data_source_param import (
CreateEvalCompletionsRunDataSourceParam as CreateEvalCompletionsRunDataSourceParam,
)
from .openeval import (
OpenEvalItem as OpenEvalItem,
OpenEvalGrader as OpenEvalGrader,
from_openeval as from_openeval,
to_openeval as to_openeval,
)( Also worth noting: the two other Codex threads on the current No urgency on my end — happy to review again whenever you've had a chance to work through these, or if it's easier for you to just take the diffs as-is and adjust from there, that works too. |
… generated evals exports
|
Thanks @adhabnr-ux! I have pushed updates addressing all the feedback:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2525c603f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| input_strings = [raw_input] if isinstance(raw_input, str) else list(raw_input) | ||
| messages = [{"role": "user", "content": s} for s in input_strings] | ||
|
|
||
| result: Dict[str, Any] = {"messages": messages} |
There was a problem hiding this comment.
When an OpenEval TestCase uses schema-valid fields such as context, retrieval_context, tools_called, or expected_tools (common for RAG and agent evals), from_openeval() initializes the result with only messages and later adds just id, expected_output, and metadata, so those inputs are silently dropped. The imported item can no longer provide retrieved docs or expected tool calls to graders or be re-exported faithfully; carry the remaining TestCase fields through, or explicitly encode context into the returned messages, instead of discarding them.
Useful? React with 👍 / 👎.
| "input": input_strings if len(input_strings) != 1 else input_strings[0], | ||
| "graders": graders, |
There was a problem hiding this comment.
If callers pass dynamically built empty messages or graders lists, this still returns an item with input: [] and/or graders: []; those violate the TestCase requirements that both the prompt array and grader list have at least one entry, despite the helper documenting that it exports a spec-valid TestCase. Please fail fast for empty lists before constructing the item so users do not write JSONL rows that downstream OpenEval validators reject.
Useful? React with 👍 / 👎.
| id: Required[str] | ||
| type: Required[str] | ||
| params: Dict[str, Any] |
There was a problem hiding this comment.
Inline graders can legally carry fields such as weight and description, but the exported OpenEvalGrader TypedDict only exposes id, type, and params. Users who type their inline graders with the SDK therefore get type errors for schema-valid grader definitions before passing them to to_openeval(), even though the helper accepts and preserves those fields at runtime; include the supported optional fields in this public type.
Useful? React with 👍 / 👎.
|
Verified this locally rather than just eyeballing the diff: pulled
So this is genuinely spec-valid now, not just structurally similar — nice work threading the |
|
Thanks for the quick turnaround, @SparshGarg999 — I pulled wrapping two This looks correct and spec-valid to me. Nice work closing out the schema gaps from the earlier feedback. |
2 participants
Fixes #3549
Summary
Add
from_openevalandto_openevaldataset conversion utilities inopenai.types.evalsto support importing and exporting portable LLM evaluation datasets matching the OpenEval standard format.from_openeval(item: OpenEvalItem)converts an OpenEval format dataset item into OpenAI Chat Completion message parameters.to_openeval(messages, ...)exports OpenAI Chat Completion messages and metadata into an OpenEval format item.How was this tested?
tests/test_openeval.pytesting serialization and deserialization of OpenEval items.