Repository navigation
feat(openenv): serve every bundle task as an OpenEnv env - #230
earakely-scale wants to merge 14 commits into
Conversation
build_app(bundle) returns the OpenEnv server for a bundle's tasks: each session plays one episode through the Episode API, with the env's MCP tools plus an end_turn tool, and the task's verifier score as the reward. Bundles are prepared as agent-env run prepares them, without the agent a client plays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onment's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e's loop The playground calls the sync step in a worker thread; the episode's tool sessions belong to the loop its reset ran on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e app's lifespan OpenEnv's HTTP MCP sessions and web playground can call one env concurrently; a step during a reset or end_turn dereferenced the turn's unopened tools, and a close racing a turn's opening left its tool sessions open. Reset, step and close now take turns. Routing is on from each startup to its shutdown, so a failed create_app no longer leaves it on and a second run of the app has it. Reset defaults to the first split's first task as OpenEnv's importers do, with all listed first; an eval named all is refused; a Path is always a folder; a dotted bundle name is served dashed, since OpenEnv's web interface reads the env name as a module path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The extra installed fastapi and uvicorn, which core now installs, so agent-env up no longer checks for them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e limit allows Each entry's lock holds a file open, so agent-env run crashed with EMFILE on a bundle of over about 1000 entries on Linux, or 250 on macOS, such as a generated task pack. The soft limit is raised toward the hard one to fit them, as Go's runtime does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
…set skips an empty split A call cancelled while end_turn or reset waited for the next turn left a live episode with no open tools, so the next step dereferenced None; it now ends the episode as an error observation. A bare reset plays the first task served even when the first split has none, and the validator test asserts that OpenEnv's validation passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alizes it jaraco.classes, which openenv's keyring needs, ships a wheel whose filename keeps the dot, so its index page was jaraco.classes and uv found no jaraco-classes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_bundle and EpisodeSpec.start built the same three-key user_overrides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lifecycle helpers say what they do Startup turns reward= into one function per task (a user's function, or the score at a step), so the episode applies it in one place. _end/_close/_ended/_ended_episode/_fail become _abandon/_abandon_when_free/_error_observation/_why_stopped/_end_with_error, and done lives only in the episode's State. A missing score now reads as the reward's own error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Every task in a bundle is now an OpenEnv env.
agent_env.openenv.build_app(bundle)returns the OpenEnv server for a bundle's tasks. Any OpenEnv client (OpenEnv's ownGenericEnvClientandMCPToolClient, verifiers, a TRL wrapper) resets an episode, acts through the env's MCP tools and gets the task's verifier score as its reward, with no per-task code.openenv>=0.8,<0.10becomes a hard dependency, the waymcpanda2a-sdkare. That range is OpenEnv's two-minor deprecation window. Onlyagent_env/openenv/imports it, so no other command pays the cost of loading it.Two fixes outside
openenv/came out of testing on PortSim's packs:agent-env runcrashes withEMFILEon a bundle of more than about 1,000 entries (Linux) or 250 (macOS), such as PortSim's generated 1,050-task pack.explorerextra is retired. OpenEnv's fastapi makes it install nothing core lacks.How a task maps onto OpenEnv
/splits,/tasks,/task)all, then a split per eval, or a mapping's keys. Each task listing carries{id, name, index, prompt, system_prompt, agent}, read without deploying.reset(split, index)EpisodeSpec) and returns once its first turn is open (deploy and setup done). The observation carries the prompt and system prompt. With neither argument, it plays the first task served, as OpenEnv's ORS and verifiers importers do.step(ListToolsAction / CallToolAction)MCPEnvironment's does.end_turn(response)(an added tool)done=Trueand every score inmetadata["scores"].SUPPORTS_CONCURRENT_SESSIONS, andmax_concurrent_envs(default 8) caps them.The choices follow OpenEnv's own envs:
jupyter_envandterminus_envcreate their sandbox in reset.close().end_turnin flight.done, no reward, andmetadata["status"] == "error", asterminus_envreports a sandbox that can't start.ToolError, and the episode goes on.resetorend_turnwaited for the turn to open ends the episode at the next step, in the same error shape.reset,step,state,closeorend_turnare refused, asMCPEnvironmentrefuses them.reward=<step id>or a function of the scores, checked over every task at startup.agent-env rundoes before its first task, except that the agent the client plays is never deployed, so it needn't exist.@localids resolve while it serves.Changes outside
agent_env/openenv/agent_env.bundle.prepare_bundle(..., played=False):run_bundle's work before its first task, moved out unchanged, soagent-env runbehaves exactly as before.played=Truedrops the references of each task's only prompted agent'sdeploy_agentsteps, keeping its envs, and preflight skips those deploys.locate_bundlemoves fromcli/run.pyintobundle/installed.py.EpisodeSpec.start(sandbox=)setsuser_overridesthe wayrun_bundledoes, through a sharedtask_step.context.sandbox_overrides.EpisodeSpec.promptsgives each outside turn's prompt and system prompt before anything runs.Episode.finishedis added.cancelledmark is now written in the task's run scope, asrun_bundle's_mark_cancelleddoes. Before, an@localtask's abandoned episode stayedrunningin the local store, because the mark went to the configured store.holding_locksraises the soft open-file limit to fit its locks, as Go's runtime does.agent-env runwithOSError: [Errno 24] Too many open files.agent-env runon PortSim's generateddock-v1-train(1,050 tasks, Linux default limit 1024) anddock-v1-train-standard(266 tasks at macOS's default of 256).explorerextra is gone.agent-env updrops its check for them, and the README and AGENTS.md drop the extra.pip install 'agentenv-framework[explorer]'still works, with pip's warning that the extra doesn't exist.jaraco.classes(needed through openenv's keyring) ships a wheel whose filename keeps the dot, so uv found nojaraco-classes.THIRD_PARTY_NOTICES.md:hf-xetandhuggingface_hub, the only versions that changed;PyJWTandtensorlake, which already disagreed with main'suv.lock.Notes for review
agent-env run's own error. Only the tasks the Episode API refuses are left out (and logged), since planning attributes problems to the bundle, not to individual tasks. So a task the Episode API would refuse (say, one prompting two agents) must still passagent-env run's checks: its agents must exist.POST /resetdeploys an episode and tears it down at once, as it does for every OpenEnv env that sets up in reset. Real episodes go over/wssessions, whichmax_concurrent_envscaps..is served with-(dock.v1at/dock-v1/...). OpenEnv's web interface reads the env name as a module path, so a dotted name failedcreate_appunderENABLE_WEB_INTERFACE, whichopenenv pushsets in a Space.allis refused, since it would hide the split that holds every task./mcpendpoint answers a plainEnvironment'stools/callwith the whole observation, not the tool's data. That's true for every plain-Environmentenv, including the ORS importer's. SoMCPToolClient.call_toolreturns the observation for these envs.Verification
Tests: unit tests in
tst/unit/openenv/(25), plus extensions to the Episode API, bundle, ledger and installed-bundle tests. Integration tests intst/integration/openenv/(6) drivebuild_appon local Docker through OpenEnv's ownGenericEnvClientandMCPToolClient; they passed twice on a Linux box.PortSim proof (agentenv-portsim 0.5.5, a Linux EC2 box, local Docker):
build_app({"train": dock-v1-train, "eval": dock-v1-eval})serves 1,050 + 50 tasks after 11 s cold, 5 s warm.GenericEnvClient, score the same as the same episode run through the Episode API in process: 1.0, 0.200181, 0.0.Chaos (
dock-v1-train-standard, 266 tasks, c6i.8xlarge). All 21 checks pass:cancelled, the lastcompleted. Never more than one deployment.transport_errorToolError.end_turnis then the error observation (done, no reward,status: error).cancelledand torn down; stopped in 0.8 s.max_concurrent_envs=3CAPACITY_REACHED.OpenEnv's own tools against the PortSim server:
openenv validate --url ... --level runtime: PASS, 6 of 6 criteria.ENABLE_WEB_INTERFACE=true): the page serves, and an eval task resets with its four tools;end_turnscores it.Regression (main
705a178and this branch on the same Linux box):make unit-testmake int-test-fastmake int-test-slowrun_docker_container_test.py::test_a_context_zip_from_a_non_s3_store_unpacks_in_a_real_local_sandbox.local_sandbox_test.py::test_teardown_stops_what_a_command_left_running_and_only_thatfailed once in this PR's CI. It failed the same way on another PR's CI today, and passed in both full runs on the box.Follow-ups (separate PRs)
explorerextra once this is released, and a page on serving tasks as OpenEnv envs is added.agent-env hf publish, with envs deployed to a remote sandbox.AutoEnv;🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new actionable issues remain.Summary
Adds
build_appto serve bundle tasks through OpenEnv, with MCP tools, outside-agent turns, verifier rewards, and session cleanup.Diagram
sequenceDiagram participant Client as OpenEnv client participant Env as AgentEnvEnvironment participant Episode as Episode API Client->>Env: reset(split, index) Env->>Episode: Start task and await first turn Episode-->>Client: Prompt and system prompt Client->>Env: List or call MCP tools Client->>Env: end_turn(response) Env->>Episode: End turn and await next turn Episode-->>Client: Next prompt or final scores and reward Client->>Env: Close session Env->>Episode: Abandon and await teardownReviews (4) · Last reviewed commit: "refactor(openenv): a served task carries..." · Reviewed by Greptile