harbor: supervise sandbox starts and clean up only our own sandboxes on shutdown

docs/source/environments/harbor.md

@@ -631,6 +631,10 @@ A backend counts as usable only if its class imports **and** Harbor's own prefli
 Credentials alone are not enough: a provider with valid keys but no SDK installed would otherwise
 report available and fail at rollout time.
 
+Set `OPENENV_HARBOR_MAX_SANDBOX_STARTS` to cap how many sandboxes may be starting at once when a
+backend throttles creation. It does not limit how many rollouts run (`MAX_CONCURRENT_ENVS` does).
+On SIGINT or SIGTERM the process stops the sandboxes it started before exiting.
+
 ### Rewards
 
 The verifier produces a dictionary. OpenEnv wants one number. The dictionary is forwarded unchanged

envs/harbor_env/README.md

@@ -643,6 +643,10 @@ A backend counts as usable only if its class imports **and** Harbor's own prefli
 Credentials alone are not enough: a provider with valid keys but no SDK installed would otherwise
 report available and fail at rollout time.
 
+Set `OPENENV_HARBOR_MAX_SANDBOX_STARTS` to cap how many sandboxes may be starting at once when a
+backend throttles creation. It does not limit how many rollouts run (`MAX_CONCURRENT_ENVS` does).
+On SIGINT or SIGTERM the process stops the sandboxes it started before exiting.
+
 ### Rewards
 
 The verifier produces a dictionary. OpenEnv wants one number. The dictionary is forwarded unchanged

rfcs/012-harbor-capture-providers.md

@@ -41,6 +41,16 @@ Upstream responses are buffered to preserve complete capture, then replayed as p
 
 A versioned external evidence report supplies stable, experimental and unstable classifications for the recorded provider/model/adapter combination. No report means unqualified; the UI requires an explicit experimental opt-in. Recorded evidence does not qualify a newly entered endpoint or pin installed agent versions. `harness_profile` selects an explicitly supported ACP or NeMo workflow on a local seam copy, without mutating global defaults. Unknown profiles fail.
 
+### Sandbox supervision
+
+Harbor stops a trial's sandbox in `Trial._finalize`. `openenv.harbor.supervisor` covers what that cannot: a signal that ends the process, provider throttling of creation, and a cancellation that abandons a start mid-flight. `run_rollout` adopts each trial into one process-wide supervisor, which wraps that trial's `agent_environment.start/stop` rather than replacing Harbor's lifecycle.
+
+`OPENENV_HARBOR_MAX_SANDBOX_STARTS` caps how many sandboxes may be starting at once, across every event loop in the process. Unset or `0` is uncapped. It is separate from `MAX_CONCURRENT_ENVS`: a rollout holds a creation slot only while its sandbox starts, so rollout concurrency can stay above what a provider accepts for creation.
+
+The supervisor owns exactly the sandboxes whose start it ran and has not yet stopped. Cleanup stops only those, never by listing or labels, so other processes' sandboxes are untouched. It is safe to call repeatedly and concurrently with Harbor's own stop: each sandbox is stopped once, on the loop that started it. A cancelled start is not abandoned; it completes, and stop waits for it (up to 120 s) before deleting. That wait also applies after Harbor's own start timeout, which delays a hung start's teardown rather than racing it.
+
+On the first sandbox start from the main thread, handlers for SIGINT and SIGTERM are installed over whatever is there, including a server's. A signal stops owned sandboxes (bounded at 180 s: the 120 s wait for an in-flight start, then 60 s to stop), restores the previous handler and re-raises the signal to it, so default termination, `KeyboardInterrupt` and server shutdown behave as before. A second signal skips the wait. A process that starts sandboxes only from worker threads calls `install_signal_handlers()` itself; the Harbor server does so at app startup, since its UI runs each rollout in a worker thread, and a signal then stops those sandboxes on their own loops. Separate verifier environments and sandboxes on an already-closed loop are out of scope.
+
 ### Text-level SFT export
 
 The training contract above is token-level and refuses evaluation rollouts. A second export is text-level: `openenv.harbor.sft.export_sft` (and `openenv harbor export-sft`) turns a rollout into one conversational row, the agent's last request (`request_messages`, `request_tools`) plus its reply. Supervised fine-tuning needs that conversation, not engine tokens, so this export accepts `eval` rollouts against hosted providers as well as `train` ones, and never reads token fields. It does not weaken the token contract: an `eval` rollout still cannot produce a `TrainingTrace`.

src/openenv/harbor/rollout.py

@@ -43,6 +43,7 @@
     HarborTurn,
     turns_from_document,
 )
+from .supervisor import get_supervisor
 
 # Harbor's own retry is deliberately off. A Harbor-level retry re-runs the agent against the SAME
 # capture session, so two attempts merge into one graph and the trace cross-check compares a
@@ -540,6 +541,7 @@ async def run_rollout(
             if not proc_env:
                 config = build_trial_config(**config_kwargs)
                 trial = await Trial.create(config)
+                get_supervisor().adopt(trial)
                 trial_result = await _run_trial(trial, on_trial_created)
             else:
                 # The override has to span `run()`, not just construction. Harbor's goose wrapper reads
@@ -572,6 +574,7 @@ async def run_rollout(
                     with ctx_env_overlay(proc_env):
                         config = build_trial_config(**config_kwargs)
                         trial = await Trial.create(config)
+                        get_supervisor().adopt(trial)
                         trial_result = await _run_trial(trial, on_trial_created)
                 else:
                     # Fallback: mutate the real environment under a lock, one rollout at a time.
@@ -580,6 +583,7 @@ async def run_rollout(
                         with process_env(seam.name, proc_env):
                             config = build_trial_config(**config_kwargs)
                             trial = await Trial.create(config)
+                            get_supervisor().adopt(trial)
                             trial_result = await _run_trial(trial, on_trial_created)
                     finally:
                         _PROC_ENV_LOCK.release()

src/openenv/harbor/serving.py

@@ -25,6 +25,8 @@
 
 from openenv.core.harness.capture import CaptureServer
 
+from .supervisor import get_supervisor
+
 # Where the capture app is mounted when the env server hosts it directly.
 CAPTURE_MOUNT = "/capture"
 
@@ -338,6 +340,10 @@ def gradio_builder(
     # their browser start rollouts on this server's endpoint, or reach a server on their own machine
     # that the page itself cannot. Nothing a real visitor does is cross-site.
     app.add_middleware(SameOrigin)
+    # The UI runs each rollout under `asyncio.run` in a worker thread, where a sandbox start cannot
+    # install signal handlers. Startup runs on the main thread inside uvicorn's own signal capture,
+    # so the supervisor wraps uvicorn's handlers and a signal stops those sandboxes on their loops.
+    app.router.on_startup.append(lambda: get_supervisor().install_signal_handlers())
     return app