The lab where self-improving agents run experiments, keep evidence and earn their upgrades.
Quickstart · Agents · Benchmarks · Documentation · Roadmap · CI
English · 简体中文
Orbit is the lab for agents that work and improve on their own. Agents now run long tasks and rewrite their own prompts, skills and memory, while people step back to review. Two questions follow: was this piece of work done right, and did that change really make the agent better? An agent's own report answers neither. Orbit sits between the agent that decides and the sandbox that isolates, and makes both answers checkable.
| Orbit provides | What it means | Status |
|---|---|---|
| Hands | Isolated Docker runs under GPU / CPU / memory leases | Implemented |
| Evidence | Code snapshot, logs, metrics, outputs and agent sessions for every run | Implemented |
| Acceptance | Independent verification, validity kept apart from execution, paired comparison with confidence intervals | Implemented |
| Evolution gate | Each change to an agent is a version with lineage, accepted only if it beats its parent on unseen tasks and does not regress in other environments | Versions and lineage implemented; the gate is in design |
A change tested only where it was made has not been tested. That is why Orbit spans code, science, GUI, tool-use and embodied benchmarks, and more than one agent. Today it runs agents such as Codex and Claude Code; most benchmark families are validated on small calibration samples only. The agent plans, analyzes and iterates; Orbit provides execution and evidence. See the roadmap.
Run → Keep evidence → Verify independently → Accept or reject the change → Run again
| Capability | What it gives you |
|---|---|
| 🧪 Reproducible execution | Docker isolation, code snapshots and resource limits for experiments you can inspect. Runs & sweeps |
| ⚙️ Shared resource scheduling | GPU, CPU and memory leases that account for external use, with clear queue reasons. Multiple machines |
| 🧭 Evidence for review | Agent sessions alongside code diffs, logs, metrics, artifacts and notes, so a result can be checked instead of taken on trust. Web UI |
| 🤖 Your existing agents | Shared skills for Codex, Claude Code, dsh and pi; in-container agents for Codex / Claude Code. Agent integration |
| 📊 Independent acceptance | Task × method × seed jobs verified apart from the agent, with execution and score validity tracked separately; Harbor, ASI and external adapters. Benchmarks |
| 🧬 Versioned changes | Evaluation configurations are versioned by content and record their parent, a note and the trials that informed the change; versions are compared on paired tasks. Benchmarks |
| 📱 Progress wherever you are | Local Web review, Feishu cards and read-only queries, plus DingTalk, WeCom, Slack and webhooks. Notifications |
Prerequisites: Linux, Node.js 24.18+ (24.x), Git and Docker usable by your user without sudo.
GPU experiments also need NVIDIA drivers and the NVIDIA Container Toolkit or CDI. CPU experiments need no GPU.
Install the npm package:
npm install -g @zjunlp/orbit
orbit doctorThe package includes the CLI, daemon, Web UI and shared skills. No source checkout or build is needed.
To try Orbit without a global installation, use npx -y @zjunlp/orbit@latest doctor.
Update Orbit
npm install -g @zjunlp/orbit@latest
orbit daemon --stop
orbit doctorStopping the old daemon lets the next command start the updated version; running containers are reattached.
Build and install from source
Requires pnpm 10. Use this to install changes that are not yet on npm:
git clone https://github.com/zjunlp/Orbit.git
cd Orbit
corepack enable
pnpm install --frozen-lockfile
pnpm pack:release
orbit_version=$(node -p 'require("./packages/orbit/package.json").version')
npm install -g "./dist/zjunlp-orbit-${orbit_version}.tgz"
orbit doctorpack:release builds the CLI, daemon, Web UI and shared skills. To run from source or contribute, see Development.
cd ~/my-project # your Git repository
orbit resources # available resources and the queue
orbit run --image python:3.12-slim --cpus 1 --mem 1 -n hello -w -- python -c 'print("hello from orbit")'
orbit ps -a # inspect run records
orbit web --open # review in your browserOrbit options go before --; the experiment command goes after it. By default, it runs in /workspace
on a snapshot of your Git repository, including uncommitted, non-ignored files. The first command
that needs the daemon starts it automatically on 127.0.0.1:7717, or a free port if occupied.
State lives in ~/.orbit; large artifacts can use a separate data directory.
Once the skills below are installed, ask your agent:
Use Orbit to run this experiment. Check resources, try several seeds, compare metrics and failures, and attach conclusions to each run.
Install your agent CLI, then choose its host plugin:
# Choose your agent
orbit plugin install claude
orbit plugin install codex
orbit plugin install dsh --profile headless
orbit plugin install piRestart the agent to load orbit for experiment execution and review, and orbit-bench for benchmark adapters, resources and validation.
| Agent | Host skills | In-container execution & session capture |
|---|---|---|
| Codex / Claude Code | ✅ | ✅ |
| dsh / pi | ✅ | Not yet implemented |
See the plugin guide for local and npm installation, updates and removal. dsh requires pnpm and an explicit profile with its skills service. Agents need network access to the local daemon; MCP is not implemented yet.
In-container agents and credentials
A run can execute Codex or Claude Code inside its container (for agent experiments and benchmarks). Orbit mounts the CLI installed on the host and injects credentials from a profile:
orbit auth add codex --harness codex --kind login # `codex login --device-auth` in a private home
orbit auth add gpt --harness codex --kind api-key --key-env OPENAI_API_KEY
orbit auth add claude --harness claude-code --kind login --key-stdin # paste a `claude setup-token` token
orbit auth add relay --harness claude-code --kind relay --base-url https://relay.example --key-env RELAY_KEY
orbit auth add claude --harness claude-code --kind relay --from-env # take key, URL, models from env / Claude settings
orbit auth list
orbit auth test claude # a real small run: the agent must answer
orbit run --auth claude --image python:3.12-slim -- claude -p --dangerously-skip-permissions "…"Secrets are read from stdin, an environment variable or a file, never from arguments. Profiles are
stored in ~/.orbit/auth.json (mode 0600) and secret values are redacted from run logs. Use
--host-network on a profile when the agent must reach a proxy on 127.0.0.1.
orbit run --image pytorch/pytorch:latest --gpus 1 --mem 32 -n baseline -- python train.py
orbit sweep --image python:3.12-slim --gpus 1 -g lr -p lr=1e-4,3e-4 -p seed=1,2,3 \
-- python train.py --lr {lr} --seed {seed}
orbit wait -g lr
orbit show <id>
orbit logs <id> -f
orbit note <id> "Compare this run with the baseline before choosing the next configuration."{key} expands to each parameter value; the sweep above submits six runs, queued as resources allow. Most commands accept --json for agents and scripts.
Full run reference: paths, resources, sweeps and record management
| Path / variable | Meaning |
|---|---|
/workspace |
Writable copy of your code (the original repository is untouched) |
$ORBIT_OUTPUT (/orbit/output) |
Put results here; they are kept and shown in the Web UI |
$ORBIT_METRICS (/orbit/metrics.jsonl) |
One JSON object per line, e.g. {"step": 10, "loss": 0.3}; charted in the Web UI |
$ORBIT_RUN_ID |
The run id |
Leased GPUs are the only GPUs visible in the container. Runs use your host uid:gid by default so
output files belong to you (--root to run as root).
orbit run --image pytorch/pytorch:latest --gpus 1 --mem 32 -n baseline -- python train.py
orbit run --dockerfile docker/Dockerfile --gpus 2 --gpu-mem 40 -- python train.py # build an image
orbit run --gpus 1 --timeout 120 -w --image ... -- python eval.py # wait; exit code = run resultCommon options:
| Option | Meaning |
|---|---|
--image / --dockerfile |
Image to use, or a Dockerfile to build (context: its directory) |
--gpus, --gpu-mem <GB>, --gpu-ids |
Number of GPUs, minimum memory per GPU, pin specific GPUs |
--cpus (default 4), --mem <GB> (default 16) |
CPU and memory limits |
-n, -g, -t KEY=VALUE |
Name, group and tags |
-m src:dst[:rw], -d name[:dst[:rw]] |
Bind-mount a host path or a registered dataset (read-only by default) |
-e KEY=VALUE |
Environment variable (stored with the run: no secrets) |
--pass-env VAR |
Pass a variable from the daemon's environment (for secrets; not stored) |
--project <dir>, --no-project, --no-snapshot |
Choose the repository, run without one, or mount it read-only instead of copying it |
--no-network, --host-network |
Disable networking, or share the host network |
--timeout <min> |
Stop the run after this long |
--node <name> |
Run on a specific node (multi-node) |
-w, --json |
Wait for the result; print JSON |
orbit sweep --image python:3.12-slim --gpus 1 -g lr -p lr=1e-4,3e-4 -p seed=1,2,3 \
-- python train.py --lr {lr} --seed {seed}
orbit wait -g lr # block until the group is done; exit 1 if any run failed{key} in the command is replaced by each value; 2 × 3 = 6 runs are submitted and queued as
resources allow.
orbit ps # active runs; queued runs show why they wait
orbit ps -a -g lr # all runs of a group
orbit show <id> # details and latest metrics
orbit logs <id> -f # follow the log (--tail BYTES to start near the end)
orbit wait <id>... # wait for specific runs
orbit cancel <id>...
orbit rm <id>... | -g <group> # permanently delete finished runs (asks you to type the id or group)
orbit note <id> "lr 3e-4 diverges after step 2k" # attach a conclusion (- reads stdin)
orbit web <id> --open # open the run in the Web UIMost commands accept --json for scripts and agents.
orbit web prints a URL that logs your browser in with the daemon's token (stored as a cookie).
Pages: runs (filter, group, multi-select), run detail (overview, logs with search, metric charts,
artifacts preview, code diff, agent session), compare (metrics of several runs), resources (GPUs,
leases, queue, worker nodes), bench jobs and comparisons, and settings (doctor, auth and notification
profiles). Keyboard shortcuts are listed with ?.
The daemon listens on 127.0.0.1 only. To view it from another machine, use an SSH tunnel:
ssh -L 7717:127.0.0.1:7717 gpu-server, then open the printed URL locally (use the port in the URL that
orbit web prints if your daemon is not on 7717).
Keep personal experiment notes, scores and artifacts in user-owned storage. The source repository’s docs/ is for reusable product and development documentation.
Compare methods under the same resource rules, keeping execution, evaluation validity and cleanup separate. Only valid scores enter aggregates.
| Integration | Use it for | Guide |
|---|---|---|
| Harbor format | TB-Science, Terminal-Bench and other Harbor datasets; bench tasks lists supported tasks |
Adapter guide |
| ASI native wrapper | ASI’s own runner, with known compatibility issues checked before launch | Compatibility |
| LongDS native CLI | DataMind's sequential Codex / Claude Code sessions and original judge, via an external adapter | LongDS setup |
| Native embodied policies | Evaluate registered models in native simulators with frozen samples and independent evidence verification | Embodied evaluation quickstart |
| External adapters | Other benchmarks via a JSON protocol, without rebuilding Orbit | ResearchClawBench example |
orbit bench add tbs ../terminal-bench-science --adapter harbor
orbit bench tasks tbs
orbit bench run tbs -t symbolic-regression -m oracle -m empty -n smoke
orbit bench show <job>
orbit bench compare <job>...Full benchmark reference: agent methods, task resources and compatibility
orbit bench add tbs ../terminal-bench-science --adapter harbor
# datasets published on Harbor Hub: download them with Harbor's CLI, then register the directory
harbor download terminal-bench/terminal-bench@latest -o ~/datasets
orbit bench add tb ~/datasets/terminal-bench --adapter harbor
orbit bench add asi ../ASI-Bench --adapter asi -c python=/path/to/venv/bin/python --pass-env JUDGE_KEY
orbit bench add rcb ../ResearchClawBench --adapter ./examples/bench-adapters/researchclawbench/adapter.py \
--pass-env OPENAI_API_KEY --pass-env OPENAI_BASE_URL
orbit bench add longds ../DataMind/longds --adapter ./examples/bench-adapters/longds/adapter.py \
-c image=orbit-longds -c python=/path/to/longds-venv/bin/python \
--pass-env JUDGE_API_KEY --pass-env JUDGE_BASE_URL --pass-env JUDGE_MODEL
orbit bench tasks tbs # tasks, and why unsupported ones cannot run
orbit bench run tbs -t symbolic-regression -m oracle -m empty \
-m gpt=codex,auth=gpt,model=gpt-5,effort=high --seeds 3 -n first-try
orbit bench jobs
orbit bench show <job>
orbit bench compare <job>... # per-task comparison; paired means use tasks valid for every method
orbit bench usage <job>... # measured peak memory / CPU / GPU per task, suggested resources
orbit bench cancel <job>
orbit bench rm-job <job> # delete a finished job with its trials and runs (type the job id)Evaluation configurations use the existing --method syntax
name=agent[,auth=P,model=M,effort=E,bundle=DIR] with agent codex, claude-code,
oracle (reference solution) or empty (submits nothing; checks that the verifier gives 0). Every
trial records execution (how the solve run ended), evaluation (valid / invalid / unknown) and
cleanup separately; only valid scores count. The Web UI shows jobs, a task × configuration matrix and
cross-job comparison under Evaluations. Configurations groups previously evaluated settings
by name and content revision; changing the model or reasoning effort creates a configuration version.
An independent Method concept remains future design. Grader secrets given with --pass-env reach
only the verify run. Per-task solve resources (GPU, memory, CPU, timeout) come from an optional JSON file given with
orbit bench add --resources; every run records its measured peaks, and a solve run killed for memory
makes the trial invalid instead of a zero. Any other benchmark needs only an adapter executable that speaks a small JSON protocol (no
rebuild of Orbit): docs/guides/bench-adapter.md.
ASI Agent runs require a complete source checkout with --fail-on-agent-error support and the
sandbox timeout fix. Orbit checks the selected Codex/Claude adapter before launching an Agent and
rejects the known bug that treats a background timeout warning as execution failure. Update the
registered ASI checkout, including any frozen copy; upgrading Orbit alone cannot fix that code.
The native attempt status is shown separately from score validity. See
ASI compatibility.
Every machine runs Orbit. The machine your agent works on (the primary) places runs on workers it has added; local resources are used first.
# on the worker (e.g. Tailscale address 100.64.0.12); its firewall must let the primary reach 7718
orbit node expose --listen 100.64.0.12:7718
orbit node token # copy the printed token
# on the primary: asks for the token (paste it, input hidden, then Enter)
orbit node add http://100.64.0.12:7718
orbit node ls
orbit data add imagenet /data/imagenet # on every node that has the data
orbit run -d imagenet --gpus 1 --image pytorch/pytorch -- python train.pyCode snapshots are sent once per content hash; datasets are never copied; logs and metrics are mirrored to the primary; outputs stay on the worker and are read through the primary. The cluster protocol is plain HTTP with a token: use Tailscale, a private network or an SSH tunnel, never the public internet. Details: docs/guides/multi-node.md.
Use orbit feishu init to create a Feishu bot by QR code, receive result cards with metric curves,
and query /gpu, /runs or /show <id>. Feishu webhooks, DingTalk, WeCom, Slack and generic webhooks are also supported.
Set up notifications and the Feishu bot
orbit notify add team --kind feishu --url-env FEISHU_HOOK --on failed,group
orbit notify add team --kind feishu --url-stdin --on failed,group
orbit notify test team
orbit notify listKinds: feishu, dingtalk, wecom, slack, webhook. Events: succeeded, failed, group (one
summary when a group has finished). Failure messages include the reason and the last log lines.
Use --url-stdin for hidden terminal input or a pipe, or --url-env VAR. The legacy --url option
warns because webhook secrets appear in process arguments. See the notification guide.
orbit feishu init phone # prints a QR code: scan it with the Feishu app and confirmScanning creates a Feishu app (bot, message permissions and the message event pre-filled) paired with whoever scanned it. Then:
- When a run or group finishes, the bot sends a card: state, resources and measured peaks, metric curves (e.g. loss), and the log tail for failures.
- Message the bot at any time (mention it in groups):
/gpushows memory, utilization and holder of each GPU plus CPU, memory and the queue;/runsthe recent runs;/show <run id>one run. Commands are read-only.
The daemon opens an outbound long connection to Feishu: no public address is needed and the daemon still
listens on localhost only. If init reports that the long connection or the test card failed, it prints
the console page to fix (subscription mode "persistent connection", permissions, publish a version);
then run orbit feishu check phone to reconnect and test.
- Several bots can be bound (with different names), but each app only once, and never to the daemons of two machines (Feishu hands each event to only one connection). Pairing with several bots duplicates notifications.
- An existing app:
orbit notify add phone --kind feishu-app --app-id cli_… --secret-stdin. Others who message the bot get a pairing code and can use it after you runorbit notify pair phone <code>; unpaired people see no data. orbit notify listshows paired users and the connection;orbit notify rm phoneremoves the bot.
Directories, ports, nodes and data management
| Item | Default | Override |
|---|---|---|
| State directory (database, profiles, config) | ~/.orbit |
ORBIT_HOME |
| Data directory (run directories, bench staging, code bundles) | the state directory | orbit config data <dir>, or ORBIT_DATA |
Daemon port (on 127.0.0.1) |
7717, else a free port | ORBIT_PORT, or orbit daemon --port (then a busy port is an error) |
| Node name | host name | ORBIT_NODE |
| GPUs Orbit may use | all | ORBIT_GPUS=0,1 |
| Daemon endpoint and token | ~/.orbit/daemon.json (0600) |
The daemon is the only writer of Orbit's state; the CLI, the Web UI and agents all go through its
local HTTP API. orbit daemon --stop stops it; running containers continue and are picked up again
when it restarts. orbit daemon runs it in the foreground.
Run data can grow to hundreds of GB. On a machine with a small system disk, put it on a large disk
once, before the first runs: orbit config data /data/$USER/orbit && orbit daemon --stop.
orbit doctor warns when the data directory or Docker's data root is on the system disk or low on
space. Folders organize runs and bench jobs without moving files: orbit run --folder rcb/noskills,
orbit mv <id...> rcb/archive, orbit folders, and orbit export rcb /path/to/copy for a real copy.
| Direction | Current progress |
|---|---|
| Digital experiments | Docker execution, resource scheduling, sweeps, trajectory records and local Web review are implemented; single-machine workflows are validated. |
| Benchmarks & multiple machines | Benchmarks have integration-specific validation records. Multi-node validation uses two daemons on one host. |
| Embodied environments | Native simulation, independent policies, evidence verification and Trial review are implemented. Physical device backends are a future direction. |
| Evolution gate and unified trajectories | Configuration versions, lineage (--parent, --informed-by) and paired statistics are implemented. The gate that accepts or rejects a change, one trajectory format across agents and benchmarks, and cross-benchmark / cross-agent transfer studies are in design and not yet validated. |
Orbit is in early development (0.1.1). See project status for validation limits and near-term work, and design for the architecture.
| I want to… | Start here |
|---|---|
| Connect agents and manage plugins | Plugin guide |
| Add a benchmark or size task resources | Benchmark adapter guide |
| Run and review an embodied evaluation | Quickstart · Benchmark coverage and verified policies |
| Add an embodied policy or native benchmark | Onboarding templates |
| Connect several machines | Multi-node guide |
| Understand architecture and support | Design · Project status · Docs map |
| Build and publish a package | Release guide |
| Present Orbit | Offline slides (Chinese) |
| Symptom | What to check |
|---|---|
orbit doctor reports docker errors |
Docker is running and your user can run docker ps without sudo |
| GPU runs fail to start | nvidia-smi works; the NVIDIA Container Toolkit or a CDI spec is installed |
A run stays queued |
orbit ps shows the reason (GPUs busy, CPUs busy outside Orbit, memory, pinned node offline, …) |
| An agent cannot reach Orbit | orbit is on its PATH; Codex sandbox allows network access |
| Snapshot too large | Add data and weights to .gitignore and mount them with -m or orbit data |
Runs fail writing $ORBIT_OUTPUT / $ORBIT_METRICS with permission denied |
Docker uses userns-remap; see the workaround |
Requires pnpm 10 (corepack enable). See the release guide for publishing.
git clone https://github.com/zjunlp/Orbit.git
cd Orbit
pnpm install
pnpm build
pnpm orbit doctor
pnpm --filter @zjunlp/orbit-web dev # Vite dev server; proxies /api to the daemon
pnpm orbit <args> # run the CLI from source
pnpm verify # lint + typecheck + module boundaries + tests + buildpackages/orbit/ the `@zjunlp/orbit` package (CLI `orbit`, daemon, modules)
packages/web/ the Web UI (React + Vite, served by the daemon)
assets/ shared project logo and icon
plugin/ the Orbit skill for Codex / Claude Code
scripts/ repository checks (module boundaries)
docs/ design.md, status.md, tasks/, guides/, ppt/
Contributors and coding agents: read AGENTS.md first.
Contributions and reproducible bug reports are welcome through GitHub Issues.
Orbit's harbor adapter reads the task format defined by Harbor
(Apache-2.0) from the Terminal-Bench team; it is an independent implementation, not an official Harbor
runner and not endorsed by its authors. Scores from Orbit are comparable to Harbor's only where they have
been checked against it; see docs/tasks/ for what was checked.
Orbit's original source code is licensed under MIT © 2026 ZJUNLP. Bundled dependencies retain their own licenses; see Third-party notices. The presentation follows AGPL-3.0, with separate licenses for its third-party assets.