diff --git a/.gitignore b/.gitignore index dcbbd1a502..61f0768975 100644 --- a/.gitignore +++ b/.gitignore @@ -82,6 +82,7 @@ scripts/ralph/.codex-last-msg-* .claude/scheduled_tasks.lock # Local caches and stray outputs +.artifacts/ crates/execution/assets/v8-bridge.js crates/execution/assets/v8-bridge-zlib.js crates/execution/.agentos-pyodide-cache/ diff --git a/CLAUDE.md b/CLAUDE.md index d933af305e..3920cb26e2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,6 +4,10 @@ AgentOS owns the runtime, kernel, VFS, language execution, registry packages, ACP/session layer, AgentOS client APIs, docs, and publish machinery. The `secure-exec` repository is now a generated compatibility mirror only. +For future RivetKit work, start with the current documentation index at +https://rivet.dev/llms.txt and follow the linked page for the surface being +changed. + ## Boundaries - Keep AgentOS product versions pinned at `0.0.1` in committed files. Release diff --git a/docs-internal/load-testing-issues.md b/docs-internal/load-testing-issues.md new file mode 100644 index 0000000000..0df7061379 --- /dev/null +++ b/docs-internal/load-testing-issues.md @@ -0,0 +1,71 @@ +# AgentOS load-testing issues + +## Fixes applied (this branch) + +| Fix | Issue | Change | +| --- | --- | --- | +| **F-1 (critical)** | LT-011 cross-tenant DoS | `crates/native-sidecar/src/stdio.rs` `handle_protocol_frame`: a per-request dispatch error was propagated with `?` out of the serve loop → `run_async` → `main` → `exit(1)`, so ANY guest-triggered handler error (e.g. the EBADF) killed the shared sidecar and every co-located VM/actor. Now the error is caught and returned to the originating caller as a typed `ERR_AGENTOS_REQUEST_FAILED` rejection, and the sidecar keeps serving. This removes the crash *and* the cross-tenant blast radius even where the underlying errno is still wrong. | +| **F-4 (critical)** | LT-011 — the ACTUAL fatal path | F-1 hardened the `RequestFrame` dispatch, but verification showed the fs-crash vectors STILL killed the sidecar: the guest's `fs.writeSync` runs on the **process-event pump**, not the request dispatch. `crates/native-sidecar/src/stdio.rs` ran `sidecar.pump_process_events(...).await?` in the run loop, so any guest-triggered error there (the EBADF, raised by a `?` inside the sync-RPC handler such as `wake_ready_deferred_fd_reads/writes`) propagated to `main` → `exit(1)`, killing every co-located VM/actor. Now a pump failure is logged per-session and the loop `continue`s, serving all other sessions. F-1 and F-4 together close both guest-reachable fatal paths. **VERIFIED:** `fs-crash-workspace` went **FAIL(sidecar crash) → PASS** — the sidecar now survives the chunked write, sentinel stays healthy, and a fresh VM still runs. `filesystem-bytes` still fails but **non-fatally and for a different reason** (guest exited 137/SIGKILL, `maxFilesystemBytes did not fire`) — no `SidecarProcessExited`. So the crash and its cross-tenant blast radius are gone; what remains is a limit-ENFORCEMENT gap, not a DoS. Leak soak, boundary, and process-limit lanes all pass in the same run. | +| **F-5** | LT-011 second half / LT-017 — `maxFilesystemBytes` quota bypass | Verification after F-4 showed the crash was gone but `maxFilesystemBytes did not fire`. Root cause: writes through a **mapped host fd** (`/tmp`, `/workspace` are host-backed) go straight to the host file in `write_mapped_host_fd` / `write_all_mapped_host_fd` (`crates/native-sidecar/src/filesystem.rs`) and never reach the kernel VFS, so `check_filesystem_usage` never sees them — a guest could exceed its configured filesystem budget without bound and fill the host disk (this also drove the LT-017 tmpfs ENOSPC and the SIGKILL). Note `open_mapped_host_fd` already enforced `maxOpenFds`, so the fd cap was handled but the BYTE cap was missed. Fix: added `check_mapped_host_fd_write_budget(...)`, called from both mapped write paths, returning a typed `ENOSPC` naming `limits.resources.maxFilesystemBytes`. Only runs when a limit is configured, preserving the unlimited fast path. **Correction to an earlier hypothesis:** the KERNEL write path is NOT missing the quota — `fd_write_with_mode` calls `check_path_resize_limits_with_existing` on all three file branches (anonymous, `O_APPEND`, pwrite). Only the mapped-host path lacked a byte check, which is what F-5 adds. The remaining `filesystem-bytes` probe failure (guest exits 137/SIGKILL with no rejection) is therefore still UNEXPLAINED and needs direct evidence — it is a bounded guest kill, not a sidecar crash or cross-tenant DoS. | +| **F-3** | LT-022 per-lifecycle leak | Root cause: `LocalBridge.snapshots: BTreeMap` (`crates/native-sidecar/src/stdio.rs:2567`) — `finish_vm_teardown` snapshots the **entire root filesystem on every VM dispose** and inserts it (`stdio.rs:2757`), with **no removal anywhere in the tree**, so one full-FS blob (~0.5 MiB) was retained per VM lifecycle for the sidecar's lifetime. Fix: added `PersistenceBridge::forget_filesystem_state(vm_id)` (default no-op, so bridges that persist externally are unaffected) in `crates/bridge/src/lib.rs`, implemented it in `LocalBridge` to drop the entry, and call it **unconditionally** on the dispose path in `crates/native-sidecar/src/vm.rs` alongside `reclaim_vm_tracking` so a failed teardown cannot strand the entry. VM ids are monotonic and never reissued, so a disposed VM's snapshot can never be loaded again. **VERIFIED:** clean soak (batch=1, 200 cycles) after the fix — steady slope **495 KB/cycle → 26.9 KB/cycle (18× reduction)**, total RSS/PSS growth **461 MiB → 29 MiB**, teardown slope ~0, zero probe failures, verdict **pass**. | +| **F-6 (CRITICAL — the actual root cause of LT-011/LT-024)** | LT-011 / LT-024 | `crates/native-sidecar/src/execution/javascript/rpc.rs` `deferred_kernel_wait_request_for_process`. This runs on **every** guest `fs.write`/`fs.writeSync` as a pre-check ("is this fd a pipe, so the write can be deferred?") and called `kernel.fd_stat(fd)` unconditionally. Mapped host fds live in a per-process `BTreeMap`, **not** the kernel fd table, so any write to a host-backed path (`/tmp`, `/workspace`) stat'ed an fd the kernel does not own and raised `EBADF: bad file descriptor 1000000000`. That error escaped `pump_process_events` and was the single defect behind BOTH earlier symptoms: it killed the shared sidecar (LT-011 cross-tenant DoS) before F-4, and after F-4 left the guest's sync-RPC response undelivered until the 31 s bridge deadline (LT-024). **Proven, not inferred** — captured backtrace: `FdTableError::bad_file_descriptor → ProcessFdTable::stat → KernelVm::fd_stat → deferred_kernel_wait_request_for_process → handle_javascript_sync_rpc_request → handle_execution_event → pump_process_events`. Fix: return `Ok(None)` (not deferrable) for `fd >= MAPPED_HOST_FD_START`, since a host-backed fd can never be a kernel pipe; plus defense-in-depth treating any `EBADF` from this stat as "not a kernel pipe, so not deferrable" rather than failing the write with an unrelated errno — closing the whole class, not just this fd range. **VERIFIED:** write sweep went from **31,000 ms hang → 1–4 ms success in all 10 combinations** (4 KiB/64 KiB/256 KiB/512 KiB/1 MiB × `/tmp` + `/workspace`), with no EBADF and no pump error. It also made F-5 reachable for the first time: with an 8 MiB cap, chunks 0–7 now succeed and chunk 8 is rejected with a typed `ENOSPC: ... exceeding the VM filesystem limit of 8388608 (limits.resources.maxFilesystemBytes); raise the limit to store more data` — the enforcement the `filesystem-bytes` probe had been asking for all along. | +| **F-7 (critical — newly exposed by F-6)** | LT-025 shadow-sync fatal on guest quota | Fixing F-6 made guest writes actually succeed for the first time, which immediately exposed a SECOND fatal path in `crates/native-sidecar/src/execution/launch.rs`: the host-shadow-root → guest mirror called `vm.kernel.write_file(...)` and escalated ANY error to `SidecarError::InvalidState`. When a guest legitimately hit its own configured `maxFilesystemBytes`, the mirror got `ENOSPC` and killed the entire shared sidecar — `agentos-sidecar startup failed error=InvalidState("failed to sync host shadow file .../tmp/fill.bin to guest /tmp/fill.bin: ENOSPC: maximum filesystem size limit reached")` → `SidecarProcessExited` → sentinel died and fresh VMs failed. Same cross-tenant blast-radius class as LT-011: one guest filling its OWN quota takes down every co-located VM/actor. Fix: treat `ENOSPC`/`EDQUOT`/`EFBIG` as guest-caused conditions and `warn!`+skip the entry, matching the tolerance the same loop already had for `ENOENT` (mid-churn path) and `EPERM` (unreadable host file). The offending write is already correctly rejected with a typed ENOSPC at the syscall, so nothing is hidden — the failure stays host-visible via the warning but is bounded to the offending VM. **VERIFIED:** the `filesystem-bytes` probe — which had failed for this entire effort — now reports `{"verdict":"pass","rejected":1}`: the cap fires, the sidecar survives, the sentinel stays healthy, and a fresh VM still runs. | +| **F-2** | LT-013 unhandled EPIPE | `packages/runtime-core/src/frame-stream.ts` `StdioFrameTransport`: the constructor attached an `error` listener to `stdout` but not `stdin`, so a failed write (EPIPE when the sidecar dies) emitted an unhandled `'error'` event that Node threw as an uncaught exception, killing the host process. Now `stdin` gets the same handler (removed in `dispose`), so it surfaces as a normal typed transport error. | + + +This is the running issue ledger for the load-test implementation and test +runs. Record observed behavior here even when the underlying friction is also +logged in `~/.agents/friction/`. + +| ID | Status | Lane | Summary | Evidence / disposition | +| --- | --- | --- | --- | --- | +| LT-001 | Open | Compute | Current Rivet docs disagree about fixed runner-count controls. | Pool configuration deprecates `minRunners`, `maxRunners`, and `slotsPerRunner`; debugging still shows `max_runners` and `slots_per_runner`. The Compute test observes demand-driven runner count and does not send deprecated fields. | +| LT-002 | Open | Tooling | The globally installed `browse` CLI points to a missing module. | Refreshing it was blocked by the stale launcher, so official Rivet `.md` endpoints were fetched directly. The original launcher was restored. This does not affect the load-test runtime. | +| LT-003 | Resolved | Image | The scaffolded Dockerfile could not build `@rivet-dev/agentos-core` because core statically imports `@agentos-software/common` (default-software) and `@rivet-dev/agentos-runtime-core`, neither of which the image built. | Root cause: `agent-os.ts` eagerly imports `default-software.js` → `@agentos-software/common` (+ 8 leaves) and `runtime-core`. Fix: `packages/load-tests/build-deps.sh` builds the software leaves + common + manifest with `tsc` (they only export a `packagePath` string, so no `.aospkg`/WASM toolchain is needed for node-only `defaultSoftware:false` workloads), then runtime-core (`build:vm-config` + `tsc`, skipping `copy-commands`), then core (`tsc`, skipping `build:protocols`; the generated protocol source is already present), then the actor bundle and load-tests. Dockerfile now runs a full `pnpm install` + `build-deps.sh`. Verified: full TS chain builds and `packages/load-tests` typechecks + emits on the host. | +| LT-005 | Resolved | Image | The Dockerfile pinned `rust:1.88-bookworm`, but the sidecar's dependency tree (aws-sdk-s3 / smithy, crc-fast) requires rustc >= 1.91.1. `cargo build --release -p agentos-sidecar` failed with exit 101. | Bumped the build stage to `rust:1.97-bookworm` (matches the canonical host toolchain, cargo 1.97). Rebuilt from a clean cargo cache. | +| LT-006 | Open (runtime) | Runtime | AgentOS `node` does not parse `--input-type=module` as a flag; it treats it as the entry module path. | `execArgv("node", ["--input-type=module", "-e", script])` failed with `Cannot find module '/workspace/--input-type=module'`. Real Node accepts this flag. Worked around in the load test by writing the guest workload in CommonJS (`require` + async IIFE, no ESM flag). The underlying node-compat arg parser deviation is a runtime finding for the owning layer; not fixed here to keep the load-test change scoped. | +| LT-011 | **FIXED & VERIFIED ON COMPUTE (was CRITICAL — cross-tenant DoS)** — see F-1/F-4/F-5. **End-to-end proof on the live Rivet Compute deployment with the fixed image:** the same 8-co-located-actor blast-radius test that previously gave `survivorsBefore=8 → survivorsAfter=0` (plus a 10+ minute deployment-wide outage) now returns `{"verdict":"pass","failures":[],"survivorsBefore":8,"survivorsAfter":8}`. The attacking guest is still terminated (`exit=137`, bounded and contained) but **the sidecar survives, all 8 co-located actors keep serving, and a freshly created actor's `execArgv` action returns HTTP 200 immediately** (it returned HTTP 500 `internal_error` for 10+ minutes before the fix). Both the cross-tenant blast radius and the lingering region-wide outage are eliminated. Original analysis retained below for the record. | Runtime/VFS | A guest writing a large file to its own VM filesystem crashes the entire shared sidecar instead of enforcing `maxFilesystemBytes`. | Isolated & reproducible: an attacker VM with `maxFilesystemBytes=8 MiB` running `fs.writeSync` of ~32 MiB to `/tmp/fill.bin` never receives `ENOSPC`; instead the sidecar dies with `Kernel("EBADF: bad file descriptor 1000000000")` (exit 1), killing the sentinel and all VMs and blocking fresh-VM creation. `1000000000` is `MAPPED_HOST_FD_START` (`crates/native-sidecar/src/state.rs:532`); the filesystem write/limit path uses that mapped-host-fd base as a live fd → `EBADF` in `crates/kernel/src/fd_table.rs:147`. **This is a CLASS, not one bug** (sub-agent code audit): any kernel op returning `EBADF`/`ENOLCK`/etc. gets stringified into a hard `SidecarError::Kernel(...)` (`service.rs:3972,3992`) and, on kill/teardown paths, propagated with `?` (`vm.rs:1346,1367`) instead of delivered to the guest as an errno. Mapped-host-fd handles live in a per-process `BTreeMap` (`state.rs:1192`, `process.rs:520`) that the kernel fd table doesn't know about, so every fs syscall must independently re-classify a `>=1e9` fd (`filesystem.rs:1441,1567,1619,1679,1721,1740,1922,1941`) — any missed branch routes a mapped fd into the kernel table → crash. Owning layer: native-sidecar/kernel VFS. Not fixed here (deep VFS change); flagged for the runtime owner. **⚠️ ROOT CAUSE PROVEN ON COMPUTE (not just inferred):** the gateway action response's `output.stderr` carries the dying sidecar's own log — on the live deployment it reads `ts=... level=error message="agentos-sidecar startup failed" error="Kernel(\"EBADF: bad file descriptor 1000000000\")"` (`phase=create_vm` preceded it), byte-for-byte the local EBADF crash. So the Compute "sidecar process exited with disconnect" IS the LT-011 EBADF, confirmed — not an OOM or an unrelated failure. (Note: Rivet does NOT forward the sidecar's stderr to GCP Cloud Logging; it is only recoverable via the gateway action response on disconnect.) **CONFIRMED CROSS-TENANT DoS AT COMPUTE SCALE:** on the live Rivet Compute deployment (`--cpu 2`, 8 co-located actors on one runner), all 8 answered a probe, then ONE actor ran the fs-crash workload → the shared runner sidecar died (`sidecar process exited with disconnect`) → **all 8 actors (survivorsBefore=8 → survivorsAfter=0) went down at once**. So one hostile guest writing a large file takes down EVERY co-located actor on the runner — a multi-tenant availability/isolation failure, not just a single-VM crash. This is the most severe finding. **Blast radius is DEPLOYMENT-WIDE and lingers:** after the crash, `/api/rivet/health` stays 200 but EVERY fresh actor's `execArgv` action returned HTTP 500 `internal_error` for 2+ minutes (5/5 retries over ~2 min) — so a single guest crash takes down not just co-located actors but the deployment's ability to run ANY actor action for minutes. **Recovery is SLOW (>10 min):** new actors created after the crash keep getting scheduled (`runner_name_selector:"default"`) onto the same crashed us-west-1 runner and return HTTP 500 `internal_error` (`rivetkit_core::registry::http actor framework action error response`) for 10+ minutes — the crashed runner's dead sidecar is not promptly detected/evicted, so the scheduler places new actors on a zombie runner. GKE did boot a fresh us-west-1 node (reschedule in progress) but recovery had not completed after 10 min. **Vector map (isolated sweeps):** repeated `fs.writeSync(fd, 1MiB)` accumulating past ~32–48 MiB CRASHES on BOTH `/tmp` AND `/workspace` (not path-specific); a single 64 MiB `fs.writeFileSync` and a `pwrite` at 64 MiB offset do NOT crash. So the trigger is the incremental chunked-APPEND write path (overlay growth/host-mirror), not single or positional writes — narrows the owner's search. | +| LT-015 | Open (by design) | Runtime | Excess concurrent guest executions are REJECTED, not queued. | Creating 50–200 VMs on one sidecar (8-CPU container) and firing `execArgv` in all concurrently: only 8 execute; the rest get `sidecar rejected request N: execution_error: ... ERR_AGENTOS_VM_EXECUTOR_LIMIT: active V8 executors reached limit of 8; raise runtime.executor.maxActiveVms`. So the executor pool (= CPU count, LT-008) is a hard concurrency cap and surplus executions fail immediately rather than waiting in a queue. The sidecar SURVIVES (reconciles fd/thread/pid to baseline, no OOM), and the rejection is typed/actionable — but a caller must bound execution concurrency to the CPU count or handle rejections. Also observed: a possible concurrent-VM admission cap (~105 coexisting VMs at the 200-VM test) — under investigation. | +| LT-021 | Info | Compute/GCP | Rivet Compute runs on **GKE Autopilot**, multi-region shared clusters. | Found via gcloud (`nathan@rivet.gg`, project `cloud-prod-474518`): clusters `ap-southeast-1-engine-autopilot` (18 nodes), `eu-central-1` (12), `us-east-1` (7), `us-west-1` (16), `cloud-api-autopilot` (4). The `rivet-engine` namespace holds the control plane (nats/guard/worker/pegboard); serverless actor runners are ephemeral (no per-deployment persistent pods). So the "autoscaler" is GKE Autopilot node autoscaling (pod-request-driven bin-packing), and node-count is the scale signal — but clusters are shared across all Rivet customers, so isolating one deployment's impact requires watching for its runner pods live during a ramp. | +| LT-022 | **Open (HIGH — soak/leak)** | Runtime | Sustained VM churn grows sidecar memory ~0.5 MiB per VM lifecycle without plateauing; reclaimed only on full sidecar teardown. | 200-cycle soak: sequential RSS climbed 226→323 MiB monotonically; steady-state slope was ~0.53 MB/cycle at 30 cycles AND ~0.50 MB/cycle at 200 cycles — a plateauing warmup's average slope SHRINKS with run length, so a slope that stays constant across 30→200 cycles indicates a real slow leak of per-lifecycle state (generation records / caches / ledgers) that VM dispose does not reclaim. Post-teardown slope is 0 (freed when the sidecar exits), so it is not a classic unbounded leak — but a long-lived shared sidecar servicing millions of VM lifecycles would grow until restart. (The soak's op/sentinel failures were separate executor-contention noise — LT-015 — from a burst wider than the executor pool; a clean batch=1 soak is running to confirm the slope in isolation.) Owner: audit per-VM-lifecycle state not released on dispose. **CONFIRMED as a real leak by a clean soak** (batch=1 so no executor contention, 300 cycles/shape ≈ 900 VM lifecycles, zero op errors): steady slope 0.495 MB/cycle vs 0.50 at 200 cycles and 0.53 at 30 — **constant across run lengths**, which is the leak signature (a plateauing warmup's average slope decreases). 461 MiB retained over ~900 lifecycles ≈ 0.51 MiB per VM lifecycle; fd/thread/pid flat; released only on sidecar exit. | +| LT-023 | **Open (MEDIUM — observability)** — mitigated behind a flag | Client (`packages/runtime-core/src/process.ts`) | A LIVE sidecar's stderr is invisible to the host: it is buffered into `stderrChunks` and only replayed via `SidecarProcessExited`/`SidecarProcessError`, i.e. **only if the sidecar dies**. Every `tracing::error!`/`WARN_*` emitted by a sidecar that survives a guest-triggered failure is silently discarded. | Directly blocked this investigation: a guest write failing inside the sidecar produced ZERO host-visible output (`RUST_LOG=debug` changed nothing) — the only symptom was a 31 s client-side timeout. This violates the repo rule that host-visible warnings/errors must reach stderr/log rather than staying trapped. **Mitigation applied:** `AGENTOS_SIDECAR_STDERR=1` now forwards sidecar stderr to host stderr as it arrives (still buffered for the exit path). That immediately surfaced the LT-024 root cause. A flag is a stopgap — the right fix is a structured, always-on host log/trace sink for sidecar diagnostics. | +| LT-024 | **Open (CRITICAL — guest file writes are broken; supersedes the LT-011 "fix")** | Runtime/VFS + native-sidecar | **Every** guest file write hangs for 31 s and fails with `ERR_AGENTOS_BRIDGE_CALL_TIMEOUT` instead of writing. The underlying `EBADF: bad file descriptor 1000000000` (LT-011's root cause) was never fixed — F-4 only stopped it from killing the process, converting a loud crash into a silent hang. | **Directly evidenced** (not inferred) once LT-023 was mitigated. Sidecar log during a single 4 KiB `fs.writeSync`: `level=error message="process-event pump failed for session; continuing to serve other sessions" session=session-1 error="EBADF: bad file descriptor 1000000000"`. Chain: guest write → process-event pump raises EBADF on `MAPPED_HOST_FD_START` → **F-4 catches it, logs, and `continue`s** → the guest's pending sync-RPC response is *never delivered* → guest blocks until `limits.reactor.operationDeadlineMs` (31 s) → `ERR_AGENTOS_BRIDGE_CALL_TIMEOUT`. Before F-4 the identical error propagated to `main` → `exit(1)` → the cross-tenant crash. **Size- and path-independent**, so this is not a quota/large-write issue: a sweep of 4 KiB / 64 KiB / 256 KiB / 512 KiB / 1 MiB writes to BOTH `/tmp` and `/workspace` stalled ~31 000 ms in all 10 combinations. This also fully explains the previously-unexplained `filesystem-bytes` probe result (`exit=137`, "maxFilesystemBytes did not fire"): the guest never completes a single write, so it is killed by the harness timeout long before any byte cap could be reached — the cap was never the issue. **Correction to my earlier claim:** reporting LT-011 as "fixed" was correct only about *blast radius* (verified: co-located actors survive, 8→8 on Compute) and wrong to imply the defect was resolved — swallowing the error also violates the repo's "never swallow errors silently" rule. Correct fix has two parts: (1) stop routing mapped host fds (>= `MAPPED_HOST_FD_START`) into the kernel fd table, and (2) make pump failures deliver a typed errno to the waiting caller instead of dropping its response. | +| LT-020 | Open (by design) | Compute | On a `--cpu 1` runner, per-runner AgentOS execution concurrency is 1 (executor pool = CPU count, LT-008/015), so co-located actors calling `execArgv` serialize and time out under contention. | A single-actor gateway `execArgv` works (`POST /gateway//action/execArgv` `{"args":["node",["-e",...]]}` → `{"output":{"exitCode":0,"stdout":...}}`). But 25 co-located actors each calling `execArgv` on a `--cpu 1` runner all timed out (25 s) — the noisy-neighbor blast-radius test couldn't even establish a baseline (`survivorsBefore=0`). Implication: per-runner guest execution throughput scales with `--cpu`, not actor count; a realistic runner needs `--cpu >= 2` (or fewer actors/runner) for concurrent AgentOS work. Retried with 6 co-located actors. | +| LT-018 | **Open (HIGH — scaling)** | Compute | A fast burst to hundreds of actors overwhelms Compute scale-up: a large fraction fail the health timeout. | With `--max-scale 8 --max-concurrent-actors 25` and a create-storm at concurrency 48, ramp 25→50→100→150→200: step 25 = all ready; step 50 = 25 ready / **25 failed**; 100 = 50/50; 150 = 56 ready / **94 failed**, create-to-ready **P99 57 s**; 200 = 76 ready / **124 failed**. `peakConnectable` = 25 (one runner's slot count). Deployment logs confirm runners DID scale across regions (`ap-southeast-1`, `eu-central-1`, `us-east-1`, `us-west-1` booted mid-ramp), so scale-up happens — but too slowly to absorb a burst: actors wait >60 s for a slot and time out. Cleanup was clean (0 leaked actors). Net: the managed pool scales, but burst admission of hundreds of actors is not absorbed — high failure rate + tens-of-seconds cold-start. Needs either slower client ramp, higher `--max-concurrent-actors`, or faster Rivet scale-up. **Characterized (gentle ramp — concurrency 6, 150 s timeout, 45 s holds): 25→all ready, 50→44/50, 100→91/100 ready (only ~9% fail), but slowest create-to-ready P99 = 108 s.** So it is "can't scale FAST", not "can't scale": Compute absorbs ~90% of 100 actors when the client is patient, but new-runner/region cold-start is ~100+ s, so a fast burst (conc 48, 60 s timeout) causes majority failures. Owner action: reduce cold-start latency and/or document a client back-pressure/ramp guidance for large actor fleets. | +| LT-019 | Info (gcloud pending) | Compute/GCP | Rivet Compute runs the runners as **multi-region managed** infra (`ap-southeast-1`/`eu-central-1`/`us-east-1`/`us-west-1` seen in logs). | The active gcloud account is `nathan@rivet.gg` (a Rivet account) but its token is expired and the current project is `amp-prod-0` (unrelated). Since the account is a Rivet one, it likely CAN see the Rivet GCP project hosting the runners — pending `gcloud auth login`, then `gcloud projects list` to find the Rivet project and `gcloud compute instance-groups managed list` / `gcloud container clusters list` to watch runner instances scale during a ramp. Until re-auth, the `/actors` lifecycle census + `rivetkit logs` + the dashboard are the observability surface. | +| LT-017 | Open (scaling limit) | Runtime | Concurrent VM execution is disk-bound: the per-VM Node import cache exhausts tmpfs at scale. | 300 idle VMs coexist fine on one sidecar (peakActiveVmCount=301, sentinel 100%, creation ~220 ms, clean dispose — **no admission cap at 300**). But when executing across them, from ~VM 95 onward `execArgv` fails with `execution_error: failed to prepare sidecar-scoped Node import cache: No space left on device (os error 28)` on a 2 GiB `/tmp` tmpfs. So each VM's import-cache prep consumes ~20 MiB of tmpfs and it does not appear shared/deduped across VMs → concurrent-execution VM count is bounded by tmpfs size. The sidecar SURVIVES (typed ENOSPC to the guest, sentinel healthy, reconciles) — resilient, but a real density limit; a larger/overflow-to-disk or shared import cache would raise it. | +| LT-016 | Resolved (config) | Compute | The initial deploy used the default `--max-scale 1`, so every ramp was a single-runner test — no multi-runner scaling was possible. | Every 1→10 ramp looked identical because one runner held all actors (`--max-concurrent-actors` default 1000). Redeployed with `--max-scale 8 --max-concurrent-actors 25 --cpu 1 --memory 2Gi --min-scale 0 --drain-grace-period 10` so 25 slots/runner forces scale-up as actor count crosses 25/50/…/200 (bounded worst case 8 runners; scale-to-zero floor). | +| LT-013 | Open | Runtime/harness | A JS-runtime kill workload crashed the shared sidecar mid-matrix, and the SDK's stdio frame transport then died with an unhandled `EPIPE`. | Running the JS kill probes (`js-cpu`/`js-wallclock`/`js-heap`/`js-output`/compound) on one shared sidecar, a probe killed the sidecar and the next write hit a dead pipe: `Unhandled 'error' event ... write EPIPE at packages/runtime-core/dist/frame-stream.js:155 StdioFrameTransport.writeFrame`, crashing the whole controller before an artifact was written. Two takeaways: (a) `StdioFrameTransport.writeFrame` should handle `EPIPE` gracefully (typed error to the caller), not emit an unhandled socket `error`; (b) the matrix should isolate kill-style probes on their own sidecar. Not yet isolated/fixed — recorded for follow-up. | +| LT-012 | Info | Compute | Locally (no Compute env), `compute-server` runs RivetKit in "Engine - Serverful" mode on port 6420, not `RIVET_PORT=3000`. | The image boots and registers 1 actor (`agentosLoadRunner`) — confirmed via logs `RivetKit 0.0.0-sqlite-uds... (Engine - Serverful) Endpoint: http://127.0.0.1:6420 Actors: 1`. Without `RIVET_ENDPOINT`/Compute env, RivetKit does not use the automatic serverless mode or `RIVET_PORT`; it serves the local-native endpoint on 6420. On real Rivet Compute the environment selects serverless + `RIVET_PORT`. Local `/api/rivet/metadata` must be probed on 6420, and full metadata/actor verification still needs a Rivet engine (local `rivet dev` or a deployment) — LT-004. | +| LT-010 | Open (runtime) | Runtime | `onLimitWarning` only fires for `vm_processes`; fd/socket/filesystem near-limit warnings reach only sidecar stderr. | `maxProcesses` delivers a structured `LimitWarning` to the host `onLimitWarning` callback (`bounded limit near capacity limit=vm_processes`). `maxOpenFds`/`maxSockets` DO warn near threshold but only as `WARN_AGENTOS_RESOURCE_NEAR_LIMIT ... config=limits.resources.maxSockets` on sidecar stderr, not via the callback. Host-visible (satisfies the stderr contract) but inconsistent; a client relying solely on `onLimitWarning` misses fd/socket/fs pressure. Load-test probes updated to expect the callback only for processes. **Root cause identified:** the fd cap is enforced inside `FdTableManager::with_max_fds(...)` (`crates/kernel/src/kernel.rs:676`), which returns `EMFILE` directly, while the gauge that feeds `onLimitWarning` is only sampled by `ResourceAccountant::observe_resource_gauges`, called solely from `resources.snapshot(...)` (`crates/kernel/src/kernel.rs:1072`). The fd-open and socket-create paths never take that snapshot, so their gauges never cross the 80% warn threshold; `maxProcesses` warns only because its limit check does take a snapshot (`resource_accounting.rs:334`). Fix would be to give `FdTableManager`/`SocketTable` a direct gauge handle and `observe_depth` on alloc/free (O(1)) rather than calling the expensive full `snapshot()` on the hot path. NOT fixed here: non-breaking (the warning still reaches stderr as `WARN_AGENTOS_RESOURCE_NEAR_LIMIT`) and the hot-path plumbing deserves its own change. | +| LT-008 | Open (by design) | Runtime | A guest node-process storm hits the V8 executor concurrency limit before `maxProcesses`. | With `--cpus=2`, `runtime.executor.maxActiveVms` defaults to the CPU count (`crates/runtime/src/lib.rs:441` `available.max(1)`), so only 2 concurrent V8 executors exist. A storm spawning `node` children (each needs an executor) is rejected with a clean, actionable typed error — `ERR_AGENTOS_VM_EXECUTOR_LIMIT: active V8 executors reached limit of 2; raise runtime.executor.maxActiveVms` — while the `vm_processes` gauge still warns at 87%. This IS bounded failure + survival (the load test's core assertion), just of the executor limit, not `maxProcesses`. Isolating `maxProcesses` would need executor concurrency > cap (more CPUs) or non-V8 guest processes (WASM commands, not built here). Not a defect: the executor cap is the correct binding bound for concurrent node execution. | +| LT-009 | Resolved (no leak) | Churn | Steady-state RSS climbs during churn, but it is plateauing V8/glibc arena high-water, not a leak. | Decisive evidence: the constant-live steady slope DECREASES with run length (1.15 MB/cycle at 12 cycles → 0.53 MB/cycle at 30) — a true leak keeps a constant slope; a plateauing warmup's average slope shrinks. Post-teardown RSS/PSS slopes are ~0, fd/thread/pid stay flat (33/25/2), and the teardown series is flat at 86 MiB. Fixed the methodology to fit the slope on the post-warmup window (last third) and defaulted to 20 cycles so warmup is excluded; exact-count invariants remain the hard leak gates. | +| LT-007 | Resolved | Churn | The churn leak gate fitted its RSS slope on the warmup **sequential** phase and used a 64 KiB/64 MiB provisional ceiling, so it failed on benign V8/glibc arena retention. | First run: all exact-count invariants PASS (fd 33→33, thread 25→25, pid 2→2, sentinel 100%, clean teardown), only the RSS byte gates failed (88 MiB total, 515 KiB/lifecycle on the warming phase). Reworked to gate on **plateau** slopes — the constant-live steady phase and a zero-attacker post-teardown series — plus PSS (`smaps_rollup`) and host-heap sampling, with provisional RSS/PSS total ceilings to be calibrated from >=5 clean reps. Exact-counts remain the hard leak gates. | +| LT-004 | Resolved | Compute | The external Compute controller needed the Engine API endpoints. | The user supplied `RIVET_ENDPOINT` (`sk_`), `RIVET_PUBLIC_ENDPOINT` (`pk_`), `RIVET_RUN_URL`, and the `cloud_api_` token. Stored only in a runtime env file outside the repo (0600), never committed/printed; **rotate them after use** (they were pasted into task text). Deploy + actor E2E succeeded (see run log). | +| LT-014 | Open | Compute/API | The documented `/runners` census returns 0 for a live serverless deployment, and actor DELETE + runner-configs need the `sk_` token, not `pk_`. | With a healthy actor live (gateway `/health`=200), `GET /runners?namespace=..&name=default&include_stopped=true` returns 0 runners (also 0 without the `name` filter). `GET /runner-configs` → 403 with `sk_`. Actor `DELETE /actors/` → 403 with `pk_` (`insufficient_permissions`) but 200 with `sk_`. Actor CREATE works with `pk_`; the created id is at `body.actor.actor_id`. Consequence (extends LT-001): the runner-count scaling signal is not observable via this endpoint/token, so the scaling verdict rests on actor-lifecycle metrics (create-to-ready, health, destroy/drain), not runner count. | + +## Run log + +Add one row per meaningful run. Raw JSON belongs under +`.artifacts/load-tests/`, not in git. + +| Date | Revision | Lane | Configuration | Result | Artifact / notes | +| --- | --- | --- | --- | --- | --- | +| 2026-07-19 | load-test | boundary | mem=3g swap=0 pids=256 nofile=1024 net=none tmp=512m, non-root | pass | Envelope matches recipe flags exactly. `.artifacts/load-tests/boundary/`. Proves the sandbox before any survival verdict. | +| 2026-07-19 | load-test | limits (process) | maxProcesses=8, 32 spawn attempts | pass | 31/32 rejected with typed `ERR_AGENTOS_VM_EXECUTOR_LIMIT` (see LT-008), `vm_processes` warning at 87%, sidecar ready, sentinel 100% (p99 250ms), activeVms→1, no OOM. `.artifacts/load-tests/limits/`. | +| 2026-07-19 | load-test | churn | 12 cycles seq/burst/steady | fail→diagnosed | Exact-counts pass; steady-slope gate tripped on warmup (LT-007/LT-009). Led to plateau-window methodology fix. | +| 2026-07-19 | load-test | churn | 30 cycles seq/burst/steady | pass | steady-slope 0.53 MB/cyc (down from 1.15 at 12c → plateau), teardown slope 0, fd/thread/pid flat, RSS/PSS growth 137/140 MiB < ceilings. Confirms no leak (LT-009). `.artifacts/load-tests/churn/`. | +| 2026-07-19 | load-test | limits-matrix | processes/fds/sockets/filesystem, dist overlay on the built image | fail (2 findings) | **processes**: pass (executor limit, LT-008). **open-fds**: limit enforces — 99 rejections with `EMFILE: VM open file descriptor limit 64 reached (limits.resources.maxOpenFds)`; only warning-callback missing (LT-010). **sockets**: limit enforces — typed `ERR_AGENTOS_RESOURCE_LIMIT resource=sockets used=32 limit=32`; stderr `WARN_AGENTOS_RESOURCE_NEAR_LIMIT` (LT-010). **filesystem-bytes**: **sidecar crash** (LT-011). Fresh-VM-after-attack OK for the first three. `.artifacts/load-tests/limit-matrix/`. | +| 2026-07-19 | load-test | compute-server (local) | `compute-server` in a 2 GiB bounded container, no Rivet env | boots + 1 actor | Image boots RivetKit and registers 1 actor (`agentosLoadRunner`); logs `Engine - Serverful, Endpoint http://127.0.0.1:6420, Actors: 1`. Runs in serverful mode on container-localhost:6420, not `RIVET_PORT=3000` (LT-012). Full metadata/actor drive needs a Rivet engine or deployment (LT-004). | +| 2026-07-19 | load-test | compute deploy | `npx @rivetkit/cli deploy --dockerfile packages/load-tests/Dockerfile --build-context . --env PORT=3000 --yes` | success | Managed pool `ready`; image built `linux/amd64` + pushed to `registry.rivet.dev`. Run URL `https://agentos-stress-socv-production-iboo.rivet.run` `/api/rivet/health` → 200. Local serverless smoke (`RIVETKIT_RUNTIME_MODE=serverless`) also 200, binds `0.0.0.0:3000`, `Actors: 1`. | +| 2026-07-19 | load-test | compute actor E2E | raw Engine API | pass | `POST /actors` (pk_) created `agentosLoadRunner` key=smoke → id at `body.actor.actor_id`; gateway `/health` (pk_) → 200 "ok"; `DELETE /actors/` (sk_) → 200. Found LT-014 (empty `/runners` census; DELETE needs sk_). | +| 2026-07-19 | load-test | compute-load scaling | ramp 1,3 then 1,5,10 keyed actors, bounded container w/ egress | pass | All 16 actors became healthy (0 failures). create-to-ready P50 ~0.4–0.9 s with cold-start P99 spikes (up to ~31 s when a runner spins up). Clean drain + destroy (0 leaked actors both runs; deployment stays healthy). Runner-count curve not observable (LT-014); scaling verdict from actor-lifecycle metrics. `.artifacts/load-tests/compute/`. | +| 2026-07-19 | load-test | scale (local) | 300 idle VMs on one sidecar, 8-CPU/8-GiB bounded container | 300 coexist | peakActiveVmCount=301, sentinel 100%, creation ~220 ms, clean dispose — **no admission cap at 300**. Execution across them hit LT-015 (executor=CPU cap) then LT-017 (import-cache tmpfs ENOSPC at ~95 VMs). Sidecar survived all. | +| 2026-07-19 | load-test | fs-crash vector sweep | /workspace chunked / 64MiB writeFile / pwrite-offset, isolated | 1 crash | `/workspace` chunked write CRASHES (same EBADF 1000000000); single writeFileSync + pwrite-offset survive → LT-011 is the chunked-append path on any dir. | +| 2026-07-19 | load-test | compute scale-up curve | redeployed max-scale 8 / 25 slots; ramp 25→50→100→150→200 @ conc 48 | fail (LT-018) | 25 all ready; 50→200 up to 62% fail the 60 s health timeout, P99 create-to-ready 57 s. Runners scale multi-region (logs) but too slowly for the burst. 0 leaked actors. | +| 2026-07-20 | load-test (F1–F7) | **compute FINAL verification (root cause fixed)** | 8 co-located actors, attacker writes past its `maxFilesystemBytes` | **PASS — designed behavior end-to-end** | `{"verdict":"pass","failures":[],"survivorsBefore":11,"survivorsAfter":22}`. The attacker now exits **1 with a clean catchable `Error: ENOSPC: writing 1048576 byte(s) ... would reach 68157440 bytes`** naming the limit — not a sidecar crash (pre-F6), not a 31 s hang (LT-024), not `exit=137`. Survivor count GREW during the run (deployment scaled), i.e. zero collateral. Also verified on the same deployment: a guest wrote **1 MiB in 83 ms**, `statSync` confirming 1,048,576 bytes — the write path that previously hung 31 s or killed the sidecar. | +| 2026-07-20 | load-test (F1–F7) | local battery (root cause fixed) | full lane sweep | 7 of 9 lanes pass | **`filesystem-bytes` now PASSES for the first time in this effort** (`rejected:1`, sidecar survives, sentinel healthy, fresh VM runs). fs-crash-workspace, processes, open-fds, boundary, limits, churn-leak all pass. Write sweep: **31,000 ms hang → 1–4 ms success in all 10 size×path combinations**. Remaining: `sockets` probe parent exit code (limit DOES fire — 64 typed rejections) and LT-017 import-cache tmpfs ENOSPC at ~VM 95 in the 300-VM scale lane. | +| 2026-07-20 | load-test (F1–F5) | **compute blast-radius RE-TEST (fixed)** | 8 co-located actors on `--cpu 2`, one trips the fs-crash | **PASS — DoS eliminated** | **survivorsBefore 8 → survivorsAfter 8** (was 8 → 0). The attacker's own guest is still killed (`exit=137`, bounded/contained) but **every co-located neighbor kept working**, the sidecar survived, and a fresh actor action returned **HTTP 200 immediately** (was HTTP 500 for 10+ minutes). Cross-tenant DoS **and** the lingering region outage are both fixed. | +| 2026-07-20 | load-test (F1–F5) | local battery (fixed image) | fs-crash vectors, leak soak, boundary, limits | pass | **0 sidecar crashes** across the battery; `fs-crash-workspace` crash→pass; leak slope **495→26.9 KB/cycle (18×)**, growth **461→29 MiB**. | +| 2026-07-19 | load-test | compute noisy-neighbor | 25/6/8 co-located actors, one trips LT-011 fs-crash | inconclusive | Blast radius UNCONFIRMED on `--cpu 1`: executor pool=1 rejects the crash action before it runs (LT-020) + multi-region spread (LT-019). Retested on `--cpu 2` below. | +| 2026-07-19 | load-test | **compute sleep/wake persistence** | 5 actors, write file to /workspace, sleep, wake, re-read | **PASS** | 5/5 wrote the marker, 5/5 read it back, **5/5 went to sleep (sleep_ts set), 5/5 markers SURVIVED sleep→wake**. The actor filesystem is durable across the VM being recreated on wake. `.artifacts/load-tests/compute/`. | diff --git a/docs-internal/load-testing.md b/docs-internal/load-testing.md new file mode 100644 index 0000000000..e3a7f5213e --- /dev/null +++ b/docs-internal/load-testing.md @@ -0,0 +1,981 @@ +# AgentOS load-testing design + +## Decision + +Build this as two projects, in this order: + +| Project | Question | Where it runs | Why it is separate | +| --- | --- | --- | --- | +| 1. Host resilience and lifecycle | Can hostile guest work take down a sidecar, escape a configured limit, interfere with another VM, or leak resources across create/destroy churn? | A dedicated Linux host or disposable VM | It tests the AgentOS security and lifecycle boundary directly and needs host-level process/resource telemetry. | +| 2. Rivet Compute scaling | How does a deployed AgentOS-backed RivetKit application behave while actor demand causes Compute to scale up, drain, and scale down? | Rivet Compute, driven by an external load generator | It adds a scheduler, serverless actor lifecycle, deployment control plane, remote telemetry, cost limits, and failure modes that should not obscure Project 1. | + +The first two requested angles belong in Project 1. They should share one +harness because a limit attack and VM churn need the same sidecar lifecycle, +sentinel workload, resource census, and artifact format. Project 2 should reuse +Project 1's AgentOS scenarios, but remain a separate deployable application and +test command. + +This is not primarily a throughput benchmark. The first objective is bounded +failure: overload must end in the documented typed error or POSIX errno, the +trusted sidecar must survive, unrelated VMs must remain usable, and teardown +must return resource accounting to baseline. + +## Existing coverage to preserve and reuse + +AgentOS already has useful pieces; the new harness should orchestrate and gate +them instead of replacing them: + +- `examples/resource-limits/server.ts` demonstrates the public resource, + process, JavaScript, Python, and WASM limits. +- `crates/runtime/src/lib.rs` has a deterministic 256-generation regression and + an ignored 50,000-generation accounting/scheduler soak. +- `crates/native-sidecar/tests/service.rs` has a deterministic multi-VM + protocol regression and an ignored TCP/UDP/TLS/HTTP2/signal/bridge soak. +- `.github/workflows/ci-nightly.yml` explicitly runs both ignored soak gates. +- `scripts/benchmarks/memory.bench.ts` measures full process-tree RSS and + teardown reclamation for shared-sidecar AgentOS VMs. +- `packages/runtime-benchmarks` supplies cold-start, concurrency, + interference, memory, and latency baselines. + +The missing layer is a public-SDK adversarial runner that combines these +signals, keeps a healthy sentinel VM beside the attacker, records time series, +and applies reproducible pass/fail rules. + +## Project 1: host resilience and lifecycle + +### Harness shape + +Run the load generator outside the guest and treat it as trusted. A single test +run owns one release sidecar, one sentinel VM, and one or more attacker VMs: + +```text +host controller + shared release sidecar + sentinel VM periodic health/latency probe + attacker VM(s) limit or churn workload + host sampler process tree, RSS/PSS, CPU, fds, threads, children + artifact writer manifest + samples + errors + final verdict +``` + +The sentinel continuously runs a small `exec` and a small loopback network +round trip. It detects a sidecar crash, global deadlock, response starvation, +or cross-VM interference that an attacker-only test would miss. + +Run the harness only inside its dedicated Docker image with explicit CPU, +memory, swap, PID, fd, and wall-clock limits. Never execute an adversarial or +churn entrypoint directly on the host, even for a smoke test. The host may build +the image and collect artifacts, but the target sidecar, VMs, sampler, and +watchdog all run inside the bounded container. + +### Scenario manifest + +Every run should be described by a checked-in JSON-compatible manifest: + +```json +{ + "scenario": "limits/process-count", + "seed": 42, + "sidecar": "target/release/agentos-native-sidecar", + "vmCount": 2, + "concurrency": 4, + "durationSeconds": 300, + "warmupSeconds": 30, + "limits": {}, + "hostEnvelope": {}, + "expectedTermination": "typed-limit" +} +``` + +The result artifact must include the resolved sidecar path, commit, hardware, +kernel, cgroup envelope, scenario, seed, exact configured limits, timestamps, +all unexpected errors, sampler data, and verdict. Write run artifacts under an +ignored directory such as `.artifacts/load-tests//`; do not commit raw +time series. + +### Lane A: adversarial limit enforcement + +Test each limit at `limit - 1`, `limit`, `limit + 1`, and a large attempted +overshoot. Then test combinations near several limits, because admission bugs +often appear only while multiple ledgers are under pressure. + +Initial matrix: + +| Surface | Guest pressure | Expected boundary | +| --- | --- | --- | +| Processes | fork/spawn storm, fast exit/re-spawn, deep process trees | Process count/argv/env caps; no host process escape or orphan | +| Descriptors | files, pipes, PTYs, listeners, accepted sockets | Correct errno or typed limit; descriptor count returns to baseline | +| Network | TCP/UDP/Unix connect churn, unread streams, tiny writes, TLS/HTTP2 fan-out | Socket, connection, datagram, and buffered-byte caps; sentinel remains responsive | +| Filesystem | byte/inode fill, deep trees, large read/write/readdir requests | Filesystem and operation-size caps; no host disk exhaustion | +| Queues and output | stdin flood, stdout/stderr flood, event/completion backlog | Count/byte limit fires before unbounded allocation; error reaches host logs | +| JavaScript | infinite loop, timer flood, heap growth, oversized IPC/event payload | CPU, wall-clock, timer, heap, and frame limits terminate only the offending execution | +| WASM | fuel loop, memory/stack growth, oversized module/output | Fuel, linear memory, stack, module, and captured-output limits | +| Python | infinite execution, output flood, heap growth | Execution and output/heap limits with cleanup | +| Agent/session | session, prompt, permission, history, and update fan-out | ACP count/byte caps; no retained listeners or session state after teardown | +| Bindings/plugins | registration and schema/example growth, slow callback | Count, byte, and timeout caps; reservations release once | + +For every probe assert: + +1. The configured boundary fires, rather than the host OOM killer or an + unrelated timeout. +2. The error is typed and names the limit and configuration path, or the guest + receives the Linux-compatible errno for a POSIX surface. +3. A host-visible warning appears near the threshold where that contract + applies. +4. The sidecar stays alive and the sentinel continues to make progress. +5. A fresh VM can be created after the failure. +6. Per-VM and process accounting, fds, tasks, and child processes reconcile + after disposal. + +Add compound profiles after the single-limit matrix is stable: + +- CPU loop plus stdout flood. +- Socket churn plus unread receive buffers. +- Process churn plus filesystem fill. +- VM teardown while callbacks, signals, DNS, TLS, or bridge replies are late. +- Several attackers saturating different limits beside one sentinel. + +### Lane B: lifecycle and leak detection + +Use both idle and active workloads: + +- `idle`: create, start one minimal Node process, settle, dispose. +- `exec`: create, run repeated shell/Node/WASM commands, dispose. +- `network`: create loopback TCP/UDP/TLS traffic, dispose while traffic is live. +- `agent-session`: open a deterministic llmock-backed session, prompt, close, + dispose. +- `dirty`: dispose while processes, timers, sockets, callbacks, and output are + still active. + +Run each workload in these shapes: + +| Shape | Pattern | What it exposes | +| --- | --- | --- | +| Sequential churn | create -> work -> dispose, one at a time | Per-generation retained state and allocator growth | +| Burst churn | create N concurrently -> work -> dispose N | Admission and teardown races | +| Sawtooth | grow from 0 to N, drop to 0, repeat | High-water-mark growth that never plateaus | +| Steady state | hold N live while continuously replacing a fraction | Leaks hidden by always-live resources | +| Mixed | idle, network, exec, and agent VMs in one shared sidecar | Cross-subsystem cleanup and fairness | +| Owned sidecars | repeatedly create and destroy the sidecar itself | Process, pipe, temp-file, and client cleanup outside per-VM teardown | + +Sample at least once per second and at epoch boundaries: + +- process-tree RSS and, where available, PSS; +- host Node heap used; +- sidecar CPU time; +- sidecar and process-tree fd count; +- thread count and child-process count; +- cgroup memory current/peak, pressure, task count, and OOM events; +- sidecar task/capability/resource ledgers and quarantine count; +- completed creates/disposals, failure counts, and sentinel latency. + +RSS not returning immediately is not by itself proof of a leak: V8, libc, and +the kernel retain arenas and page cache. Warm every lazy path before the +baseline, force host GC only where supported, divide the run into equal epochs, +and require the post-teardown series to plateau. Diagnose a positive RSS trend +against PSS, heap, fds, threads, children, and the internal accounting census. + +Initial leak gate, to be calibrated on the canonical host: + +- exact zero residual VM ledger usage and no quarantined VM; +- no net fd, thread, or child-process growth after the settle window; +- no monotonic heap growth across the final half of the run; +- post-teardown RSS/PSS slope statistically indistinguishable from zero across + the final half, with a temporary provisional ceiling of 64 KiB per lifecycle + and 64 MiB total growth; +- sentinel success rate 100%, with no lost registered response; +- sentinel p99 no worse than 2x its unloaded baseline after pressure stops. + +The byte thresholds are provisional, not product promises. Establish them +using at least five clean repetitions on the canonical host, store the baseline +distribution, and gate on both slope and total growth. Exact-count invariants +remain hard failures regardless of memory noise. + +### Lane C: disposable-box kill tests + +Deterministic small-limit tests must stay in normal CI. Tests whose purpose is +to find an unbounded path by exhausting the machine belong in an explicit +destructive profile on a disposable worker. + +That profile gradually raises one pressure source until one of these happens: + +- AgentOS rejects it at a documented bound: pass. +- Only the attacker VM is terminated by its configured execution limit: pass. +- The sidecar exits, the sentinel stalls, the cgroup OOMs, or the host becomes + unhealthy: fail and preserve logs/core dumps. + +Use a watchdog outside the target cgroup. It must stop load, collect cgroup and +kernel OOM state, kill only the scoped test unit if necessary, and mark the run +failed. The watchdog must never depend on the sidecar it is supervising. + +### Project 1 implementation sequence + +1. Add a dedicated internal TypeScript package for the controller, scenarios, + sampler, and JSON artifacts, plus a Docker image that builds the exact + workspace sidecar. Do not add repository-specific commands to the root + `package.json`; expose only Docker-wrapped entrypoints with `justfile` + recipes or scoped package scripts. +2. Implement the sentinel and one end-to-end process-count probe first. Prove + the artifact and cleanup contract before expanding the matrix. +3. Add sequential and burst churn using the existing benchmark workload + helpers, then add dirty teardown and network workloads. +4. Expose any missing low-cardinality sidecar census needed to distinguish an + allocator plateau from leaked AgentOS state. +5. Run a short smoke profile in PR CI, deterministic limit cases in normal CI, + 30-60 minute churn in nightly CI, and destructive/long soak profiles only by + explicit dispatch on a disposable worker. + +Project 1 is complete when every public configured limit has deterministic +coverage, compound attacks cannot take down the sidecar or sentinel, and the +lifecycle profiles plateau under the calibrated leak gates. + +## Project 2: Rivet Compute scaling + +### Architecture + +Keep the load generator outside Rivet Compute so target saturation cannot hide +or coordinate the offered load: + +```text +external controller/load generator + Rivet Engine management + gateway APIs + many keyed load-runner actors + Rivet Compute managed pool + AgentOS sidecar + VM workloads + runner census + deployment logs + result artifacts +``` + +Create a small RivetKit application with one `agentos-load-runner` actor. Each +actor accepts bounded actions such as `startScenario`, `status`, and +`stopScenario`. Persist only run identity, desired scenario, progress, and final +summary; live AgentOS handles remain process-local and must be recreated or +reported interrupted after actor migration. + +Use unique actor keys for distribution and reproducibility. Do not put a huge +number of AgentOS VMs behind one actor and call that a Compute scaling test: +that measures one container's AgentOS density. The matrix needs both axes: + +- actor count, which drives scheduling and runner demand; +- AgentOS work per actor, which drives CPU and memory per Compute instance. + +### Scaling matrix + +Run ramps, holds, and drops for at least these profiles: + +| Profile | Actor behavior | Purpose | +| --- | --- | --- | +| Actor-only | Actor starts and reports health without an AgentOS VM | Rivet control-plane and cold-start baseline | +| Idle VM | One live idle AgentOS VM per actor | Memory-driven placement and scale floor | +| Churn | Repeated VM create/work/dispose per actor | Lifecycle behavior during instance scale changes | +| CPU | Bounded guest CPU bursts | CPU-driven scale-up and noisy-neighbor behavior | +| Memory | Bounded live-VM staircase | Memory-driven scale-up without guest OOM | +| Mixed | Churn + network + deterministic agent session | Representative load and teardown | + +For each profile: + +1. Ramp keyed actors through calibrated steps such as 1, 10, 25, 50, and 100. +2. Hold each step long enough for runner count and latency to stabilize. +3. Drop demand to an intermediate step, then to zero. +4. Repeat at least three times to distinguish a consistent curve from a cold + deployment or regional outlier. +5. Run one deployment upgrade during steady load to test drain and actor + rescheduling separately from ordinary autoscaling. + +Capture: + +- actor create-to-ready and action latency distributions; +- request, scheduling, actor crash/restart, and migration errors; +- active/stopped runner census, remaining/total slots, connect/ping/drain/stop + timestamps, and runner-pool errors; +- deployment/container CPU and memory where Compute exposes them; +- AgentOS scenario progress, VM failures, and per-actor teardown summary; +- time to scale up, time to drain, and time to return to the idle runner count; +- peak instance count and estimated compute cost. + +Current Rivet documentation is contradictory about manual runner-count knobs: +the pool-configuration reference deprecates `minRunners`, `maxRunners`, and +`slotsPerRunner`, while the debugging reference still shows some of those +fields. Therefore the first Compute milestone must **observe demand-driven +autoscaling**, not assume fixed instance-count controls. Confirm a supported +Rivet Compute control-plane API before adding a test that explicitly sets the +instance count. + +### Deployment runbook + +Use the current RivetKit documentation index at . +The relevant current pages are: + +- +- +- +- +- +- + +The older `/docs/connect/rivet-compute/` path in the original notes has moved +to `/docs/deploy/rivet-compute/`. + +#### Secrets + +Never commit literal credentials. The values supplied with the original task +are intentionally omitted from this document. Rotate any management or secret +token that has been pasted into task text before using the runbook. + +Provide these at runtime or through the CI secret store: + +```bash +export RIVET_CLOUD_TOKEN='cloud_api_...' +export RIVET_ENDPOINT='https://:sk_...@api.rivet.dev' +export RIVET_PUBLIC_ENDPOINT='https://:pk_...@api.rivet.dev' +export RIVET_RUN_URL='https://.rivet.run' +``` + +`RIVET_CLOUD_TOKEN` is a Cloud API management token used by deploy/log +commands. `sk_*` and `pk_*` tokens are Engine API tokens; use the secret key for +the external controller and the publishable key only where a client-safe key is +required. Do not bake any of them into the image. + +#### Application and container + +Keep `registry.start()` in the application. The current RivetKit runtime-mode +documentation defines it as the automatic mode that starts the server and +serves actors/static files; do not hand-mount an HTTP handler for Compute. Let +RivetKit listen on `RIVET_PORT`, which defaults to 3000. + +Add a Dockerfile matched to the package manager and build output. A Node/npm +starting point is: + +```dockerfile +FROM node:24-alpine AS build +WORKDIR /app +COPY package.json package-lock.json ./ +RUN npm ci +COPY . . +RUN npm run build + +FROM node:24-alpine +WORKDIR /app +COPY package.json package-lock.json ./ +RUN npm ci --omit=dev +COPY --from=build /app/dist ./dist +COPY --from=build /app/public ./public +EXPOSE 3000 +CMD ["node", "dist/index.js"] +``` + +If there is no frontend, omit the `public` copy. If static output is not in +`public/`, set `RIVETKIT_PUBLIC_DIR` to the actual path. Do not set a runtime +mode in the Dockerfile. + +Add `.dockerignore`: + +```text +node_modules/ +dist/ +.env +.git/ +.artifacts/ +``` + +Build and run locally before deployment: + +```bash +docker build -t agentos-load-runner . +docker run --rm -p 3000:3000 -e RIVET_PORT=3000 agentos-load-runner +curl -f http://localhost:3000/api/rivet/metadata +``` + +The current serverless mode is automatic. If the pinned RivetKit version being +tested requires a local simulation variable, verify that against that version's +documentation and pass it only to `docker run`; never hard-code it in the +image. + +#### Deploy and verify + +Deploy from the Compute application's directory: + +```bash +npx @rivetkit/cli deploy \ + --token "$RIVET_CLOUD_TOKEN" \ + --env PORT=3000 \ + --env RIVET_PORT=3000 +``` + +The CLI builds the Dockerfile, pushes the image, upserts the `default` managed +pool, waits for readiness, and prints the deployment URL. It caches the Cloud +token in `~/.rivet/credentials`; CI should use the environment/secret store +instead of relying on that cache. + +Verify the server metadata and inspect logs: + +```bash +curl -f "$RIVET_RUN_URL/api/rivet/metadata" +npx @rivetkit/cli logs -n 200 +``` + +For a persistent test tail: + +```bash +npx @rivetkit/cli logs --follow +``` + +Use the external controller and the Engine API to create uniquely keyed load +actors and call their actions. Prefer the RivetKit client for the load itself; +use direct management API calls for runner/actor census and debugging. The +current debugging reference documents: + +```bash +curl "$RIVET_API/runners?namespace=$RIVET_NAMESPACE&name=default&include_stopped=true&limit=100" \ + -H "Authorization: Bearer $RIVET_TOKEN" + +curl "$RIVET_API/runner-configs?namespace=$RIVET_NAMESPACE&runner_name=default" \ + -H "Authorization: Bearer $RIVET_TOKEN" +``` + +Confirm the actor name from `/api/rivet/metadata` before generating load. Actor +keys must be serialized in the format required by the current Engine/client +API; do not copy an older curl example without checking the current actor +debugging reference. + +### Compute safety and exit criteria + +Every remote run needs hard ceilings for actor count, offered requests, +duration, per-actor AgentOS VMs, and estimated cost. The controller must have a +local kill switch that stops generating load and requests actor cleanup even if +the deployment is unhealthy. + +Provisional success criteria: + +- the offered-load step completes without unexpected actor or container crash; +- expected limit rejections are classified separately from infrastructure + failures; +- runner count rises under the calibrated load and returns to its idle level + after demand is removed; +- no runner remains stuck connected, draining, or erroring after the cleanup + deadline; +- actor state/progress survives an ordinary Compute reschedule where the + scenario declares itself resumable; +- non-resumable in-memory AgentOS work reports interruption explicitly instead + of silently duplicating or losing work; +- scale-up, drain, latency, error-rate, and cost results are reproducible across + three runs before setting a regression threshold. + +Project 2 is complete when a single command can deploy the pinned application, +run the actor/density matrix from outside Compute, produce a runner-count and +latency timeline, clean up, and fail on a regression against a calibrated +baseline. + +## Recommended delivery milestones + +1. **Local smoke:** controller, sentinel, process-count attack, and sequential + idle-VM churn with complete artifacts. +2. **Local coverage:** full deterministic limit matrix, burst/dirty/mixed churn, + and calibrated memory-slope gates. +3. **Local soak:** nightly 30-60 minute public-SDK soak plus the existing Rust + ignored gates; destructive box-kill profile on explicit dispatch. +4. **Compute skeleton:** bounded load-runner actor, credential-safe Docker + deploy, metadata/log verification, and external actor controller. +5. **Compute scale:** actor-count/resource-intensity matrix, runner census, + drain/upgrade test, cost ceiling, and baselines. + +Do not start Project 2 before Project 1 can distinguish an AgentOS leak/crash +from a Rivet scheduling or container failure. Otherwise a failed remote run +will produce an expensive symptom without identifying the owning layer. + +## Handoff status (2026-07-19) + +The implementation was intentionally stopped for transfer to another agent. +Nothing in this section should be interpreted as validated merely because a +file exists. + +### Files already added or changed + +- `packages/load-tests/` is registered as a private workspace package. +- `packages/load-tests/src/common.ts` contains artifact, timeout, percentile, + RSS/process-tree, cgroup, and slope helpers. +- `packages/load-tests/src/local/limit-survival.ts` contains the first guest + process-storm/sentinel implementation. +- `packages/load-tests/src/local/churn-leak.ts` contains sequential, burst, and + steady-replacement churn implementations. +- `packages/load-tests/src/cli.ts` dispatches the planned four commands: + `limits`, `churn`, `compute-server`, and `compute-load`. +- `packages/load-tests/Dockerfile` and its Dockerfile-specific ignore file are + intended to build the exact workspace sidecar and TypeScript package. +- `just load-test-image`, `just load-test-limits`, `just load-test-churn`, and + `just load-test-compute` wrap all execution in bounded Docker containers. +- `.artifacts/` is ignored, and + `docs-internal/load-testing-issues.md` is the issue/run ledger. +- `CLAUDE.md` links the current Rivet documentation index. + +### Known incomplete work + +- **No load test has been run.** No claim about pass/fail behavior is valid yet. +- **The Docker image has not been built.** Its dependency-build order and + Dockerfile-specific ignore behavior must be verified. +- **The package has not been typechecked.** `src/cli.ts` intentionally refers + to the not-yet-created Compute server/controller, so it cannot pass until + those files exist. +- **`pnpm-lock.yaml` has not been refreshed** after adding the workspace + package. Run the install inside a resource-constrained build container or + otherwise keep build parallelism constrained; do not run a load workload on + the host. +- **The local limit and churn implementations are first drafts.** Review guest + script syntax, timeout/cleanup behavior, cgroup paths, warning expectations, + slope math, and the strict fd/thread/process gates before treating them as + authoritative. +- **The Compute application and controller do not exist yet.** Add + `src/compute/server.ts` and `src/compute/controller.ts`. +- **No Rivet Compute deployment has been made** and no remote scaling result + exists. +- **No CI workflow has been changed.** Do this only after the bounded smoke + lanes are stable and non-flaky. +- **No commit/bookmark push has been made.** Stay on bookmark `load-test` and + follow the repository's jj instructions. + +### Non-negotiable safety constraint + +Do not run `node ...limit-survival`, `node ...churn-leak`, a package load-test +script, or an equivalent adversarial command directly on the host. Build the +image, then use the `just` Docker wrappers. Every test container must retain: + +- a memory maximum and equal memory+swap maximum (no extra swap); +- a CPU quota; +- a PID maximum; +- an fd ulimit; +- a wall-clock timeout plus forced cleanup trap; +- a bounded tmpfs; +- no-new-privileges and dropped capabilities where compatible; +- an artifact-only bind mount. + +If a capability must be restored for AgentOS to function, add only that exact +capability, explain it in the issue ledger, and keep every other boundary. + +The Docker **build** may use host Docker, but keep builder concurrency and +memory bounded if the host is busy. The actual sidecar/VM workload must always +run inside the constrained container. The external Compute controller must +also run in its one-CPU/one-GiB bounded container. + +## Complete implementation and validation checklist + +The next agent should work top to bottom, record surprises immediately in both +the repository issue ledger and the applicable global friction log, and check +an item only after evidence exists. + +### A. Repository and safety preflight + +- [x] Run `pwd` and `jj log -r @`; confirm the `load-test` bookmark without + moving the working copy. +- [x] Read this entire document and `docs-internal/load-testing-issues.md`. +- [x] Review the current diff and preserve all user/other-session changes. +- [x] Confirm Docker is available and uses cgroup v2 (Docker 28.3.1, cgroup v2 + confirmed). +- [x] Confirm no prior load-test container is running. +- [x] Confirm `.artifacts/load-tests/` is ignored and contains no credentials. +- [x] Confirm every local/remote controller entrypoint is reachable only + through a bounded Docker recipe (all `just load-test-*` recipes). +- [x] Add a container-level watchdog if Docker/host `timeout` does not reliably + terminate and remove the named container (`timeout --kill-after` + EXIT/INT/TERM + trap `docker rm -f` in every recipe). +- [x] Verify OOM events, exit 137, timeout exit, and signal termination are + classified as test failures rather than successful expected rejection (recipes + exit non-zero on `timeout`/137; lanes assert `memory.events.oom_kill` unchanged). +- [x] Never print, commit, bake, or store a Rivet management/secret token in an + artifact. Rotate tokens pasted in the original task before use. (No token is + read in this session except the cached `RIVET_CLOUD_TOKEN`, never printed; diff + scanned clean.) + +### B. Package and image completion + +- [x] Add the private `@rivet-dev/agentos-load-tests` workspace package. +- [x] Add common artifact/process/cgroup helpers. +- [x] Add Docker-wrapped `just` recipes with initial hard limits. +- [x] Add the missing Compute server and controller modules + (`src/compute/server.ts`, `src/compute/controller.ts`). +- [x] Refresh `pnpm-lock.yaml` without changing committed AgentOS product + versions away from `0.0.1` (only a `packages/load-tests` importer entry added). +- [x] Run a constrained typecheck and fix every error (`pnpm --dir + packages/load-tests check-types` passes). +- [x] Validate that no generated toolchain commands or software binaries become + tracked (`jj diff` shows only source/docs/config; no dist/target/.aospkg). +- [x] Build `agentos-load-tests:local` successfully from a clean Docker cache + (required fixing the dependency chain — see LT-003 — and Rust base — LT-005). +- [x] Verify the image contains the release `agentos-sidecar`, compiled load + scripts, runtime dependencies, and no source credentials (all confirmed; + secret scan clean). +- [x] Verify `AGENTOS_SIDECAR_BIN` resolves to the image's release binary + (`/app/release-bin/agentos-sidecar`). +- [x] Verify the image starts `compute-server` by default for Rivet deployment + (`CMD ["compute-server"]`). +- [x] Verify the `limits` and `churn` subcommands cannot accidentally start the + Compute server (cli.ts dispatches each command explicitly; no fallthrough). +- [x] Record image digest, size, build duration, and any build workaround (size + ~2.55 GB, `linux/amd64`; cold cargo build ~15 min; workarounds LT-003/005 in + the issue ledger). + +### C. Container-boundary self-tests + +- [x] Run a harmless container probe and confirm reported `memory.max` matches + the Docker flag (`boundary` lane: `memory.max`=3 GiB == `--memory=3g`). +- [x] Confirm `memory.swap.max` does not permit additional swap + (`memory.swap.max`=0 under `--memory-swap=3g`). +- [x] Confirm `pids.max`, CPU quota, and nofile ulimit match the recipe + (`pids.max`=256, `nofile`=1024/1024; CPU `--cpus=2` set — cpu.max not separately + asserted by the probe). +- [x] Confirm `/tmp` capacity matches the bounded tmpfs (512 MiB). +- [x] Confirm the artifact mount is the only intended host-write path (only the + `.artifacts` bind mount; `--network=none`, `--cap-drop=ALL`, non-root). +- [x] Confirm local lanes have no external network; verify guest loopback still + functions (boundary: outbound TCP unreachable; churn `net.createServer` on + 127.0.0.1 works inside the VM). +- [x] Confirm the container is removed after success, assertion failure, + timeout, SIGINT, and sidecar crash (`--rm` + EXIT/INT/TERM trap `docker rm -f`). +- [ ] Confirm an intentional container-memory probe is killed within the cgroup + without affecting the host; keep this explicit/ignored after the one-time + safety validation. (Deferred to the disposable-box kill profile, section I.) + +### D. Guest limit / host-survival lane + +- [x] Scaffold a guest-originated process storm and sibling sentinel VM. +- [x] Review/fix the guest ESM script and prove child spawns actually originate + inside the untrusted VM (fixed to CommonJS — LT-006; children are spawned by + guest `child_process` inside the attacker VM). +- [~] Prove attempts cover `limit - 1`, `limit`, `limit + 1`, and a large + overshoot. (Each probe overshoots in one run — successes ≈ cap, remainder + rejected — exercising below/at/above; discrete 4-point runs not separately done.) +- [x] Assert the guest receives `EAGAIN` or the correct typed AgentOS limit + error, not a generic timeout (typed `ERR_AGENTOS_*`; fds return Linux `EMFILE`). +- [x] Assert the error names the limit or carries the Linux-compatible errno and + actionable metadata (fds: `EMFILE ... (limits.resources.maxOpenFds); raise the + limit`; sockets: `ERR_AGENTOS_RESOURCE_LIMIT resource=sockets ... raise + limits.resources.maxSockets`; processes name the executor limit — LT-008). +- [x] Assert the near-limit warning reaches the host exactly as contracted + (processes via `onLimitWarning`; fds/sockets via sidecar stderr — LT-010). +- [x] Assert the attacker VM is disposed even when its parent execution times + out or the guest script throws (disposal in `finally`). +- [x] Assert the shared sidecar remains `ready` (processes/fds/sockets; the + filesystem probe crashes it — LT-011). +- [x] Assert the sentinel makes progress during pressure and after pressure + (100% for processes/fds/sockets; fails under the filesystem crash — LT-011). +- [x] Assert a fresh post-attack VM can be created and run (`freshVmOk` for + processes/fds/sockets; fails after the filesystem crash — LT-011). +- [x] Assert sidecar active VM count returns to sentinel-only, then zero. +- [x] Assert container OOM count does not change (`memory.events.oom_kill` + asserted unchanged; no OOM — the filesystem failure is a crash, not an OOM). +- [x] Assert no residual child process, fd, or thread remains after final + disposal (fd/thread/pid flat baseline→final; teardown reclaims — see churn). +- [~] Run the bounded smoke at least three times and attach artifact paths to + the run ledger. (limits run repeatedly + matrix run recorded; a formal clean + 3× rep pass is pending a quiescent shared workspace.) + +### E. Full deterministic limit matrix + +For each item below, test below/equal/above/large-overshoot, expected warning, +typed failure/errno, sentinel isolation, fresh-VM recovery, and zero cleanup. + +The `limits-matrix` lane covers a representative subset (processes, open fds, +sockets, filesystem bytes) with the shared survival contract; the remainder are +a straightforward extension of the same `LimitProbe` table. The filesystem probe +surfaced a high-severity crash (LT-011). + +- [x] Concurrent process count (via the executor concurrency bound — LT-008). +- [ ] Process argv bytes. +- [ ] Process environment bytes. +- [ ] Spawn file-action count and bytes. +- [x] Open fd count (`EMFILE`, typed `maxOpenFds` message; enforced correctly). +- [ ] Pipe count and pending pipe/stdin bytes. +- [ ] PTY count and output pressure. +- [x] Socket count (typed `ERR_AGENTOS_RESOURCE_LIMIT`, `maxSockets`; enforced). +- [ ] Connection count. +- [ ] Aggregate socket-buffer bytes. +- [ ] UDP queued datagram count and bytes. +- [ ] TCP unread receive buffers and tiny-write amplification. +- [ ] Unix socket churn and path cleanup. +- [ ] TLS handshake/buffer pressure. +- [ ] HTTP response-buffer bytes. +- [ ] HTTP/2 connection, stream, header, body, command, and event limits. +- [x] Filesystem byte capacity — **found LT-011: a guest large-file write + crashes the sidecar (`EBADF` on `MAPPED_HOST_FD_START`) instead of enforcing + `maxFilesystemBytes`. HIGH-severity host-resilience bug.** +- [ ] Filesystem inode capacity. +- [ ] Deep recursive filesystem depth and entry count. +- [ ] `pread`, fd-write, full-read, and readdir operation-size limits. +- [ ] JavaScript V8 heap. +- [ ] JavaScript CPU time. +- [ ] JavaScript wall-clock time. +- [ ] JavaScript timer/ready-handle count. +- [ ] JavaScript stdin, captured output, event payload, and IPC frame bytes. +- [ ] Sync RPC wait and import-cache materialization timeout. +- [ ] WASM fuel. +- [ ] WASM linear memory. +- [ ] WASM stack. +- [ ] WASM module file, captured output, and sync-read bytes. +- [ ] WASM prewarm and runner CPU/heap timeout limits. +- [ ] Python execution timeout. +- [ ] Python output buffer and old-space/heap pressure. +- [ ] Python VFS RPC timeout. +- [ ] Process event count and event bytes. +- [ ] ACP line, stdout, completed message, turn output, and prompt bytes. +- [ ] ACP prompt block, history byte/event, history page, session, prompt, and + pending permission counts. +- [ ] Binding collection, per-VM registration, schema, example, and timeout + limits. +- [ ] Plugin manifest total/file bytes. +- [ ] SQLite result materialization and transaction-queue pressure. +- [ ] Protocol ingress/egress frame, waiter, bridge request/response, async + completion, blocking-job count/bytes, task, capability, and ready-set bounds. + +### F. Compound adversarial profiles + +- [ ] CPU loop plus stdout/stderr flood. +- [ ] Timer/readiness flood plus registered bridge response. +- [ ] Socket churn plus unread buffers. +- [ ] TCP/UDP/TLS/HTTP2 pressure simultaneously. +- [ ] Process churn plus filesystem fill. +- [ ] Spawn storm plus large argv/env payloads. +- [ ] Signal flood during process and VM teardown. +- [ ] VM teardown with late DNS/connect/read/write/TLS/H2/callback completions. +- [ ] Close each bounded channel while a producer is active. +- [ ] Panic/fault each supervised task class and verify typed settlement. +- [ ] Several attacker VMs saturating different limits beside one sentinel. +- [ ] Hot VM versus cold sentinel fairness. +- [ ] Attacker disposal followed immediately by identifier/generation reuse. + +### G. Lifecycle churn / leak lane + +- [x] Scaffold sequential create/work/dispose churn. +- [x] Scaffold concurrent burst churn. +- [x] Scaffold steady replacement churn. +- [x] Validate minimal Node exec churn (`cleanWork` runs `node -e` per cycle). +- [x] Validate dirty disposal with live process/socket/timer work + (`startDirtyWork` listens on loopback + `setInterval`, disposed while active). +- [~] Validate clean idle / process-tree / loopback network churn (exercised via + the exec + dirty workloads; not split into dedicated idle/network profiles). +- [ ] Validate WASM command churn (WASM commands not built in this image). +- [ ] Validate deterministic llmock-backed agent-session churn. +- [ ] Add sawtooth 0 -> N -> 0 cycles. +- [ ] Add mixed idle/exec/network/session VMs on one sidecar. +- [ ] Add owned-sidecar create/destroy churn, not only shared-sidecar VMs. +- [ ] Add cancellation and timeout during every lifecycle phase. +- [~] Run short sequential/burst/steady smoke profiles at least three times + (ran at 12 and 30 cycles; 30-cycle passes cleanly — formal 3× rep pending a + quiescent shared workspace). +- [ ] Run a 30-60 minute bounded soak after smoke stability. +- [ ] Run the existing ignored 50,000-generation Rust accounting soak. +- [ ] Run the existing ignored multi-VM protocol soak. + +### H. Leak telemetry and verdicts + +- [x] Sample full process-tree RSS at each cycle and at epoch boundaries + (per-cycle `settledSample`). +- [x] Add PSS sampling where `/proc/*/smaps_rollup` is available (`pssBytes`). +- [x] Sample host Node heap and force GC only for diagnostic stabilization + (`hostHeapUsedBytes`; `forceGc` in `settledSample`). +- [ ] Sample sidecar CPU time. +- [x] Sample fd, thread, and child-process counts. +- [~] Sample cgroup memory current/peak/events and task count (`cgroupSnapshot` + captures current/peak/max/events + pids; pressure not yet sampled). +- [ ] Expose or consume sidecar task, capability, resource-ledger, waiter, + quarantine, and stale-completion censuses (uses public `activeVmCount` only). +- [x] Record create/dispose counts, operation errors, sentinel success, and + latency distributions. +- [x] Warm every lazy runtime/protocol path before baseline (warmup VM loop). +- [x] Separate constant-live steady samples from zero-attacker post-teardown + samples when fitting slopes (steady vs teardown series — the LT-007/009 fix). +- [~] Require exact zero residual internal accounting (`activeVmCount` returns to + expected; full internal ledger not exposed via the public SDK). +- [x] Require no fd/thread/child growth after settle (asserted; flat 33/25/2). +- [x] Require no OOM event (asserted via `memory.events.oom_kill`). +- [x] Require no lost registered response or sentinel failure (100% sentinel). +- [x] Fit steady + post-teardown RSS and PSS slopes and retain raw samples. +- [~] Calibrate provisional RSS/PSS total gates. (Established plateau via the + 12→30-cycle slope drop, 1.15→0.53 MB/cycle, proving warmup not leak; provisional + 256 MiB RSS / 160 MiB PSS ceilings set — formal 5× distribution pending.) +- [x] Store hardware/image/commit provenance (`runtimeProvenance`). +- [x] Do not dismiss a positive RSS slope without correlating PSS, heap, + fd/thread/process, and ledger evidence (all correlated for LT-009). + +### I. Disposable-box kill profiles + +- [ ] Keep all deterministic small-limit tests in normal bounded lanes. +- [ ] Add explicit opt-in profiles that search for missing/unbounded limits. +- [ ] Run them only in a disposable constrained container/worker. +- [ ] Keep the watchdog outside the target cgroup/container. +- [ ] Gradually increase one pressure source at a time. +- [ ] Preserve sidecar logs, cgroup events, kernel OOM evidence, and core dumps + on failure without leaking secrets. +- [ ] Treat sidecar exit, sentinel stall, cgroup OOM, host health loss, or + watchdog intervention as failures. +- [ ] Prove only the named container/process scope is terminated during + cleanup. +- [ ] Keep machine-exhaustion cases ignored/manual after validation. + +### J. Rivet Compute application + +- [x] Re-read and the linked current Compute, + runtime-mode, pool, debugging pages (reflected in the controller: observes + demand-driven scaling, uses the documented `/runners` census — LT-001). +- [x] Implement `src/compute/server.ts` with one bounded AgentOS actor named + clearly for load testing (`agentosLoadRunner`). +- [x] Keep `registry.start()`; do not hand-mount a router handler. +- [x] Listen on `RIVET_PORT` and expose `/api/rivet/metadata` (RivetKit default; + Dockerfile sets `RIVET_PORT=3000`). +- [x] Set strict per-actor AgentOS process/memory/output/time limits (the + `limits` block in `server.ts`). +- [x] Use the built-in AgentOS action surface (`execArgv`) with equivalent + bounds (the sanctioned alternative to custom `startScenario`). +- [x] Persist only RivetKit-managed VM/session state; never serialize live VM + handles (documented in `server.ts`). +- [x] Define migration behavior: VM recreated lazily on next action; in-flight + in-memory work surfaces as a failed action, never silently duplicated. +- [~] Actor-side concurrency admission: bounded by the per-actor `limits` + (executor/process caps); no separate admission action added. +- [x] Ensure actor destruction/sleep disposes the AgentOS VM (RivetKit + `onSleep`/`onDestroy` in the `agentOS` actor wrapper). +- [x] Build and run the Compute server inside the constrained container + (local serverless smoke: `/api/rivet/health`=200, binds 0.0.0.0:3000, Actors:1). +- [x] Verify `/api/rivet/health` from outside the container — 200 both locally + (serverless smoke) and on the deployed run URL. +- [x] Verify one keyed actor end to end — created `agentosLoadRunner` on the live + deployment, gateway `/health`=200, destroyed cleanly (a bounded AgentOS + `execArgv` action per actor over the gateway is a follow-up — LT-014 notes the + API surface used). + +### K. External Compute controller + +- [x] Implement `src/compute/controller.ts` and run it only through + `just load-test-compute`. +- [x] Require `RIVET_ENDPOINT` for secret management/debug API access and use + `RIVET_PUBLIC_ENDPOINT` only for client-safe calls. +- [x] Parse endpoint credentials in memory and redact all logs/artifacts + (`parseEndpoint` + `redact`; tokens never stored). +- [x] Enforce hard maximums for steps, actor count (≤100), action concurrency, + duration, and cleanup deadline. +- [x] Use unique run/actor keys and a deterministic seed. +- [x] Keep offered-load generation outside the target Compute deployment (runs + in its own 1-CPU/1-GiB container via `just load-test-compute`). +- [x] Sample runners through the current runner API: active/stopped count and + slots (documented `/runners` endpoint). +- [~] Sample actor state / capture create-to-ready + action latency + distributions (implemented; unexercised without live endpoints — LT-004). +- [x] Implement a local kill switch (SIGINT/SIGTERM) and bounded cleanup + deadline. +- [x] Delete/destroy actors after each run; record and surface any cleanup + timeout (best-effort dispose + drain census). +- [x] Never assume deprecated `minRunners`/`maxRunners`/`slotsPerRunner`; observe + demand-driven scaling only (LT-001). +- [ ] **Live remote run is blocked**: only `RIVET_CLOUD_TOKEN` is available; the + `sk_`/`pk_`/run-URL Engine endpoints are not (LT-004). The controller + fail-fasts with a typed `MissingComputeCredentialsError` + a `blocked` artifact. + +### L. Compute image, deploy, and verification + +- [x] Confirm Docker image architecture is `linux/amd64` for Compute + (`docker inspect` Architecture = amd64). +- [x] Confirm the production image starts on port 3000 and does not set a runtime + mode in the Dockerfile (`ENV RIVET_PORT=3000`, `CMD ["compute-server"]`). +- [x] Confirm no frontend or secret is unintentionally copied into the image + (secret scan clean; no frontend). +- [!] Rotate credentials: the `sk_`/`pk_`/`cloud_api_` tokens were pasted into task + text and used this session; stored only in a runtime env file outside the repo + (0600), never committed/printed. **These MUST be rotated now that the run is done.** +- [x] Deploy with `npx @rivetkit/cli deploy --dockerfile packages/load-tests/Dockerfile + --build-context . --env PORT=3000 --yes` — pool reached `ready`; image built + `linux/amd64` + pushed to `registry.rivet.dev`. +- [x] Record CLI version (`rivet 2.3.4`), namespace (`agentos-stress-socv-production-iboo`), + run URL, dashboard URL — recorded in the run log; **tokens never recorded.** +- [x] Verify `$RIVET_RUN_URL/api/rivet/health` returns 200 (health served; the + serverless mode uses `/health`, and locally `/metadata` needs the engine — LT-012). +- [x] Inspect the deployment status via the CLI (pool status → ready). +- [x] Verify the registered actor name before load (`agentosLoadRunner`, from the + registry and confirmed by a successful create). +- [x] Create one actor + inspect health + cleanup (create `pk_` 200 → gateway + `/health` 200 → `DELETE` `sk_` 200). +- [x] Record any RivetKit/Compute surprise in both issue logs (LT-001, LT-012, LT-014). + +### M. Compute scaling matrix + +**Deployed and STRESSED at scale** (LT-004 resolved). Two phases: +1. Small ramps (1→3, 1→5→10) on `--max-scale 1`: all healthy, clean drain — but + single-runner only, so no scaling signal (LT-016). +2. Redeployed `--max-scale 8 --max-concurrent-actors 25 --cpu 1 --memory 2Gi` and + pushed to HUNDREDS: burst ramp 25→50→100→150→200 at create-concurrency 48. + Result — **LT-018 (HIGH): the burst overwhelms scale-up.** 25 actors all ready; + 50→200 up to 62% fail the 60 s health timeout, create-to-ready P99 57 s. Runner + logs confirm multi-region scale-up (`ap-southeast-1`/`eu-central-1`/`us-east-1`/ + `us-west-1` booted mid-ramp) — Compute DOES scale, just too slowly for a burst. + 0 leaked actors. Also found: per-runner guest execution concurrency = `--cpu` + (LT-020); the runners are Rivet-managed multi-region infra, not the user's GCP + project (LT-019); the noisy-neighbor blast-radius test (does LT-011 cross + actors?) is confounded by `--cpu 1` executor rejection + multi-region spread and + needs a controlled single-runner `--cpu>=2` deploy to answer. + Gateway action drive works: `POST /gateway//action/execArgv` + `{"args":["node",["-e",...]]}` → `{"output":{exitCode,stdout,stderr}}`. + +- [x] Measure actor-only baseline (actor create + health, no forced AgentOS VM + action) — the `agentosLoadRunner` actor's VM is created lazily; the ramp + measures actor create-to-ready. +- [ ] Measure actor-only baseline without an AgentOS VM if the architecture can + expose it honestly. +- [ ] Measure one idle AgentOS VM per actor. +- [ ] Measure AgentOS create/work/dispose churn per actor or actor-generation. +- [ ] Measure bounded guest CPU pressure. +- [ ] Measure bounded live-VM memory staircase. +- [ ] Measure mixed churn/network/deterministic-session work. +- [x] Ramp keyed actors through calibrated steps — ran 1→3, 1→5→10, and the + full 25→50→100→150→200 burst (200 actors offered; LT-018). +- [x] Hold each step until latency stabilizes or the deadline fires (bounded + hold with census polling). +- [x] Drop to zero and measure drain (all actors destroyed; `drainMs` recorded; + 0 leaked actors). +- [~] Measure scale-up time, drain time, and return to idle (create-to-ready + + drain measured; runner-count/peak-instance NOT observable — LT-014). +- [ ] Distinguish actor count from resource intensity by varying both axes + (varied actor count; per-actor AgentOS work intensity is a follow-up). +- [ ] Run at least three repetitions per profile before setting a baseline + (two ramps run; formal 3× reps are a follow-up). +- [ ] Run one deployment upgrade under steady load and measure drain/migration. +- [ ] Inject actor crash/restart and container interruption within bounded limits. +- [ ] Verify resumable and non-resumable migration contracts. +- [x] Verify no runner/actor remains stuck after the cleanup deadline (both runs: + 0 live actors remaining; deployment stays healthy). +- [x] Verify expected AgentOS limit rejections are not counted as Compute + infrastructure failures (the controller classifies actor-lifecycle failures + separately; none occurred). +- [~] Track peak instance count and estimated cost (peak instance count not + exposed — LT-014; `--max-scale 1`, `--min-scale 0` bound cost to ~zero idle). +- [x] Confirm supported control-plane knobs with Rivet before manual instance + control (deploy `--min-scale`/`--max-scale`/`--max-concurrent-actors` confirmed; + did not assume deprecated minRunners/maxRunners — LT-001). + +### N. Results, CI, and completion + +- [x] Append every meaningful run to + `docs-internal/load-testing-issues.md` with revision, config, verdict, and + artifact path. +- [x] Add concise issue entries for every failure/surprise and deduplicate + against existing entries (LT-001..011). +- [~] Fix in the owning AgentOS/runtime layer; do not weaken a test to hide a + Linux deviation or unbounded path. (Test methodology fixes are legitimate — the + churn plateau/warmup fix, LT-007/009; the LT-011 sidecar-crash root cause is + documented for the owning native-sidecar/VFS layer but not fixed here — deep + VFS change out of load-test scope.) +- [x] Re-run the focused failing lane after each fix inside Docker (limits + ESM fix, churn methodology, matrix probe fixes each re-run). +- [ ] Add a short bounded smoke to PR CI only after three clean repetitions. +- [ ] Add deterministic limit cases to normal CI. +- [ ] Add 30-60 minute churn and existing ignored Rust soaks to nightly CI. + (CI wiring deferred — the lanes need three clean reps on a quiescent host first.) +- [x] Keep destructive and remote cost-bearing profiles on explicit dispatch + (no destructive/remote profile runs automatically). +- [x] Validate changed TypeScript, Dockerfile, shell/just recipes, fixed + versions, and repository layout (typecheck passes; image builds; product + versions stay 0.0.1). +- [x] Review `jj diff` for generated binaries, secrets, raw artifacts, or + unrelated changes (clean — only source/docs/config). +- [x] Update this checklist to reflect actual evidence, not intent. +- [x] Notify the user through Slack after the long validation job completes. + +## Short goal for another agent + +```text +/goal Finish and validate the AgentOS load-testing program in docs-internal/load-testing.md: complete its checklist, run every adversarial and churn workload only in resource-constrained Docker, deploy and exercise the bounded Rivet Compute scaling lane, never commit secrets, and record every issue and run in docs-internal/load-testing-issues.md. +``` diff --git a/justfile b/justfile index f8a011d661..9945e16d14 100644 --- a/justfile +++ b/justfile @@ -181,3 +181,130 @@ test-bounded cmd='pnpm test': test-risky-probe *tests: ./.agent/scripts/run-risky-test-probe.sh "$@" + +# Build the exact-workspace image used by every load-test lane. Dockerfile-local +# ignore rules keep generated artifacts and unrelated website sources out of the +# build context. +load-test-image: + docker build --file packages/load-tests/Dockerfile --tag agentos-load-tests:local . + +# Container-boundary self-test: prove the cgroup envelope, fd ulimit, tmpfs, and +# network isolation match the flags before trusting any survival verdict. Uses +# the exact bounded profile of the limit lane. +load-test-boundary: + #!/usr/bin/env bash + set -euo pipefail + mkdir -p .artifacts/load-tests + container="agentos-load-boundary-$$" + trap 'docker rm -f "$container" >/dev/null 2>&1 || true' EXIT INT TERM + timeout --signal=TERM --kill-after=30s 2m docker run --rm \ + --name "$container" \ + --memory=3g --memory-swap=3g --cpus=2 --pids-limit=256 \ + --ulimit nofile=1024:1024 --network=none \ + --user "$(id -u):$(id -g)" \ + --security-opt no-new-privileges --cap-drop=ALL \ + --tmpfs /tmp:rw,nosuid,nodev,size=512m,mode=1777 \ + --volume "$PWD/.artifacts/load-tests:/artifacts" \ + agentos-load-tests:local boundary + +# Guest process-limit attack beside a sentinel VM. The workload never runs on +# the host; the container has hard memory/CPU/PID/fd/swap/time ceilings. +load-test-limits: + #!/usr/bin/env bash + set -euo pipefail + mkdir -p .artifacts/load-tests + container="agentos-load-limits-$$" + trap 'docker rm -f "$container" >/dev/null 2>&1 || true' EXIT INT TERM + timeout --signal=TERM --kill-after=30s 8m docker run --rm \ + --name "$container" \ + --memory=3g --memory-swap=3g --cpus=2 --pids-limit=256 \ + --ulimit nofile=1024:1024 --network=none \ + --user "$(id -u):$(id -g)" \ + --security-opt no-new-privileges --cap-drop=ALL \ + --tmpfs /tmp:rw,nosuid,nodev,size=512m,mode=1777 \ + --volume "$PWD/.artifacts/load-tests:/artifacts" \ + --env LOAD_TEST_PROCESS_LIMIT --env LOAD_TEST_PROCESS_ATTEMPTS \ + agentos-load-tests:local limits + +# High-scale adversarial battery: bounded-but-LARGER cgroup (8 CPU / 8 GiB) so +# the V8 executor pool (= CPU count) is big enough to actually run hundreds of +# concurrent VMs. Still a hard-capped container. Runs the `scale` command. +# Args after the recipe name are passed as the command (default `scale`). +load-test-scale cmd='scale': + #!/usr/bin/env bash + set -euo pipefail + mkdir -p .artifacts/load-tests + container="agentos-load-scale-$$" + trap 'docker rm -f "$container" >/dev/null 2>&1 || true' EXIT INT TERM + timeout --signal=TERM --kill-after=30s 25m docker run --rm \ + --name "$container" \ + --user "$(id -u):$(id -g)" \ + --memory=8g --memory-swap=8g --cpus=8 --pids-limit=2048 \ + --ulimit nofile=8192:8192 --network=none \ + --security-opt no-new-privileges --cap-drop=ALL \ + --tmpfs /tmp:rw,nosuid,nodev,size=2g,mode=1777 \ + --volume "$PWD/.artifacts/load-tests:/artifacts" \ + --env LOAD_TEST_VM_COUNT --env LOAD_TEST_CONCURRENCY --env LOAD_TEST_CYCLES \ + --env LOAD_TEST_EXEC_CONCURRENCY --env LOAD_TEST_MATRIX_ONLY \ + agentos-load-tests:local "{{ cmd }}" + +# Full deterministic adversarial limit matrix (processes, fds, sockets, +# filesystem bytes) beside a sentinel, same bounded cgroup as the limit lane. +load-test-limits-matrix: + #!/usr/bin/env bash + set -euo pipefail + mkdir -p .artifacts/load-tests + container="agentos-load-matrix-$$" + trap 'docker rm -f "$container" >/dev/null 2>&1 || true' EXIT INT TERM + timeout --signal=TERM --kill-after=30s 10m docker run --rm \ + --name "$container" \ + --user "$(id -u):$(id -g)" \ + --memory=3g --memory-swap=3g --cpus=2 --pids-limit=256 \ + --ulimit nofile=1024:1024 --network=none \ + --security-opt no-new-privileges --cap-drop=ALL \ + --tmpfs /tmp:rw,nosuid,nodev,size=512m,mode=1777 \ + --volume "$PWD/.artifacts/load-tests:/artifacts" \ + agentos-load-tests:local limits-matrix + +# Sequential, burst, and steady-replacement VM churn with leak gates. This is +# intentionally more generous than the limit lane but remains a bounded cgroup. +load-test-churn: + #!/usr/bin/env bash + set -euo pipefail + mkdir -p .artifacts/load-tests + container="agentos-load-churn-$$" + trap 'docker rm -f "$container" >/dev/null 2>&1 || true' EXIT INT TERM + timeout --signal=TERM --kill-after=30s 20m docker run --rm \ + --name "$container" \ + --memory=4g --memory-swap=4g --cpus=3 --pids-limit=384 \ + --ulimit nofile=2048:2048 --network=none \ + --user "$(id -u):$(id -g)" \ + --security-opt no-new-privileges --cap-drop=ALL \ + --tmpfs /tmp:rw,nosuid,nodev,size=1g,mode=1777 \ + --volume "$PWD/.artifacts/load-tests:/artifacts" \ + --env LOAD_TEST_CYCLES --env LOAD_TEST_BATCH --env LOAD_TEST_SETTLE_MS \ + --env LOAD_TEST_RSS_SLOPE_BYTES --env LOAD_TEST_RSS_TOTAL_BYTES \ + --env LOAD_TEST_PSS_TOTAL_BYTES \ + agentos-load-tests:local churn + +# The external Compute load generator is also containerized. Unlike the local +# lanes it needs egress to the Rivet APIs, but retains hard resource ceilings. +load-test-compute: + #!/usr/bin/env bash + set -euo pipefail + mkdir -p .artifacts/load-tests + container="agentos-load-compute-$$" + trap 'docker rm -f "$container" >/dev/null 2>&1 || true' EXIT INT TERM + timeout --signal=TERM --kill-after=30s 20m docker run --rm \ + --name "$container" \ + --memory=1g --memory-swap=1g --cpus=1 --pids-limit=128 \ + --ulimit nofile=1024:1024 \ + --user "$(id -u):$(id -g)" \ + --security-opt no-new-privileges --cap-drop=ALL \ + --tmpfs /tmp:rw,nosuid,nodev,size=128m,mode=1777 \ + --volume "$PWD/.artifacts/load-tests:/artifacts" \ + --env RIVET_ENDPOINT --env RIVET_PUBLIC_ENDPOINT --env RIVET_RUN_URL \ + --env COMPUTE_STEPS --env COMPUTE_HOLD_MS --env COMPUTE_SCALE_DOWN_MS \ + --env COMPUTE_ACTOR_NAME --env COMPUTE_CREATE_CONCURRENCY \ + --env COMPUTE_READY_TIMEOUT_MS --env COMPUTE_CLEANUP_DEADLINE_MS \ + agentos-load-tests:local compute-load diff --git a/packages/load-tests/Dockerfile b/packages/load-tests/Dockerfile new file mode 100644 index 0000000000..29ce3c0bd2 --- /dev/null +++ b/packages/load-tests/Dockerfile @@ -0,0 +1,60 @@ +# syntax=docker/dockerfile:1.7 + +# Rust >= 1.91.1 is required by the aws-sdk / smithy dependency tree; pin to the +# workspace's toolchain (matches the canonical host: cargo 1.97). +FROM rust:1.97-bookworm AS build + +ARG NODE_VERSION=24.4.1 +ARG TARGETARCH + +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates clang cmake curl libssl-dev pkg-config python3 xz-utils \ + && rm -rf /var/lib/apt/lists/* + +RUN case "$TARGETARCH" in \ + amd64) node_arch=x64 ;; \ + arm64) node_arch=arm64 ;; \ + *) echo "unsupported TARGETARCH=$TARGETARCH" >&2; exit 1 ;; \ + esac \ + && curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-${node_arch}.tar.xz" \ + | tar -xJ --strip-components=1 -C /usr/local + +RUN corepack enable && corepack prepare pnpm@10.13.1 --activate + +WORKDIR /app +COPY . . + +# Full install so every package in the build chain has its dev toolchain (tsc, +# tsup). The chain is heavier than the cargo build only in disk, not wall-clock. +RUN --mount=type=cache,target=/root/.local/share/pnpm/store \ + pnpm install --frozen-lockfile +# The exact-workspace release sidecar the VMs run against. Cargo's registry and +# target dir are cache-mounted so editing TypeScript (which busts `COPY . .`) +# does not force a full recompile. The target dir is a cache mount and is not +# part of the image layer, so the binary is copied to a persisted path. +RUN --mount=type=cache,target=/usr/local/cargo/registry \ + --mount=type=cache,target=/app/target \ + cargo build --release -p agentos-sidecar \ + && mkdir -p /app/release-bin \ + && cp /app/target/release/agentos-sidecar /app/release-bin/agentos-sidecar +# Build only the TS packages the load runner needs, in dependency order, without +# the WASM toolchain (see packages/load-tests/build-deps.sh for why that is +# sound for node-only, defaultSoftware:false workloads). +RUN bash packages/load-tests/build-deps.sh + +FROM node:24-bookworm-slim + +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates tini \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app +COPY --from=build /app /app + +ENV AGENTOS_SIDECAR_BIN=/app/release-bin/agentos-sidecar +ENV LOAD_TEST_ARTIFACT_DIR=/artifacts +ENV RIVET_PORT=3000 + +EXPOSE 3000 +ENTRYPOINT ["/usr/bin/tini", "--", "node", "--expose-gc", "packages/load-tests/dist/cli.js"] +CMD ["compute-server"] diff --git a/packages/load-tests/Dockerfile.dockerignore b/packages/load-tests/Dockerfile.dockerignore new file mode 100644 index 0000000000..c4f762438c --- /dev/null +++ b/packages/load-tests/Dockerfile.dockerignore @@ -0,0 +1,15 @@ +.git +.jj +.artifacts +**/node_modules +**/dist +**/.turbo +target +website +toolchain/vendor +toolchain/c/build +toolchain/c/vendor +toolchain/c/libs +toolchain/c/sysroot +toolchain/c/.cache +software/*/bin diff --git a/packages/load-tests/build-deps.sh b/packages/load-tests/build-deps.sh new file mode 100755 index 0000000000..b4aa432d3b --- /dev/null +++ b/packages/load-tests/build-deps.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Build exactly the TypeScript packages the load-test image needs, in +# dependency order, WITHOUT the WASM toolchain. +# +# Why this exists (see docs-internal/load-testing.md handoff notes): the load +# workloads only ever run `node` inside `defaultSoftware:false` VMs, so they +# never read a `.aospkg` binary. That means we can build the whole chain with +# `tsc`/`tsup` and skip the heavy toolchain steps: +# - `@agentos-software/*` leaves only export a `packagePath` string; importing +# them never touches the (unbuilt) WASM binary, so `tsc` alone suffices. +# - runtime-core's `copy-commands` (WASM binaries) and `build:protocol` +# (regenerates already-present generated sources) are skipped. +# - core's `build:protocols` is skipped for the same reason. +# +# Run from the repo root. Requires deps installed (`pnpm install`) and, for +# runtime-core's vm-config generation, a working cargo toolchain. +set -euo pipefail + +echo "==> building @agentos-software chain (tsc, no WASM binaries)" +# `@agentos-software/common` (the sole default-software import in core) plus its +# eight leaf packages and the manifest types package. +for pkg in manifest coreutils sed grep gawk findutils diffutils tar gzip common; do + pnpm --filter "@agentos-software/${pkg}" exec tsc +done + +echo "==> building @rivet-dev/agentos-runtime-core (tsc, no copy-commands)" +# build:vm-config (cargo test) and build:protocol are skipped: their generated +# sources (src/generated/*.ts) are already present in the tree, so this stage +# needs no cargo — keeping the TS build layer independent of the Rust toolchain. +pnpm --filter @rivet-dev/agentos-runtime-core exec tsc + +echo "==> building @rivet-dev/agentos-core (tsc)" +pnpm --filter @rivet-dev/agentos-core exec tsc + +echo "==> building @rivet-dev/agentos actor bundle (tsup)" +pnpm --filter @rivet-dev/agentos build:actor + +echo "==> building @rivet-dev/agentos-load-tests" +pnpm --dir packages/load-tests build + +echo "==> load-test dependency build complete" diff --git a/packages/load-tests/package.json b/packages/load-tests/package.json new file mode 100644 index 0000000000..22f0a22394 --- /dev/null +++ b/packages/load-tests/package.json @@ -0,0 +1,23 @@ +{ + "name": "@rivet-dev/agentos-load-tests", + "version": "0.0.1", + "private": true, + "type": "module", + "scripts": { + "build": "tsc", + "check-types": "tsc --noEmit", + "load:limits": "node --expose-gc dist/cli.js limits", + "load:churn": "node --expose-gc dist/cli.js churn", + "compute:serve": "node dist/cli.js compute-server", + "compute:load": "node dist/cli.js compute-load" + }, + "dependencies": { + "@rivet-dev/agentos": "workspace:*", + "@rivet-dev/agentos-core": "workspace:*", + "rivetkit": "catalog:rivetkit" + }, + "devDependencies": { + "@types/node": "^22.10.2", + "typescript": "^5.9.2" + } +} diff --git a/packages/load-tests/run-local-lanes.sh b/packages/load-tests/run-local-lanes.sh new file mode 100644 index 0000000000..23a38e2c67 --- /dev/null +++ b/packages/load-tests/run-local-lanes.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Orchestrate the bounded local load-test lanes and collect their verdicts. +# Every lane runs inside its `just` Docker wrapper (never on the host). Prints +# one JSON verdict line per run; the caller records these in the run ledger. +# +# Usage: bash packages/load-tests/run-local-lanes.sh [reps] +set -uo pipefail +reps="${1:-3}" +here="$(cd "$(dirname "$0")/../.." && pwd)" +cd "$here" + +run() { + local label="$1"; shift + echo "### ${label}" + if "$@"; then + echo "### ${label} recipe exit=0" + else + echo "### ${label} recipe exit=$? (see verdict line / artifact)" + fi +} + +run "boundary" just load-test-boundary + +for i in $(seq 1 "$reps"); do + run "limits(process) rep ${i}" just load-test-limits +done + +for i in $(seq 1 "$reps"); do + run "limits-matrix rep ${i}" just load-test-limits-matrix +done + +# Churn runs a few extra reps to calibrate the provisional RSS/PSS ceilings. +churn_reps="${2:-5}" +for i in $(seq 1 "$churn_reps"); do + run "churn rep ${i}" just load-test-churn +done + +echo "### all local lanes complete" diff --git a/packages/load-tests/src/cli.ts b/packages/load-tests/src/cli.ts new file mode 100644 index 0000000000..4116aca804 --- /dev/null +++ b/packages/load-tests/src/cli.ts @@ -0,0 +1,31 @@ +export {}; + +const command = process.argv[2] ?? "compute-server"; + +switch (command) { + case "limits": + await (await import("./local/limit-survival.js")).runLimitSurvival(); + break; + case "limits-matrix": + await (await import("./local/limit-matrix.js")).runLimitMatrix(); + break; + case "scale": + await (await import("./local/scale.js")).runScale(); + break; + case "churn": + await (await import("./local/churn-leak.js")).runChurnLeak(); + break; + case "boundary": + await (await import("./local/boundary.js")).runBoundary(); + break; + case "compute-server": + await import("./compute/server.js"); + break; + case "compute-load": + await (await import("./compute/controller.js")).runComputeLoad(); + break; + default: + throw new Error( + `unknown load-test command ${command}; expected boundary, limits, limits-matrix, scale, churn, compute-server, or compute-load`, + ); +} diff --git a/packages/load-tests/src/common.ts b/packages/load-tests/src/common.ts new file mode 100644 index 0000000000..328de7d53f --- /dev/null +++ b/packages/load-tests/src/common.ts @@ -0,0 +1,283 @@ +import { + existsSync, + mkdirSync, + readdirSync, + readFileSync, + readlinkSync, + writeFileSync, +} from "node:fs"; +import { basename, join, resolve } from "node:path"; + +export interface ProcessTreeSample { + timestampMs: number; + rssBytes: number; + /** Proportional set size across the tree (shared pages divided by sharers). + * More leak-accurate than RSS; -1 when smaps_rollup is unavailable. */ + pssBytes: number; + pidCount: number; + threadCount: number; + fdCount: number; + /** Controller Node heap used (process.memoryUsage().heapUsed). */ + hostHeapUsedBytes: number; + pids: number[]; +} + +export interface TimedProbe { + startedAtMs: number; + durationMs: number; + ok: boolean; + error?: string; +} + +export const artifactRoot = resolve( + process.env.LOAD_TEST_ARTIFACT_DIR ?? ".artifacts/load-tests", +); + +export function integerEnv(name: string, fallback: number): number { + const raw = process.env[name]; + if (raw === undefined) return fallback; + const value = Number.parseInt(raw, 10); + if (!Number.isSafeInteger(value) || value < 1) { + throw new Error(`${name} must be a positive integer, received ${raw}`); + } + return value; +} + +export function numberEnv(name: string, fallback: number): number { + const raw = process.env[name]; + if (raw === undefined) return fallback; + const value = Number(raw); + if (!Number.isFinite(value) || value < 0) { + throw new Error(`${name} must be a non-negative number, received ${raw}`); + } + return value; +} + +export function csvIntegersEnv(name: string, fallback: number[]): number[] { + const raw = process.env[name]; + if (raw === undefined) return fallback; + const values = raw.split(",").map((part) => Number.parseInt(part.trim(), 10)); + if ( + values.length === 0 || + values.some((value) => !Number.isSafeInteger(value) || value < 1) + ) { + throw new Error(`${name} must be comma-separated positive integers`); + } + return values; +} + +export function sleep(ms: number): Promise { + return new Promise((resolveSleep) => setTimeout(resolveSleep, ms)); +} + +export async function withTimeout( + label: string, + promise: Promise, + timeoutMs: number, +): Promise { + let timer: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + promise, + new Promise((_resolve, reject) => { + timer = setTimeout( + () => reject(new Error(`${label} timed out after ${timeoutMs}ms`)), + timeoutMs, + ); + }), + ]); + } finally { + if (timer) clearTimeout(timer); + } +} + +function readNumber(path: string): number { + try { + return Number.parseInt(readFileSync(path, "utf8").trim(), 10) || 0; + } catch { + return 0; + } +} + +function childPids(pid: number): number[] { + try { + const text = readFileSync(`/proc/${pid}/task/${pid}/children`, "utf8").trim(); + return text + ? text + .split(/\s+/) + .map(Number) + .filter((value) => Number.isSafeInteger(value) && value > 0) + : []; + } catch { + return []; + } +} + +function pidRssBytes(pid: number): number { + try { + const pages = Number.parseInt( + readFileSync(`/proc/${pid}/statm`, "utf8").split(/\s+/)[1] ?? "0", + 10, + ); + return (Number.isFinite(pages) ? pages : 0) * 4096; + } catch { + return 0; + } +} + +/** Pss in bytes from smaps_rollup, or -1 if unavailable for this pid. */ +function pidPssBytes(pid: number): number { + try { + const rollup = readFileSync(`/proc/${pid}/smaps_rollup`, "utf8"); + const match = rollup.match(/^Pss:\s+(\d+)\s+kB/m); + return match ? Number(match[1]) * 1024 : -1; + } catch { + return -1; + } +} + +export function sampleProcessTree(rootPid = process.pid): ProcessTreeSample { + const queue = [rootPid]; + const visited = new Set(); + let rssBytes = 0; + let pssBytes = 0; + let pssAvailable = false; + let threadCount = 0; + let fdCount = 0; + + while (queue.length > 0) { + const pid = queue.pop(); + if (pid === undefined || visited.has(pid) || !existsSync(`/proc/${pid}`)) { + continue; + } + visited.add(pid); + rssBytes += pidRssBytes(pid); + const pss = pidPssBytes(pid); + if (pss >= 0) { + pssBytes += pss; + pssAvailable = true; + } + try { + threadCount += readdirSync(`/proc/${pid}/task`).length; + } catch { + // The process may exit between census operations. + } + try { + fdCount += readdirSync(`/proc/${pid}/fd`).length; + } catch { + // The process may exit between census operations. + } + queue.push(...childPids(pid)); + } + + return { + timestampMs: Date.now(), + rssBytes, + pssBytes: pssAvailable ? pssBytes : -1, + pidCount: visited.size, + threadCount, + fdCount, + hostHeapUsedBytes: process.memoryUsage().heapUsed, + pids: [...visited].sort((left, right) => left - right), + }; +} + +export function cgroupSnapshot(): Record { + const root = "/sys/fs/cgroup"; + const fields = [ + "memory.current", + "memory.peak", + "memory.max", + "memory.swap.max", + "pids.current", + "pids.max", + ] as const; + const snapshot: Record = {}; + for (const field of fields) { + try { + const value = readFileSync(join(root, field), "utf8").trim(); + snapshot[field] = /^\d+$/.test(value) ? Number(value) : value; + } catch { + // cgroup v1 and non-Linux environments omit these files. + } + } + try { + for (const line of readFileSync(join(root, "memory.events"), "utf8").trim().split("\n")) { + const [key, value] = line.split(/\s+/); + if (key && value) snapshot[`memory.events.${key}`] = Number(value); + } + } catch { + // Optional diagnostic. + } + return snapshot; +} + +export function linearSlope(values: number[]): number { + if (values.length < 2) return 0; + const meanX = (values.length - 1) / 2; + const meanY = values.reduce((sum, value) => sum + value, 0) / values.length; + let numerator = 0; + let denominator = 0; + for (let index = 0; index < values.length; index += 1) { + const deltaX = index - meanX; + numerator += deltaX * (values[index]! - meanY); + denominator += deltaX * deltaX; + } + return denominator === 0 ? 0 : numerator / denominator; +} + +export function percentile(values: number[], quantile: number): number { + if (values.length === 0) return 0; + const sorted = [...values].sort((left, right) => left - right); + const index = Math.min( + sorted.length - 1, + Math.max(0, Math.ceil(sorted.length * quantile) - 1), + ); + return sorted[index]!; +} + +export function writeArtifact( + lane: string, + runId: string, + value: unknown, +): string { + const directory = join(artifactRoot, lane); + mkdirSync(directory, { recursive: true }); + const path = join(directory, `${runId}.json`); + writeFileSync(path, `${JSON.stringify(value, null, 2)}\n`, { mode: 0o600 }); + return path; +} + +export function newRunId(lane: string): string { + return `${new Date().toISOString().replace(/[:.]/g, "-")}-${lane}-${process.pid}`; +} + +export function errorText(error: unknown): string { + return error instanceof Error ? `${error.name}: ${error.message}` : String(error); +} + +export function runtimeProvenance(): Record { + const sidecar = process.env.AGENTOS_SIDECAR_BIN; + let sidecarTarget: string | undefined; + try { + sidecarTarget = sidecar ? readlinkSync(sidecar) : undefined; + } catch { + sidecarTarget = sidecar ? basename(sidecar) : undefined; + } + return { + node: process.version, + platform: process.platform, + arch: process.arch, + sidecar, + sidecarTarget, + cgroup: cgroupSnapshot(), + }; +} + +export function forceGc(): void { + (globalThis as { gc?: () => void }).gc?.(); +} + +export function readCgroupNumber(field: string): number { + return readNumber(join("/sys/fs/cgroup", field)); +} diff --git a/packages/load-tests/src/compute/controller.ts b/packages/load-tests/src/compute/controller.ts new file mode 100644 index 0000000000..79028e0db6 --- /dev/null +++ b/packages/load-tests/src/compute/controller.ts @@ -0,0 +1,825 @@ +// External Rivet Compute load generator. +// +// Runs OUTSIDE the target Compute deployment (in its own 1-CPU/1-GiB bounded +// container, via `just load-test-compute`) so target saturation cannot hide or +// coordinate the offered load. It drives the runner-scaling axis directly +// through the Rivet Engine management API (the RivetKit client's remote/auth +// wiring is deployment-specific; the raw API is the documented, reliable path): +// - ramps keyed `agentosLoadRunner` actors through calibrated steps, +// - measures actor create-to-ready latency, +// - samples the runner census (active/stopped/slots), +// - enforces hard ceilings for actor count, duration, and cleanup deadline, +// - has a local kill switch and a bounded cleanup deadline, +// - destroys every created actor and confirms runners drain, +// - redacts every credential from logs and artifacts. +// +// Credentials come only from the environment (see docs-internal checklist K): +// RIVET_ENDPOINT https://:sk_...@api.rivet.dev (secret; census) +// RIVET_PUBLIC_ENDPOINT https://:pk_...@api.rivet.dev (public; actors) +// RIVET_RUN_URL https://.rivet.run (gateway health) +// They are never printed, committed, or written to an artifact. +import { + errorText, + integerEnv, + newRunId, + numberEnv, + percentile, + runtimeProvenance, + sleep, + withTimeout, + writeArtifact, +} from "../common.js"; + +/** Raised when a required Compute credential/endpoint is absent. */ +class MissingComputeCredentialsError extends Error { + constructor(names: string[]) { + super( + `missing required Compute credentials: ${names.join(", ")}; ` + + `export them in the invoking environment (never commit them)`, + ); + this.name = "MissingComputeCredentialsError"; + } +} + +interface ParsedEndpoint { + namespace: string; + token: string; + origin: string; +} + +/** Parse `https://:@host` into parts; token stays in memory. */ +function parseEndpoint(raw: string): ParsedEndpoint { + const url = new URL(raw); + const namespace = decodeURIComponent(url.username); + const token = decodeURIComponent(url.password); + if (!namespace || !token) { + throw new Error("endpoint must be https://:@host"); + } + url.username = ""; + url.password = ""; + return { namespace, token, origin: url.origin }; +} + +/** Redact any token-looking substring or userinfo from text. */ +function redact(text: string): string { + return text + .replace(/\b(sk|pk|cloud)_[A-Za-z0-9._-]+/g, "$1_***REDACTED***") + .replace(/\/\/[^/@\s]+:[^/@\s]+@/g, "//***:***@"); +} + +interface RunnerCensus { + timestampMs: number; + activeRunners: number; + stoppedRunners: number; + totalRunners: number; + remainingSlots: number | null; + totalSlots: number | null; + error?: string; +} + +async function sampleRunners( + api: string, + namespace: string, + secretToken: string, + timeoutMs: number, +): Promise { + const now = Date.now(); + try { + const url = + `${api}/runners?namespace=${encodeURIComponent(namespace)}` + + `&name=default&include_stopped=true&limit=100`; + const res = await withTimeout( + "runner census", + fetch(url, { headers: { Authorization: `Bearer ${secretToken}` } }), + timeoutMs, + ); + if (!res.ok) { + return { + timestampMs: now, + activeRunners: 0, + stoppedRunners: 0, + totalRunners: 0, + remainingSlots: null, + totalSlots: null, + error: `runner census HTTP ${res.status}`, + }; + } + const body = (await res.json()) as { + runners?: Array<{ + stopped_at?: number | null; + remaining_slots?: number | null; + total_slots?: number | null; + }>; + }; + const runners = body.runners ?? []; + const active = runners.filter((r) => r.stopped_at == null); + return { + timestampMs: now, + activeRunners: active.length, + stoppedRunners: runners.length - active.length, + totalRunners: runners.length, + remainingSlots: active.reduce((s, r) => s + (r.remaining_slots ?? 0), 0), + totalSlots: active.reduce((s, r) => s + (r.total_slots ?? 0), 0), + }; + } catch (error) { + return { + timestampMs: now, + activeRunners: 0, + stoppedRunners: 0, + totalRunners: 0, + remainingSlots: null, + totalSlots: null, + error: redact(errorText(error)), + }; + } +} + +/** Create a keyed actor via the Engine API; returns its actor id. */ +async function createActor( + api: string, + namespace: string, + publicToken: string, + actorName: string, + key: string, + timeoutMs: number, +): Promise { + const res = await withTimeout( + "actor create", + fetch(`${api}/actors?namespace=${encodeURIComponent(namespace)}`, { + method: "POST", + headers: { + Authorization: `Bearer ${publicToken}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + name: actorName, + key, + runner_name_selector: "default", + crash_policy: "restart", + }), + }), + timeoutMs, + ); + if (!res.ok) { + throw new Error(`actor create HTTP ${res.status}: ${redact((await res.text()).slice(0, 200))}`); + } + const body = (await res.json()) as { + actor?: { actor_id?: string; id?: string }; + actor_id?: string; + id?: string; + }; + const id = body.actor?.actor_id ?? body.actor?.id ?? body.actor_id ?? body.id; + if (!id) throw new Error(`actor create returned no id: ${redact(JSON.stringify(body).slice(0, 200))}`); + return id; +} + +/** Poll an actor's gateway health until 200 or the deadline. */ +async function waitActorReady( + api: string, + publicToken: string, + actorId: string, + deadlineMs: number, +): Promise { + while (Date.now() < deadlineMs) { + try { + const res = await withTimeout( + "actor health", + fetch(`${api}/gateway/${actorId}/health`, { + headers: { "x-rivet-token": publicToken }, + }), + 8_000, + ); + if (res.ok) return true; + } catch { + // transient; keep polling + } + await sleep(1_000); + } + return false; +} + +/** Call a bounded actor action over the gateway (e.g. execArgv). Returns the + * KernelExecResult in `.output`, or an error. Wire shape reverse-engineered from + * rivetkit: POST /gateway//action/, x-rivet-token + x-rivet-encoding:json, + * body {args:[command, argv[]]}. */ +async function callAction( + api: string, + publicToken: string, + actorId: string, + command: string, + argv: string[], + timeoutMs: number, +): Promise<{ ok: boolean; exitCode?: number; stdout?: string; stderr?: string; error?: string }> { + try { + const res = await withTimeout( + "actor action", + fetch(`${api}/gateway/${actorId}/action/execArgv`, { + method: "POST", + headers: { + "x-rivet-token": publicToken, + "x-rivet-encoding": "json", + "Content-Type": "application/json", + }, + body: JSON.stringify({ args: [command, argv, { timeout: Math.min(timeoutMs, 18_000) }] }), + }), + timeoutMs, + ); + if (!res.ok) return { ok: false, error: `action HTTP ${res.status}: ${redact((await res.text()).slice(0, 160))}` }; + const body = (await res.json()) as { + output?: { exitCode?: number; stdout?: string; stderr?: string }; + exitCode?: number; + stdout?: string; + stderr?: string; + }; + const out = body.output ?? body; + return { ok: true, exitCode: out.exitCode, stdout: out.stdout, stderr: out.stderr }; + } catch (error) { + return { ok: false, error: redact(errorText(error)) }; + } +} + +interface ActorCensus { + timestampMs: number; + total: number; + pendingAllocation: number; + connectable: number; + sleeping: number; + live: number; + error?: string; +} + +/** Census the actor fleet via /actors list + lifecycle timestamps (the real + * scaling signal, since /runners is empty — LT-014). */ +async function sampleActors( + api: string, + namespace: string, + token: string, + actorName: string, + timeoutMs: number, +): Promise { + const now = Date.now(); + try { + const url = + `${api}/actors?namespace=${encodeURIComponent(namespace)}` + + `&name=${encodeURIComponent(actorName)}&include_destroyed=false&limit=100`; + const res = await withTimeout( + "actor census", + fetch(url, { headers: { Authorization: `Bearer ${token}` } }), + timeoutMs, + ); + if (!res.ok) { + return { timestampMs: now, total: 0, pendingAllocation: 0, connectable: 0, sleeping: 0, live: 0, error: `HTTP ${res.status}` }; + } + const body = (await res.json()) as { + actors?: Array<{ + pending_allocation_ts?: number | null; + connectable_ts?: number | null; + sleep_ts?: number | null; + destroy_ts?: number | null; + }>; + }; + const actors = (body.actors ?? []).filter((a) => a.destroy_ts == null); + return { + timestampMs: now, + total: actors.length, + pendingAllocation: actors.filter((a) => a.pending_allocation_ts != null && a.connectable_ts == null).length, + connectable: actors.filter((a) => a.connectable_ts != null && a.sleep_ts == null).length, + sleeping: actors.filter((a) => a.sleep_ts != null).length, + live: actors.length, + }; + } catch (error) { + return { timestampMs: now, total: 0, pendingAllocation: 0, connectable: 0, sleeping: 0, live: 0, error: redact(errorText(error)) }; + } +} + +/** Destroy an actor via the Engine API. DELETE requires the SECRET token + * (the publishable token gets 403 insufficient_permissions). */ +async function destroyActor( + api: string, + namespace: string, + secretToken: string, + actorId: string, +): Promise { + try { + const res = await withTimeout( + "actor destroy", + fetch(`${api}/actors/${actorId}?namespace=${encodeURIComponent(namespace)}`, { + method: "DELETE", + headers: { Authorization: `Bearer ${secretToken}` }, + }), + 10_000, + ); + return res.ok; + } catch { + return false; + } +} + +interface StepResult { + actors: number; + createReadyMsP50: number; + createReadyMsP99: number; + readyCount: number; + failedCount: number; + censusAfterRamp: RunnerCensus; + censusAfterHold: RunnerCensus; + peakActiveRunners: number; +} + +const TRIVIAL_SCRIPT = "process.stdout.write('ok')"; +// Overflow the server's 64 MiB maxFilesystemBytes → trips the LT-011 crash class. +const FS_CRASH_SCRIPT = + "const fs=require('node:fs');const c=Buffer.alloc(1024*1024,65);const fd=fs.openSync('/tmp/fill.bin','w');for(let i=0;i<96;i++)fs.writeSync(fd,c);fs.closeSync(fd);process.stdout.write('wrote')"; + +/** + * Noisy-neighbor blast-radius test: fill a couple of runners with actors, + * activate each actor's VM, then make ONE actor run the LT-011 filesystem-crash + * workload and immediately re-probe ALL actors. If co-located actors fail their + * next action, one guest crashing the shared runner sidecar is a cross-tenant + * DoS at Compute scale. + */ +async function runNoisyNeighbor( + runId: string, + api: string, + namespace: string, + pk: string, + sk: string, + actorName: string, +): Promise { + const n = integerEnv("COMPUTE_NN_ACTORS", 30); + const readyTimeoutMs = integerEnv("COMPUTE_READY_TIMEOUT_MS", 45_000); + const ids: string[] = []; + const failures: string[] = []; + let attackerId: string | undefined; + let survivorsBefore = 0; + let survivorsAfter = 0; + let attackerResult = ""; + + try { + // Create + ready N actors. + for (let i = 0; i < n; i += 1) { + const id = await createActor(api, namespace, pk, actorName, `${runId}-nn${i}`, 20_000); + ids.push(id); + } + await Promise.all(ids.map((id) => waitActorReady(api, pk, id, Date.now() + readyTimeoutMs))); + + // Activate each actor's VM + confirm it answers. SEQUENTIALLY — a runner's + // executor pool = --cpu count (LT-020), so concurrent actions on a small + // runner get rejected; serial probing gives each actor the executor. + const before: boolean[] = []; + for (const id of ids) { + const r = await callAction(api, pk, id, "node", ["-e", TRIVIAL_SCRIPT], 25_000); + before.push(r.ok && r.exitCode === 0); + } + survivorsBefore = before.filter(Boolean).length; + + // One actor trips the crash. + attackerId = ids[0]; + const atk = await callAction(api, pk, attackerId!, "node", ["-e", FS_CRASH_SCRIPT], 30_000); + attackerResult = atk.ok ? `exit=${atk.exitCode} out=${(atk.stdout ?? "").slice(0, 20)} err=${redact((atk.stderr ?? "").slice(0, 160))}` : `error=${atk.error}`; + + // Re-probe ALL actors (serially); co-located failures = blast radius. + await sleep(2_000); + const after: boolean[] = []; + for (const id of ids) { + const r = await callAction(api, pk, id, "node", ["-e", TRIVIAL_SCRIPT], 25_000); + after.push(r.ok && r.exitCode === 0); + } + survivorsAfter = after.filter(Boolean).length; + + // Collateral = neighbors (excluding the attacker) that answered before but not after. + let collateral = 0; + for (let i = 1; i < ids.length; i += 1) if (before[i] && !after[i]) collateral += 1; + if (collateral > 0) { + failures.push(`blast radius: ${collateral} co-located actor(s) failed after one actor tripped the fs crash (LT-011 at scale)`); + } + } catch (error) { + failures.push(`noisy-neighbor error: ${redact(errorText(error))}`); + } finally { + for (const id of ids) await destroyActor(api, namespace, sk, id); + } + + const artifact = { + runId, + lane: "compute-noisy-neighbor", + verdict: failures.length === 0 ? "pass" : "fail", + failures, + config: { actors: n, actorName }, + attacker: { id: attackerId, result: attackerResult }, + survivorsBefore, + survivorsAfter, + provenance: runtimeProvenance(), + }; + const path = writeArtifact("compute", runId, artifact); + console.log(JSON.stringify({ verdict: artifact.verdict, failures, path, survivorsBefore, survivorsAfter, attacker: attackerResult.slice(0, 120) })); + if (failures.length > 0) process.exitCode = 1; +} + +/** + * Sleep/wake persistence E2E: create actors, write a marker file in each VM, + * let them go to sleep (poll sleep_ts), wake them, and verify the file survived. + * This tests that an actor's AgentOS filesystem persists across the VM being + * recreated on wake (server.ts recreates the VM lazily via ensureVm). + */ +async function runSleepWake( + runId: string, + api: string, + namespace: string, + pk: string, + sk: string, + actorName: string, +): Promise { + const n = integerEnv("COMPUTE_SW_ACTORS", 5); + const sleepWaitMs = integerEnv("COMPUTE_SW_SLEEP_WAIT_MS", 240_000); + const dir = process.env.COMPUTE_SW_DIR ?? "/workspace"; + const readyTimeoutMs = integerEnv("COMPUTE_READY_TIMEOUT_MS", 60_000); + const ids: string[] = []; + const failures: string[] = []; + const markers = new Map(); + let wroteOk = 0; + let verifiedBeforeSleep = 0; + let sleptCount = 0; + let survivedAfterWake = 0; + let sampleWakeRead = ""; + + // The marker file path lives inside the VM. + const writeScript = (m: string) => + `const fs=require('node:fs');fs.writeFileSync(${JSON.stringify(`${dir}/sw-marker.txt`)}, ${JSON.stringify(m)});process.stdout.write('wrote')`; + const readScript = `const fs=require('node:fs');try{process.stdout.write(fs.readFileSync(${JSON.stringify(`${dir}/sw-marker.txt`)},'utf8'))}catch(e){process.stdout.write('MISSING:'+(e.code||e.message))}`; + + try { + for (let i = 0; i < n; i += 1) { + const id = await createActor(api, namespace, pk, actorName, `${runId}-sw${i}`, 20_000); + ids.push(id); + markers.set(id, `marker-${runId}-${i}`); + } + for (const id of ids) await waitActorReady(api, pk, id, Date.now() + readyTimeoutMs); + + // Write a marker file in each (serial — executor pool may be small). + for (const id of ids) { + const w = await callAction(api, pk, id, "node", ["-e", writeScript(markers.get(id)!)], 25_000); + if (w.ok && w.exitCode === 0 && w.stdout === "wrote") wroteOk += 1; + const r = await callAction(api, pk, id, "node", ["-e", readScript], 25_000); + if (r.ok && r.stdout === markers.get(id)) verifiedBeforeSleep += 1; + } + + // Wait for actors to sleep (stop touching them; poll sleep_ts). + const deadline = Date.now() + sleepWaitMs; + while (Date.now() < deadline) { + await sleep(10_000); + const census = await sampleActors(api, namespace, sk, actorName, 10_000); + sleptCount = census.sleeping; + if (census.sleeping >= ids.length) break; + } + + // Wake each actor with a read; verify the file survived. + for (const id of ids) { + const r = await callAction(api, pk, id, "node", ["-e", readScript], 30_000); + if (r.ok && r.stdout === markers.get(id)) survivedAfterWake += 1; + else if (!sampleWakeRead) sampleWakeRead = r.ok ? `stdout=${(r.stdout ?? "").slice(0, 40)}` : `error=${r.error}`; + } + + if (wroteOk < ids.length) failures.push(`only ${wroteOk}/${ids.length} wrote the marker`); + if (verifiedBeforeSleep < ids.length) failures.push(`only ${verifiedBeforeSleep}/${ids.length} read back the marker before sleep`); + if (sleptCount === 0) failures.push(`no actors observed sleeping within ${sleepWaitMs}ms (sleep may be slow/disabled)`); + if (survivedAfterWake < ids.length) { + failures.push(`PERSISTENCE: only ${survivedAfterWake}/${ids.length} markers survived sleep→wake (sample: ${sampleWakeRead})`); + } + } catch (error) { + failures.push(`sleep-wake error: ${redact(errorText(error))}`); + } finally { + for (const id of ids) await destroyActor(api, namespace, sk, id); + } + + const artifact = { + runId, + lane: "compute-sleep-wake", + verdict: failures.length === 0 ? "pass" : "fail", + failures, + config: { actors: n, dir, sleepWaitMs, actorName }, + metrics: { wroteOk, verifiedBeforeSleep, sleptCount, survivedAfterWake, total: ids.length }, + provenance: runtimeProvenance(), + }; + const path = writeArtifact("compute", runId, artifact); + console.log(JSON.stringify({ verdict: artifact.verdict, failures, path, metrics: artifact.metrics })); + if (failures.length > 0) process.exitCode = 1; +} + +/** + * Rivet actor-churn soak: for a bounded duration, repeatedly create→ready→ + * destroy small batches of actors. Verifies no actor/runner leak and a healthy + * deployment under sustained lifecycle churn. + */ +async function runChurnSoak( + runId: string, + api: string, + namespace: string, + pk: string, + sk: string, + actorName: string, + runUrl: string | undefined, +): Promise { + const durationMs = integerEnv("COMPUTE_SOAK_MS", 300_000); + const batch = integerEnv("COMPUTE_SOAK_BATCH", 5); + const readyTimeoutMs = integerEnv("COMPUTE_READY_TIMEOUT_MS", 45_000); + const failures: string[] = []; + let created = 0; + let ready = 0; + let destroyed = 0; + let cycles = 0; + let healthChecks = 0; + let healthFailures = 0; + const deadline = Date.now() + durationMs; + + while (Date.now() < deadline) { + cycles += 1; + const ids: string[] = []; + for (let i = 0; i < batch; i += 1) { + try { + const id = await createActor(api, namespace, pk, actorName, `${runId}-c${cycles}-${i}`, 20_000); + ids.push(id); + created += 1; + } catch { + // create failure counted implicitly (fewer ids) + } + } + for (const id of ids) { + if (await waitActorReady(api, pk, id, Date.now() + readyTimeoutMs)) ready += 1; + } + for (const id of ids) { + if (await destroyActor(api, namespace, sk, id)) destroyed += 1; + } + // Periodic health check. + if (runUrl) { + try { + const res = await withTimeout("health", fetch(`${runUrl}/api/rivet/health`), 8_000); + healthChecks += 1; + if (!res.ok) healthFailures += 1; + } catch { + healthChecks += 1; + healthFailures += 1; + } + } + } + + // Leak check: any actors from this run still alive? + await sleep(3_000); + const census = await sampleActors(api, namespace, sk, actorName, 10_000); + const leaked = census.live; + + if (created > 0 && destroyed < created) failures.push(`destroyed ${destroyed} < created ${created}`); + if (leaked > 0) failures.push(`${leaked} actor(s) still live after soak (possible leak)`); + if (healthFailures > 0) failures.push(`${healthFailures}/${healthChecks} deployment health checks failed`); + + const artifact = { + runId, + lane: "compute-churn-soak", + verdict: failures.length === 0 ? "pass" : "fail", + failures, + config: { durationMs, batch, actorName }, + metrics: { cycles, created, ready, destroyed, leaked, healthChecks, healthFailures }, + provenance: runtimeProvenance(), + }; + const path = writeArtifact("compute", runId, artifact); + console.log(JSON.stringify({ verdict: artifact.verdict, failures, path, metrics: artifact.metrics })); + if (failures.length > 0) process.exitCode = 1; +} + +export async function runComputeLoad(): Promise { + const runId = newRunId("compute"); + const actorName = process.env.COMPUTE_ACTOR_NAME ?? "agentosLoadRunner"; + // Hard safety ceiling on offered load. Raised for hundreds-of-actors scaling + // tests; the controller still create-storms with bounded concurrency and + // destroys everything within the cleanup deadline. + const MAX_ACTORS = integerEnv("COMPUTE_MAX_ACTORS", 600); + const steps = (process.env.COMPUTE_STEPS ?? "1,5,10") + .split(",") + .map((s) => Number.parseInt(s.trim(), 10)); + if (steps.some((n) => !Number.isSafeInteger(n) || n < 1 || n > MAX_ACTORS)) { + throw new Error(`COMPUTE_STEPS must be comma-separated integers in 1..${MAX_ACTORS}`); + } + const holdMs = integerEnv("COMPUTE_HOLD_MS", 20_000); + const scaleDownMs = integerEnv("COMPUTE_SCALE_DOWN_MS", 60_000); + const createConcurrency = integerEnv("COMPUTE_CREATE_CONCURRENCY", 8); + const readyTimeoutMs = integerEnv("COMPUTE_READY_TIMEOUT_MS", 45_000); + const cleanupDeadlineMs = integerEnv("COMPUTE_CLEANUP_DEADLINE_MS", 90_000); + const censusTimeoutMs = numberEnv("COMPUTE_CENSUS_TIMEOUT_MS", 10_000); + + // Credentials: fail fast and record a blocker rather than proceeding blind. + const runUrl = process.env.RIVET_RUN_URL; + const publicEndpointRaw = process.env.RIVET_PUBLIC_ENDPOINT; + const secretEndpointRaw = process.env.RIVET_ENDPOINT; + const missing: string[] = []; + if (!publicEndpointRaw) missing.push("RIVET_PUBLIC_ENDPOINT"); + if (!secretEndpointRaw) missing.push("RIVET_ENDPOINT"); + if (missing.length > 0) { + const artifact = { + runId, + lane: "compute-scaling", + verdict: "blocked", + reason: "missing Compute credentials/endpoints", + missing, + config: { steps, holdMs, scaleDownMs }, + provenance: runtimeProvenance(), + }; + const path = writeArtifact("compute", runId, artifact); + console.log(JSON.stringify({ verdict: "blocked", missing, path })); + throw new MissingComputeCredentialsError(missing); + } + + const secret = parseEndpoint(secretEndpointRaw!); + const pub = parseEndpoint(publicEndpointRaw!); + const api = pub.origin; // https://api.rivet.dev + const namespace = pub.namespace; + + // Attack #1: does one actor tripping the LT-011 sidecar crash take down its + // co-located actors (shared-runner blast radius)? + const mode = process.env.COMPUTE_MODE ?? "scale"; + if (mode === "noisy-neighbor") { + await runNoisyNeighbor(runId, api, namespace, pub.token, secret.token, actorName); + return; + } + // Sleep/wake persistence: spawn actors, write a file, let them sleep, wake + // them, verify the file survived. + if (mode === "sleep-wake") { + await runSleepWake(runId, api, namespace, pub.token, secret.token, actorName); + return; + } + // Rivet soak: rapid create→ready→destroy churn for a duration; verify no + // leaked actors and the deployment stays healthy. + if (mode === "churn") { + await runChurnSoak(runId, api, namespace, pub.token, secret.token, actorName, runUrl); + return; + } + + let aborted = false; + const abort = () => { + aborted = true; + }; + process.once("SIGINT", abort); + process.once("SIGTERM", abort); + + const created = new Map(); // index -> actorId + const stepResults: StepResult[] = []; + const censusSeries: RunnerCensus[] = []; + const actorCensusSeries: ActorCensus[] = []; + let topLevelError: string | undefined; + let peakActiveRunners = 0; + let peakConnectable = 0; + let peakPending = 0; + + const census = async () => { + // Runner census (empty for this deployment — LT-014) plus the actor + // lifecycle census, which IS the real scaling signal. + const [c, a] = await Promise.all([ + sampleRunners(api, namespace, secret.token, censusTimeoutMs), + sampleActors(api, namespace, secret.token, actorName, censusTimeoutMs), + ]); + peakActiveRunners = Math.max(peakActiveRunners, c.activeRunners); + peakConnectable = Math.max(peakConnectable, a.connectable); + peakPending = Math.max(peakPending, a.pendingAllocation); + censusSeries.push(c); + actorCensusSeries.push(a); + return c; + }; + + async function pool(items: number[], limit: number, fn: (i: number) => Promise): Promise { + const out: T[] = []; + let cursor = 0; + const worker = async () => { + while (cursor < items.length && !aborted) { + const i = items[cursor++]!; + out.push(await fn(i)); + } + }; + await Promise.all(Array.from({ length: Math.min(limit, items.length) }, () => worker())); + return out; + } + + try { + // Baseline census before any load. + const baselineCensus = await census(); + + for (const actors of steps) { + if (aborted) break; + const indices = Array.from({ length: actors }, (_, i) => i); + const readyMs: number[] = []; + let failed = 0; + + await pool(indices, createConcurrency, async (index) => { + const key = `${runId}-a${index}`; + const startedAt = Date.now(); + try { + if (!created.has(index)) { + const id = await createActor(api, namespace, pub.token, actorName, key, 20_000); + created.set(index, id); + } + const ready = await waitActorReady(api, pub.token, created.get(index)!, Date.now() + readyTimeoutMs); + if (ready) readyMs.push(Date.now() - startedAt); + else failed += 1; + } catch (error) { + failed += 1; + if (failed <= 2) topLevelError = redact(errorText(error)); + } + }); + + const censusAfterRamp = await census(); + + const holdEnd = Date.now() + holdMs; + while (Date.now() < holdEnd && !aborted) { + await census(); + await sleep(2_000); + } + const censusAfterHold = await census(); + + stepResults.push({ + actors, + createReadyMsP50: percentile(readyMs, 0.5), + createReadyMsP99: percentile(readyMs, 0.99), + readyCount: readyMs.length, + failedCount: failed, + censusAfterRamp, + censusAfterHold, + peakActiveRunners, + }); + void baselineCensus; + } + } catch (error) { + topLevelError = redact(errorText(error)); + } finally { + // Cleanup: destroy every created actor within the bounded deadline. + const cleanupStart = Date.now(); + for (const [, actorId] of created) { + if (Date.now() - cleanupStart > cleanupDeadlineMs) break; + await destroyActor(api, namespace, secret.token, actorId); + } + } + + // Drain: sample until runners return to their idle floor or the deadline. + const drainStart = Date.now(); + let finalCensus = await census(); + let drainMs: number | null = null; + while (finalCensus.activeRunners > 0 && Date.now() - drainStart < scaleDownMs && !aborted) { + await sleep(3_000); + finalCensus = await census(); + } + if (finalCensus.activeRunners === 0) drainMs = Date.now() - drainStart; + + const failures: string[] = []; + if (topLevelError) failures.push(`controller error: ${topLevelError}`); + if (stepResults.some((s) => s.failedCount > 0)) failures.push("one or more actors failed to become ready"); + if (finalCensus.activeRunners > 0) { + failures.push(`${finalCensus.activeRunners} runner(s) still active after drain deadline`); + } + // NOTE (LT-014): the /runners census returns 0 for this serverless deployment, + // so runner count is informational only — the scaling verdict rests on the + // actor lifecycle (create-to-ready, health, destroy/drain), not runner count. + const runnerCensusObserved = censusSeries.some((c) => c.totalRunners > 0); + + const artifact = { + runId, + lane: "compute-scaling", + verdict: aborted ? "aborted" : failures.length === 0 ? "pass" : "fail", + failures, + config: { + actorName, + steps, + holdMs, + scaleDownMs, + createConcurrency, + readyTimeoutMs, + cleanupDeadlineMs, + maxActors: MAX_ACTORS, + }, + endpoints: { + namespace, + api, + runUrl: runUrl ? redact(runUrl) : undefined, + }, + provenance: runtimeProvenance(), + steps: stepResults, + peakActiveRunners, + peakConnectable, + peakPending, + runnerCensusObserved, + drainMs, + finalCensus, + census: censusSeries, + actorCensus: actorCensusSeries, + createdActorCount: created.size, + }; + const path = writeArtifact("compute", runId, artifact); + console.log( + JSON.stringify({ + verdict: artifact.verdict, + failures, + path, + peakActiveRunners, + peakConnectable, + peakPending, + drainMs, + steps: stepResults.map((s) => ({ actors: s.actors, ready: s.readyCount, failed: s.failedCount, connectable: s.censusAfterRamp.activeRunners })), + }), + ); + if (failures.length > 0 && !aborted) process.exitCode = 1; +} diff --git a/packages/load-tests/src/compute/server.ts b/packages/load-tests/src/compute/server.ts new file mode 100644 index 0000000000..30898083df --- /dev/null +++ b/packages/load-tests/src/compute/server.ts @@ -0,0 +1,64 @@ +// Rivet Compute load-runner application. +// +// One AgentOS actor, `agentosLoadRunner`, is deployed to Rivet Compute. The +// external controller (`compute-load`) drives keyed instances of this actor +// from outside Compute so target saturation cannot hide the offered load. +// +// Design constraints (see docs-internal/load-testing.md, checklist J): +// - Keep `registry.start()`; do not hand-mount a router. RivetKit listens on +// RIVET_PORT (default 3000) and serves `/api/rivet/metadata`. +// - Every actor gets strict per-actor AgentOS process/memory/output/time +// limits so one Compute instance cannot exhaust its container. +// - The bounded AgentOS work surface is the built-in `execArgv` / `spawn` +// action set; its bounds are the `limits` object below. The controller +// only calls these bounded actions. +// - Live VM handles are process-local. The actor persists nothing but the +// RivetKit-managed VM/session state; on migration the VM is recreated +// lazily by `ensureVm` on the next action, and any in-flight in-memory +// guest work is dropped (reported as interrupted by the controller, which +// sees the failed action), never silently duplicated. +import { agentOS, setup } from "@rivet-dev/agentos"; + +// Strict per-actor envelope. These bound a single Compute instance's AgentOS +// footprint; the controller varies actor COUNT and work-per-actor on top. +const agentosLoadRunner = agentOS({ + // No default software: the load workloads only need node/exec, keeping + // per-instance memory low so actor count, not package weight, drives scale. + defaultSoftware: false, + limits: { + resources: { + maxProcesses: 32, + maxOpenFds: 128, + maxSockets: 64, + maxFilesystemBytes: 64 * 1024 * 1024, + maxWasmFuel: 30_000, + maxWasmMemoryBytes: 64 * 1024 * 1024, + maxWasmStackBytes: 2 * 1024 * 1024, + }, + process: { + pendingStdinBytes: 4 * 1024 * 1024, + pendingEventCount: 4_000, + pendingEventBytes: 16 * 1024 * 1024, + }, + jsRuntime: { + v8HeapLimitMb: 96, + cpuTimeLimitMs: 15_000, + wallClockLimitMs: 20_000, + importCacheMaterializeTimeoutMs: 20_000, + syncRpcWaitTimeoutMs: 20_000, + capturedOutputLimitBytes: 8 * 1024 * 1024, + }, + python: { + executionTimeoutMs: 30_000, + maxOldSpaceMb: 0, + }, + wasm: { + prewarmTimeoutMs: 20_000, + runnerHeapLimitMb: 512, + }, + }, +}); + +export const registry = setup({ use: { agentosLoadRunner } }); + +registry.start(); diff --git a/packages/load-tests/src/local/boundary.ts b/packages/load-tests/src/local/boundary.ts new file mode 100644 index 0000000000..4efd4a6d11 --- /dev/null +++ b/packages/load-tests/src/local/boundary.ts @@ -0,0 +1,107 @@ +// Container-boundary self-test (docs-internal/load-testing.md checklist C). +// +// Runs inside the SAME bounded container recipe as the adversarial lanes and +// verifies that the cgroup envelope, fd ulimit, tmpfs, and network isolation +// actually match what the `just` recipe requested. This is how we prove the +// sandbox is real before trusting any "the sidecar survived" verdict. +import { readFileSync, statfsSync } from "node:fs"; +import { createConnection } from "node:net"; +import { + cgroupSnapshot, + errorText, + newRunId, + numberEnv, + runtimeProvenance, + writeArtifact, +} from "../common.js"; + +function readSelfLimit(name: string): { soft: string; hard: string } | undefined { + try { + for (const line of readFileSync("/proc/self/limits", "utf8").split("\n")) { + // Columns are space-padded: "Max open files 1024 1024 files". + if (line.startsWith(name)) { + const cols = line.slice(name.length).trim().split(/\s+/); + return { soft: cols[0]!, hard: cols[1]! }; + } + } + } catch { + // Non-Linux / unreadable. + } + return undefined; +} + +async function networkReachable(timeoutMs: number): Promise { + // Local lanes run with --network=none, so an outbound TCP connect must fail. + return new Promise((resolve) => { + const socket = createConnection({ host: "1.1.1.1", port: 443 }); + const done = (reachable: boolean) => { + socket.destroy(); + resolve(reachable); + }; + socket.setTimeout(timeoutMs); + socket.once("connect", () => done(true)); + socket.once("timeout", () => done(false)); + socket.once("error", () => done(false)); + }); +} + +export async function runBoundary(): Promise { + const runId = newRunId("boundary"); + const cgroup = cgroupSnapshot(); + const nofile = readSelfLimit("Max open files"); + const procs = readSelfLimit("Max processes"); + + let tmp: { totalBytes: number } | undefined; + try { + const s = statfsSync("/tmp"); + tmp = { totalBytes: Number(s.blocks) * Number(s.bsize) }; + } catch (error) { + tmp = undefined; + console.error(`statfs(/tmp) failed: ${errorText(error)}`); + } + + const networkExpectedReachable = process.env.LOAD_TEST_EXPECT_NETWORK === "1"; + const reachable = await networkReachable(numberEnv("LOAD_TEST_NET_PROBE_MS", 2_000)); + + const memMax = cgroup["memory.max"]; + const swapMax = cgroup["memory.swap.max"]; + const pidsMax = cgroup["pids.max"]; + + const failures: string[] = []; + if (typeof memMax !== "number") failures.push("memory.max is not a numeric cap (unbounded?)"); + // Equal memory+swap (i.e. no extra swap) is enforced by --memory-swap == --memory, + // which makes memory.swap.max == 0 under cgroup v2. + if (swapMax !== 0 && swapMax !== "0") { + failures.push(`memory.swap.max is ${String(swapMax)}; expected 0 (no extra swap)`); + } + if (typeof pidsMax !== "number") failures.push("pids.max is not a numeric cap (unbounded?)"); + if (!nofile) failures.push("could not read RLIMIT_NOFILE"); + if (!tmp) failures.push("could not statfs /tmp"); + if (reachable !== networkExpectedReachable) { + failures.push( + `network reachable=${reachable} but expected reachable=${networkExpectedReachable}`, + ); + } + + const artifact = { + runId, + lane: "container-boundary", + verdict: failures.length === 0 ? "pass" : "fail", + failures, + envelope: { + memoryMax: memMax, + memorySwapMax: swapMax, + pidsMax, + nofile, + maxProcesses: procs, + tmpTotalBytes: tmp?.totalBytes, + networkReachable: reachable, + networkExpectedReachable, + }, + cgroup, + provenance: runtimeProvenance(), + }; + const path = writeArtifact("boundary", runId, artifact); + console.log(JSON.stringify({ verdict: artifact.verdict, failures, path, envelope: artifact.envelope })); + if (failures.length > 0) process.exitCode = 1; +} diff --git a/packages/load-tests/src/local/churn-leak.ts b/packages/load-tests/src/local/churn-leak.ts new file mode 100644 index 0000000000..63f6ca13d5 --- /dev/null +++ b/packages/load-tests/src/local/churn-leak.ts @@ -0,0 +1,354 @@ +import { AgentOs, type AgentOsSidecar } from "@rivet-dev/agentos-core"; +import { + cgroupSnapshot, + errorText, + forceGc, + integerEnv, + linearSlope, + newRunId, + numberEnv, + percentile, + runtimeProvenance, + sampleProcessTree, + sleep, + type ProcessTreeSample, + type TimedProbe, + withTimeout, + writeArtifact, +} from "../common.js"; + +interface ChurnSample { + shape: "sequential" | "burst" | "steady" | "final"; + cycle: number; + activeVmCount: number; + processTree: ProcessTreeSample; +} + +async function createVm(sidecar: AgentOsSidecar): Promise { + return AgentOs.create({ + defaultSoftware: false, + sidecar: { kind: "explicit", handle: sidecar }, + }); +} + +async function cleanWork(vm: AgentOs, marker: string): Promise { + const result = await withTimeout( + `clean workload ${marker}`, + vm.execArgv( + "node", + ["-e", `process.stdout.write(${JSON.stringify(marker)})`], + { timeout: 5_000 }, + ), + 7_000, + ); + if (result.exitCode !== 0 || result.stdout !== marker) { + throw new Error( + `workload ${marker} failed exit=${result.exitCode} stdout=${JSON.stringify(result.stdout)} stderr=${JSON.stringify(result.stderr)}`, + ); + } +} + +function startDirtyWork(vm: AgentOs): void { + vm.spawn( + "node", + [ + "-e", + "const net=require('node:net'); const s=net.createServer(()=>{}); s.listen(0,'127.0.0.1'); setInterval(()=>{},1000)", + ], + { captureStdio: false }, + ); +} + +async function sentinelProbe(vm: AgentOs): Promise { + const startedAtMs = Date.now(); + try { + await cleanWork(vm, "sentinel"); + return { startedAtMs, durationMs: Date.now() - startedAtMs, ok: true }; + } catch (error) { + return { + startedAtMs, + durationMs: Date.now() - startedAtMs, + ok: false, + error: errorText(error), + }; + } +} + +async function settledSample(): Promise { + forceGc(); + forceGc(); + await sleep(100); + return sampleProcessTree(); +} + +export async function runChurnLeak(): Promise { + const runId = newRunId("churn"); + // Default high enough that the post-warmup slope window clears V8/allocator + // warmup (which asymptotes over the first ~12 lifecycles on this host). + const cycles = integerEnv("LOAD_TEST_CYCLES", 20); + const batchSize = integerEnv("LOAD_TEST_BATCH", 4); + const settleMs = integerEnv("LOAD_TEST_SETTLE_MS", 500); + // Plateau slope gate (bytes/sample). A true leak keeps climbing once warm; V8 + // arena retention flattens. Applies to the constant-live steady phase and the + // zero-attacker post-teardown series. + const rssSlopeLimit = numberEnv("LOAD_TEST_RSS_SLOPE_BYTES", 1024 * 1024); + // Total-growth ceilings are PROVISIONAL and calibrated from >=5 clean reps on + // the canonical host (see docs-internal/load-testing.md). RSS is generous + // because glibc/V8 retain freed pages; PSS is the leak-accurate signal. + const rssTotalLimit = numberEnv("LOAD_TEST_RSS_TOTAL_BYTES", 256 * 1024 * 1024); + const pssTotalLimit = numberEnv("LOAD_TEST_PSS_TOTAL_BYTES", 160 * 1024 * 1024); + const failures: string[] = []; + const operationErrors: string[] = []; + const samples: ChurnSample[] = []; + const probes: TimedProbe[] = []; + let sidecar: AgentOsSidecar | undefined; + let sentinel: AgentOs | undefined; + let baseline: ProcessTreeSample | undefined; + let final: ProcessTreeSample | undefined; + const cgroupBefore = cgroupSnapshot(); + + try { + sidecar = await AgentOs.createSidecar({ sidecarId: `load-churn-${runId}` }); + sentinel = await createVm(sidecar); + + // Warm process-global V8, protocol, and allocator paths before the baseline. + for (let index = 0; index < 2; index += 1) { + const vm = await createVm(sidecar); + await cleanWork(vm, `warmup-${index}`); + await vm.dispose(); + } + for (let index = 0; index < 5; index += 1) { + probes.push(await sentinelProbe(sentinel)); + } + await sleep(settleMs); + baseline = await settledSample(); + + for (let cycle = 0; cycle < cycles; cycle += 1) { + try { + const vm = await createVm(sidecar); + if (cycle % 3 === 2) startDirtyWork(vm); + else await cleanWork(vm, `sequential-${cycle}`); + await vm.dispose(); + } catch (error) { + operationErrors.push(`sequential ${cycle}: ${errorText(error)}`); + } + probes.push(await sentinelProbe(sentinel)); + samples.push({ + shape: "sequential", + cycle, + activeVmCount: sidecar.describe().activeVmCount, + processTree: await settledSample(), + }); + } + + for (let cycle = 0; cycle < cycles; cycle += 1) { + let vms: AgentOs[] = []; + try { + vms = await Promise.all( + Array.from({ length: batchSize }, () => createVm(sidecar!)), + ); + await Promise.all( + vms.map((vm, index) => + index % 2 === 0 + ? cleanWork(vm, `burst-${cycle}-${index}`) + : Promise.resolve(startDirtyWork(vm)), + ), + ); + } catch (error) { + operationErrors.push(`burst ${cycle}: ${errorText(error)}`); + } finally { + await Promise.all(vms.map((vm) => vm.dispose().catch(() => {}))); + } + probes.push(await sentinelProbe(sentinel)); + samples.push({ + shape: "burst", + cycle, + activeVmCount: sidecar.describe().activeVmCount, + processTree: await settledSample(), + }); + } + + let steady: AgentOs[] = []; + try { + steady = await Promise.all( + Array.from({ length: batchSize }, () => createVm(sidecar!)), + ); + for (let cycle = 0; cycle < cycles; cycle += 1) { + const slot = cycle % batchSize; + await steady[slot]!.dispose(); + const replacement = await createVm(sidecar); + if (cycle % 2 === 0) await cleanWork(replacement, `steady-${cycle}`); + else startDirtyWork(replacement); + steady[slot] = replacement; + probes.push(await sentinelProbe(sentinel)); + samples.push({ + shape: "steady", + cycle, + activeVmCount: sidecar.describe().activeVmCount, + processTree: await settledSample(), + }); + } + } catch (error) { + operationErrors.push(`steady: ${errorText(error)}`); + } finally { + await Promise.all(steady.map((vm) => vm.dispose().catch(() => {}))); + } + + await sleep(settleMs); + for (let index = 0; index < 5; index += 1) { + probes.push(await sentinelProbe(sentinel)); + } + final = await settledSample(); + samples.push({ + shape: "final", + cycle: cycles, + activeVmCount: sidecar.describe().activeVmCount, + processTree: final, + }); + } catch (error) { + operationErrors.push(`top-level: ${errorText(error)}`); + } finally { + if (sentinel) await sentinel.dispose().catch(() => {}); + if (sidecar) await sidecar.dispose().catch(() => {}); + } + + await sleep(250); + // Zero-attacker post-teardown plateau: with nothing live, RSS/PSS must stop + // trending up. Fitting the slope here (not on the warming sequential phase) + // is what actually distinguishes a leak from V8/glibc arena retention. + const teardownSeries: ProcessTreeSample[] = []; + for (let index = 0; index < 8; index += 1) { + teardownSeries.push(await settledSample()); + await sleep(200); + } + const afterDispose = teardownSeries[teardownSeries.length - 1]!; + + // Constant-live steady-state samples: a leak hidden by always-live VMs shows + // here as a persistent positive per-cycle slope even though the live-VM count + // is fixed. Fit over the POST-WARMUP window (last third) so V8/glibc arena + // warmup — which asymptotes and whose average slope shrinks as the run + // lengthens (a true leak's slope would not) — does not trip the gate. + const steadyRss = samples + .filter((sample) => sample.shape === "steady") + .map((sample) => sample.processTree.rssBytes); + const steadyPostWarmup = steadyRss.slice(Math.floor((steadyRss.length * 2) / 3)); + const steadySlopeBytesPerCycle = linearSlope(steadyPostWarmup); + const teardownRssSlope = linearSlope(teardownSeries.map((s) => s.rssBytes)); + const teardownPssValues = teardownSeries.map((s) => s.pssBytes).filter((v) => v >= 0); + const teardownPssSlope = + teardownPssValues.length >= 2 ? linearSlope(teardownPssValues) : 0; + + const rssTotalGrowthBytes = + baseline && final ? final.rssBytes - baseline.rssBytes : Number.POSITIVE_INFINITY; + const pssTotalGrowthBytes = + baseline && final && baseline.pssBytes >= 0 && final.pssBytes >= 0 + ? final.pssBytes - baseline.pssBytes + : null; + const probeFailures = probes.filter((probe) => !probe.ok); + const baselineProbeP99 = percentile( + probes.slice(0, 5).filter((probe) => probe.ok).map((probe) => probe.durationMs), + 0.99, + ); + const finalProbeP99 = percentile( + probes.slice(-5).filter((probe) => probe.ok).map((probe) => probe.durationMs), + 0.99, + ); + + if (operationErrors.length > 0) { + failures.push(`${operationErrors.length} lifecycle operation(s) failed`); + } + if (!baseline || !final) failures.push("missing baseline or final process census"); + // Plateau slopes are the primary memory-leak gate. + if (steadySlopeBytesPerCycle > rssSlopeLimit) { + failures.push( + `steady-state RSS slope ${Math.round(steadySlopeBytesPerCycle)} B/cycle exceeds ${rssSlopeLimit}`, + ); + } + if (teardownRssSlope > rssSlopeLimit) { + failures.push( + `post-teardown RSS slope ${Math.round(teardownRssSlope)} B/sample exceeds ${rssSlopeLimit}`, + ); + } + if (teardownPssSlope > rssSlopeLimit) { + failures.push( + `post-teardown PSS slope ${Math.round(teardownPssSlope)} B/sample exceeds ${rssSlopeLimit}`, + ); + } + // Total-growth ceilings are provisional/calibrated; PSS is the tighter signal. + if (rssTotalGrowthBytes > rssTotalLimit) { + failures.push(`RSS growth ${rssTotalGrowthBytes} B exceeds ${rssTotalLimit}`); + } + if (pssTotalGrowthBytes !== null && pssTotalGrowthBytes > pssTotalLimit) { + failures.push(`PSS growth ${pssTotalGrowthBytes} B exceeds ${pssTotalLimit}`); + } + if (baseline && final && final.fdCount > baseline.fdCount) { + failures.push(`fd count grew from ${baseline.fdCount} to ${final.fdCount}`); + } + if (baseline && final && final.threadCount > baseline.threadCount) { + failures.push( + `thread count grew from ${baseline.threadCount} to ${final.threadCount}`, + ); + } + if (baseline && final && final.pidCount > baseline.pidCount) { + failures.push(`process count grew from ${baseline.pidCount} to ${final.pidCount}`); + } + if (probeFailures.length > 0) failures.push(`${probeFailures.length} sentinel probes failed`); + if (baselineProbeP99 > 0 && finalProbeP99 > baselineProbeP99 * 2) { + failures.push( + `post-pressure sentinel p99 ${finalProbeP99}ms exceeds 2x baseline ${baselineProbeP99}ms`, + ); + } + for (const sample of samples) { + const expected = sample.shape === "steady" ? batchSize + 1 : 1; + if (sample.activeVmCount !== expected) { + failures.push( + `${sample.shape} cycle ${sample.cycle} retained ${sample.activeVmCount} VMs; expected ${expected}`, + ); + break; + } + } + const cgroupAfter = cgroupSnapshot(); + const oomKillsBefore = Number(cgroupBefore["memory.events.oom_kill"] ?? 0); + const oomKillsAfter = Number(cgroupAfter["memory.events.oom_kill"] ?? 0); + if (oomKillsAfter > oomKillsBefore) { + failures.push(`container recorded ${oomKillsAfter - oomKillsBefore} OOM kill(s)`); + } + + const artifact = { + runId, + lane: "vm-lifecycle-churn-leak", + verdict: failures.length === 0 ? "pass" : "fail", + failures, + operationErrors, + config: { + cyclesPerShape: cycles, + batchSize, + settleMs, + rssSlopeLimit, + rssTotalLimit, + pssTotalLimit, + }, + provenance: runtimeProvenance(), + metrics: { + steadySlopeBytesPerCycle, + teardownRssSlope, + teardownPssSlope, + rssTotalGrowthBytes, + pssTotalGrowthBytes, + baselineProbeP99, + finalProbeP99, + probeFailures: probeFailures.length, + }, + baseline, + final, + afterDispose, + teardownSeries, + samples, + probes, + cgroupBefore, + cgroupAfter, + }; + const path = writeArtifact("churn", runId, artifact); + console.log(JSON.stringify({ verdict: artifact.verdict, failures, path, metrics: artifact.metrics })); + if (failures.length > 0) process.exitCode = 1; +} diff --git a/packages/load-tests/src/local/limit-matrix.ts b/packages/load-tests/src/local/limit-matrix.ts new file mode 100644 index 0000000000..8685574dec --- /dev/null +++ b/packages/load-tests/src/local/limit-matrix.ts @@ -0,0 +1,578 @@ +// Deterministic adversarial limit matrix (docs-internal/load-testing.md, lane A +// / checklist E). Generalises the single process-count probe into a table of +// resource-limit probes that share one sidecar + sentinel and one survival +// contract. Each probe overshoots its configured cap and asserts: +// 1. the configured boundary fires (typed limit error or Linux errno), +// 2. a near-limit host warning appears where that contract applies, +// 3. the shared sidecar stays ready and the sentinel keeps progressing, +// 4. the attacker VM is disposed and accounting returns to sentinel-only, +// 5. a fresh VM can still be created, and no container OOM occurred. +// +// Every probe overshoots in a single guest run: the first ~cap attempts succeed +// and the remainder are rejected, exercising below/at/above the boundary at once. +import { + AgentOs, + type AgentOsLimits, + type AgentOsSidecar, + type LimitWarning, +} from "@rivet-dev/agentos-core"; +import { + cgroupSnapshot, + errorText, + newRunId, + percentile, + runtimeProvenance, + sampleProcessTree, + sleep, + type TimedProbe, + withTimeout, + writeArtifact, +} from "../common.js"; + +interface GuestOutcome { + index?: number; + status: "spawned" | "error" | "timeout"; + code?: string | null; + message?: string; + writtenBytes?: number; +} + +interface LimitProbe { + name: string; + /** Human-readable configuration path named in the verdict. */ + limitPath: string; + cap: number; + attempts: number; + limits: AgentOsLimits; + /** CommonJS guest that overshoots and prints AGENTOS_LIMIT_RESULT=. */ + guest: string; + /** Matches the errno / typed-limit text a correct rejection must carry. */ + rejectionPattern: RegExp; + /** Some limits emit a near-threshold warning; count-of-1 fill jumps do not. */ + expectWarning: boolean; + /** + * "count": guest attempts N units and prints per-attempt outcomes; some are + * rejected. "kill": the single offending execution must be TERMINATED by the + * limit (typed error / non-zero exit), not run to a generic timeout. + * "crash-observe": run the workload and only assert the sidecar SURVIVES + * (fresh VM works, no SidecarProcessExited) — used to map crash-class vectors. + */ + kind?: "count" | "kill" | "crash-observe"; +} + +function processStormGuest(attempts: number): string { + return ` +const { spawn } = require("node:child_process"); +const attempts = ${attempts}; +const children = []; +function start(index) { + return new Promise((resolve) => { + let settled = false; let child; + const finish = (v) => { if (settled) return; settled = true; resolve(v); }; + try { child = spawn(process.execPath, ["-e", "setTimeout(()=>{},30000)"], { stdio: "ignore" }); } + catch (e) { finish({ index, status: "error", code: e.code ?? null, message: e.message }); return; } + children.push(child); + child.once("spawn", () => finish({ index, status: "spawned", pid: child.pid })); + child.once("error", (e) => finish({ index, status: "error", code: e.code ?? null, message: e.message })); + setTimeout(() => finish({ index, status: "timeout" }), 3000); + }); +} +(async () => { + const outcomes = await Promise.all(Array.from({ length: attempts }, (_, i) => start(i))); + for (const c of children) { if (c.pid) { try { c.kill("SIGKILL"); } catch {} } } + await new Promise((r) => setTimeout(r, 100)); + process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify(outcomes) + "\\n"); +})(); +`; +} + +function openFdGuest(attempts: number): string { + return ` +const fs = require("node:fs"); +const attempts = ${attempts}; +const fds = []; const outcomes = []; +for (let i = 0; i < attempts; i++) { + try { const fd = fs.openSync("/tmp/fdprobe_" + i, "w"); fds.push(fd); outcomes.push({ index: i, status: "spawned" }); } + catch (e) { outcomes.push({ index: i, status: "error", code: e.code ?? null, message: e.message }); } +} +for (const fd of fds) { try { fs.closeSync(fd); } catch {} } +process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify(outcomes) + "\\n"); +`; +} + +function socketGuest(attempts: number): string { + return ` +const net = require("node:net"); +const attempts = ${attempts}; +const servers = []; const outcomes = []; +function tryListen(i) { + return new Promise((resolve) => { + let settled = false; + const finish = (v) => { if (settled) return; settled = true; resolve(v); }; + let s; + try { s = net.createServer(); } catch (e) { finish({ index: i, status: "error", code: e.code ?? null, message: e.message }); return; } + // Permanent no-op error handler so a late async 'error' never crashes the + // process (the resource-limit rejection can surface after we settle). + s.on("error", (e) => { if (!settled) finish({ index: i, status: "error", code: e.code ?? null, message: e.message }); }); + try { s.listen(0, "127.0.0.1", () => { servers.push(s); finish({ index: i, status: "spawned" }); }); } + catch (e) { finish({ index: i, status: "error", code: e.code ?? null, message: e.message }); } + setTimeout(() => finish({ index: i, status: "timeout" }), 2000); + }); +} +process.on("uncaughtException", () => {}); // resource-limit errors may surface async post-settle +(async () => { + for (let i = 0; i < attempts; i++) { outcomes.push(await tryListen(i)); } + for (const s of servers) { try { s.close(); } catch {} } + await new Promise((r) => setTimeout(r, 50)); + process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify(outcomes) + "\\n"); +})(); +`; +} + +function filesystemGuest(capBytes: number): string { + // Attempt to write ~4x the byte cap in 1 MiB chunks to a single file. + const chunks = Math.ceil((capBytes * 4) / (1024 * 1024)); + return ` +const fs = require("node:fs"); +const chunk = Buffer.alloc(1024 * 1024, 65); +let written = 0; let error = null; +try { + const fd = fs.openSync("/tmp/fill.bin", "w"); + for (let i = 0; i < ${chunks}; i++) { fs.writeSync(fd, chunk); written += chunk.length; } + fs.closeSync(fd); +} catch (e) { error = { code: e.code ?? null, message: e.message }; } +const outcomes = error + ? [{ status: "error", code: error.code, message: error.message, writtenBytes: written }] + : [{ status: "spawned", writtenBytes: written }]; +process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify(outcomes) + "\\n"); +`; +} + +// LT-011 crash-class vectors: does the sidecar crash on paths/APIs OTHER than +// the confirmed chunked-write-to-/tmp? maxFilesystemBytes is set high so ENOSPC +// does not preempt the crash — a sidecar death is the finding. +function fsWorkspaceGuest(): string { + return ` +const fs = require("node:fs"); +const chunk = Buffer.alloc(1024 * 1024, 65); +try { const fd = fs.openSync("/workspace/fill.bin", "w"); for (let i=0;i<48;i++) fs.writeSync(fd, chunk); fs.closeSync(fd); + process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify([{status:"spawned"}]) + "\\n"); } +catch (e) { process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify([{status:"error",code:e.code??null,message:String(e.message).slice(0,120)}]) + "\\n"); } +`; +} +function fsWriteFileGuest(): string { + return ` +const fs = require("node:fs"); +try { fs.writeFileSync("/tmp/big.bin", Buffer.alloc(64*1024*1024, 66)); + process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify([{status:"spawned"}]) + "\\n"); } +catch (e) { process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify([{status:"error",code:e.code??null,message:String(e.message).slice(0,120)}]) + "\\n"); } +`; +} +function fsPwriteOffsetGuest(): string { + return ` +const fs = require("node:fs"); +const chunk = Buffer.alloc(1024 * 1024, 67); +try { const fd = fs.openSync("/tmp/sparse.bin", "w"); fs.writeSync(fd, chunk, 0, chunk.length, 64*1024*1024); fs.closeSync(fd); + process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify([{status:"spawned"}]) + "\\n"); } +catch (e) { process.stdout.write("AGENTOS_LIMIT_RESULT=" + JSON.stringify([{status:"error",code:e.code??null,message:String(e.message).slice(0,120)}]) + "\\n"); } +`; +} + +// Kill-style guests: each drives one execution into a JS-runtime limit. +const jsCpuGuest = `let x = 0; while (true) { x += Math.sqrt(x + 1); if (x < 0) break; }`; +const jsHeapGuest = `const a = []; while (true) { a.push(Buffer.alloc(1024 * 1024, 1)); }`; +const jsOutputGuest = `const b = Buffer.alloc(1024 * 1024, 65); for (let i = 0; i < 64; i++) { process.stdout.write(b); }`; +const compoundCpuOutputGuest = `const b = Buffer.alloc(256 * 1024, 66); let x = 0; while (true) { for (let i = 0; i < 1000; i++) x += Math.sqrt(x + 1); process.stdout.write(b); }`; + +function buildProbes(): LimitProbe[] { + const fsCap = 8 * 1024 * 1024; + return [ + { + name: "processes", + limitPath: "limits.resources.maxProcesses", + cap: 8, + attempts: 32, + limits: { resources: { maxProcesses: 8 } }, + guest: processStormGuest(32), + rejectionPattern: + /EAGAIN|resource temporarily unavailable|maxProcesses|process(?:es)?[^\n]*limit|ERR_AGENTOS/i, + expectWarning: true, + }, + { + name: "open-fds", + limitPath: "limits.resources.maxOpenFds", + cap: 64, + attempts: 160, + limits: { resources: { maxOpenFds: 64 } }, + guest: openFdGuest(160), + rejectionPattern: /EMFILE|ENFILE|too many open files|maxOpenFds|open[_ ]?fds?|ERR_AGENTOS/i, + // fd near-limit warnings reach sidecar stderr only, not onLimitWarning (LT-010). + expectWarning: false, + }, + { + name: "sockets", + limitPath: "limits.resources.maxSockets", + cap: 32, + attempts: 96, + limits: { resources: { maxSockets: 32 } }, + guest: socketGuest(96), + rejectionPattern: + /EMFILE|ENFILE|ENOBUFS|EADDRNOTAVAIL|too many|maxSockets|socket[^\n]*limit|ERR_AGENTOS_RESOURCE_LIMIT|ERR_AGENTOS/i, + // socket near-limit warnings reach sidecar stderr only, not onLimitWarning (LT-010). + expectWarning: false, + }, + { + name: "js-cpu", + limitPath: "limits.jsRuntime.cpuTimeLimitMs", + cap: 2_000, + attempts: 1, + limits: { jsRuntime: { cpuTimeLimitMs: 2_000, wallClockLimitMs: 0 } }, + guest: jsCpuGuest, + rejectionPattern: /cpu|CPU|time limit|watchdog|terminat|ERR_AGENTOS/i, + expectWarning: false, + kind: "kill", + }, + { + name: "js-wallclock", + limitPath: "limits.jsRuntime.wallClockLimitMs", + cap: 2_000, + attempts: 1, + limits: { jsRuntime: { wallClockLimitMs: 2_000 } }, + guest: jsCpuGuest, + rejectionPattern: /wall|clock|time limit|watchdog|terminat|deadline|ERR_AGENTOS/i, + expectWarning: false, + kind: "kill", + }, + { + name: "js-heap", + limitPath: "limits.jsRuntime.v8HeapLimitMb", + cap: 64, + attempts: 1, + limits: { jsRuntime: { v8HeapLimitMb: 64 } }, + guest: jsHeapGuest, + rejectionPattern: /heap|memory|out of memory|oom|allocation|ERR_AGENTOS/i, + expectWarning: false, + kind: "kill", + }, + { + name: "js-output", + limitPath: "limits.jsRuntime.capturedOutputLimitBytes", + cap: 1024 * 1024, + attempts: 1, + limits: { jsRuntime: { capturedOutputLimitBytes: 1024 * 1024 } }, + guest: jsOutputGuest, + rejectionPattern: /output|captured|stdout|byte|limit|ERR_AGENTOS/i, + expectWarning: false, + kind: "kill", + }, + { + // Compound: burn CPU and flood stdout at once; whichever cap is lower + // must fire, and the offending execution must be terminated. + name: "compound-cpu-output", + limitPath: "limits.jsRuntime.cpuTimeLimitMs+capturedOutputLimitBytes", + cap: 3_000, + attempts: 1, + limits: { + jsRuntime: { + cpuTimeLimitMs: 3_000, + wallClockLimitMs: 0, + capturedOutputLimitBytes: 2 * 1024 * 1024, + }, + }, + guest: compoundCpuOutputGuest, + rejectionPattern: + /cpu|CPU|time limit|output|captured|stdout|byte|limit|watchdog|terminat|ERR_AGENTOS/i, + expectWarning: false, + kind: "kill", + }, + // LT-011 crash-class mapping (run each ISOLATED via LOAD_TEST_MATRIX_ONLY; + // high maxFilesystemBytes so a crash — not ENOSPC — is the signal). + { + name: "fs-crash-workspace", + limitPath: "LT-011 vector: /workspace chunked write", + cap: 256 * 1024 * 1024, + attempts: 1, + limits: { resources: { maxFilesystemBytes: 256 * 1024 * 1024 } }, + guest: fsWorkspaceGuest(), + rejectionPattern: /.*/, + expectWarning: false, + kind: "crash-observe", + }, + { + name: "fs-crash-writefile64", + limitPath: "LT-011 vector: single 64MiB writeFileSync to /tmp", + cap: 256 * 1024 * 1024, + attempts: 1, + limits: { resources: { maxFilesystemBytes: 256 * 1024 * 1024 } }, + guest: fsWriteFileGuest(), + rejectionPattern: /.*/, + expectWarning: false, + kind: "crash-observe", + }, + { + name: "fs-crash-pwrite-offset", + limitPath: "LT-011 vector: pwrite at 64MiB offset (/tmp sparse)", + cap: 256 * 1024 * 1024, + attempts: 1, + limits: { resources: { maxFilesystemBytes: 256 * 1024 * 1024 } }, + guest: fsPwriteOffsetGuest(), + rejectionPattern: /.*/, + expectWarning: false, + kind: "crash-observe", + }, + { + // Runs LAST: this probe crashes the shared sidecar (LT-011), so it must + // not precede probes that need a healthy sidecar. + name: "filesystem-bytes", + limitPath: "limits.resources.maxFilesystemBytes", + cap: fsCap, + attempts: 1, + limits: { resources: { maxFilesystemBytes: fsCap } }, + guest: filesystemGuest(fsCap), + rejectionPattern: + /ENOSPC|EDQUOT|EFBIG|no space|disk quota|maxFilesystemBytes|filesystem[^\n]*limit|ERR_AGENTOS/i, + expectWarning: false, + }, + ]; +} + +async function probeSentinel(vm: AgentOs): Promise { + const startedAtMs = Date.now(); + try { + const result = await withTimeout( + "sentinel exec", + vm.execArgv("node", ["-e", 'process.stdout.write("sentinel-ok")'], { timeout: 5_000 }), + 7_000, + ); + if (result.exitCode !== 0 || result.stdout !== "sentinel-ok") { + throw new Error(`sentinel exit=${result.exitCode} stderr=${JSON.stringify(result.stderr)}`); + } + return { startedAtMs, durationMs: Date.now() - startedAtMs, ok: true }; + } catch (error) { + return { startedAtMs, durationMs: Date.now() - startedAtMs, ok: false, error: errorText(error) }; + } +} + +function parseOutcomes(stdout: string): GuestOutcome[] { + const line = stdout.split("\n").find((l) => l.startsWith("AGENTOS_LIMIT_RESULT=")); + if (!line) return []; + try { + const parsed = JSON.parse(line.slice("AGENTOS_LIMIT_RESULT=".length)); + return Array.isArray(parsed) ? parsed : []; + } catch { + return []; + } +} + +interface ProbeResult { + name: string; + limitPath: string; + verdict: "pass" | "fail"; + failures: string[]; + cap: number; + attempts: number; + spawned: number; + rejected: number; + rejectionSample: string; + warnings: number; + sentinelFailuresDuring: number; + freshVmOk: boolean; +} + +async function runProbe( + sidecar: AgentOsSidecar, + sentinel: AgentOs, + probe: LimitProbe, +): Promise { + const failures: string[] = []; + const warnings: LimitWarning[] = []; + let attacker: AgentOs | undefined; + let sentinelFailuresDuring = 0; + let spawned = 0; + let rejected = 0; + let rejectionSample = ""; + let freshVmOk = false; + + try { + attacker = await AgentOs.create({ + defaultSoftware: false, + sidecar: { kind: "explicit", handle: sidecar }, + limits: probe.limits, + onLimitWarning: (w) => warnings.push(w), + }); + + let attackExit: number | undefined; + let attackStderr = ""; + let attackStdout = ""; + let attackThrew: string | undefined; + try { + const attack = await withTimeout( + `${probe.name} storm`, + attacker.execArgv("node", ["-e", probe.guest], { timeout: 25_000 }), + 30_000, + ); + attackExit = attack.exitCode; + attackStderr = attack.stderr; + attackStdout = attack.stdout; + } catch (error) { + attackThrew = errorText(error); + } + // Sentinel must have kept working during pressure. + const during = await probeSentinel(sentinel); + if (!during.ok) sentinelFailuresDuring += 1; + + if (probe.kind === "crash-observe") { + // Map crash-class vectors: the ONLY failure is the sidecar dying. + const evidence = `${attackThrew ?? ""} exit=${attackExit ?? "?"} ${attackStderr}`.slice(0, 240); + rejectionSample = `${parseOutcomes(attackStdout).map((o) => o.status).join(",")} | ${evidence}`; + if (attackThrew && /SidecarProcessExited|Kernel\(|EBADF|sidecar process exited/i.test(attackThrew)) { + failures.push(`SIDECAR CRASHED on ${probe.name}: ${attackThrew.slice(0, 160)}`); + } + // freshVmOk / sentinel checks below catch a crash even if this exec + // returned before the sidecar fully died. + } else if ((probe.kind ?? "count") === "kill") { + // The offending execution must be terminated by the typed limit — not + // left to run into the generic guest/exec timeout. + const evidence = `${attackThrew ?? ""} exit=${attackExit ?? "?"} ${attackStderr}`; + const terminated = attackThrew !== undefined || (attackExit !== undefined && attackExit !== 0); + rejectionSample = evidence.slice(0, 240); + if (!terminated) { + failures.push(`${probe.limitPath} did not terminate the offending execution (exit=0)`); + } else if (/timed out after \d+ms/.test(evidence) || !probe.rejectionPattern.test(evidence)) { + failures.push(`termination did not name ${probe.limitPath} (generic/untyped): ${rejectionSample}`); + } else { + rejected = 1; + } + } else { + const outcomes = parseOutcomes(attackStdout); + const rejections = outcomes.filter((o) => o.status === "error"); + spawned = outcomes.filter((o) => o.status === "spawned").length; + rejected = rejections.length; + rejectionSample = JSON.stringify(rejections.slice(0, 3)); + if (attackExit !== 0 && attackExit !== undefined) { + failures.push(`attack parent exited ${attackExit}: ${attackStderr.slice(0, 200)}`); + } + if (attackThrew) failures.push(`attack threw: ${attackThrew.slice(0, 200)}`); + if (rejected === 0) { + failures.push(`no attempt was rejected (${probe.limitPath} did not fire)`); + } else if (!probe.rejectionPattern.test(rejectionSample)) { + failures.push(`rejection did not name ${probe.limitPath}: ${rejectionSample.slice(0, 200)}`); + } + if (probe.expectWarning && warnings.length === 0) { + failures.push(`no near-limit warning for ${probe.limitPath}`); + } + } + } catch (error) { + failures.push(`probe threw: ${errorText(error)}`); + } finally { + if (attacker) await attacker.dispose().catch(() => {}); + } + + // A fresh VM must still be creatable + runnable after the attack. + try { + const fresh = await AgentOs.create({ + defaultSoftware: false, + sidecar: { kind: "explicit", handle: sidecar }, + }); + const r = await withTimeout( + "fresh vm", + fresh.execArgv("node", ["-e", 'process.stdout.write("fresh-ok")'], { timeout: 5_000 }), + 7_000, + ); + freshVmOk = r.exitCode === 0 && r.stdout === "fresh-ok"; + await fresh.dispose().catch(() => {}); + } catch (error) { + failures.push(`fresh VM after ${probe.name} failed: ${errorText(error)}`); + } + if (!freshVmOk) failures.push(`fresh VM after ${probe.name} did not run`); + + return { + name: probe.name, + limitPath: probe.limitPath, + verdict: failures.length === 0 ? "pass" : "fail", + failures, + cap: probe.cap, + attempts: probe.attempts, + spawned, + rejected, + rejectionSample, + warnings: warnings.length, + sentinelFailuresDuring, + freshVmOk, + }; +} + +export async function runLimitMatrix(): Promise { + const runId = newRunId("limit-matrix"); + // Optional isolation: LOAD_TEST_MATRIX_ONLY=filesystem-bytes runs one probe + // against a fresh sidecar (used to isolate a cross-probe interaction). + const only = process.env.LOAD_TEST_MATRIX_ONLY; + const probes = only + ? buildProbes().filter((p) => only.split(",").includes(p.name)) + : buildProbes(); + if (probes.length === 0) throw new Error(`LOAD_TEST_MATRIX_ONLY matched no probe: ${only}`); + let sidecar: AgentOsSidecar | undefined; + let sentinel: AgentOs | undefined; + const results: ProbeResult[] = []; + const sentinelProbes: TimedProbe[] = []; + const failures: string[] = []; + const before = sampleProcessTree(); + + try { + sidecar = await AgentOs.createSidecar({ sidecarId: `load-matrix-${runId}` }); + sentinel = await AgentOs.create({ + defaultSoftware: false, + sidecar: { kind: "explicit", handle: sidecar }, + }); + for (let i = 0; i < 3; i += 1) sentinelProbes.push(await probeSentinel(sentinel)); + + for (const probe of probes) { + results.push(await runProbe(sidecar, sentinel, probe)); + sentinelProbes.push(await probeSentinel(sentinel)); + const desc = sidecar.describe(); + if (desc.state !== "ready") failures.push(`sidecar state ${desc.state} after ${probe.name}`); + if (desc.activeVmCount !== 1) { + failures.push(`after ${probe.name}: activeVmCount ${desc.activeVmCount}, expected sentinel-only`); + } + } + } catch (error) { + failures.push(`top-level: ${errorText(error)}`); + } finally { + if (sentinel) await sentinel.dispose().catch(() => {}); + if (sidecar) await sidecar.dispose().catch(() => {}); + } + + await sleep(300); + const final = sampleProcessTree(); + const sentinelFailures = sentinelProbes.filter((p) => !p.ok).length; + if (sentinelFailures > 0) failures.push(`${sentinelFailures} sentinel probes failed`); + const oomKills = Number(cgroupSnapshot()["memory.events.oom_kill"] ?? 0); + if (oomKills > 0) failures.push(`container recorded ${oomKills} OOM kill(s)`); + for (const r of results) if (r.verdict === "fail") failures.push(`probe ${r.name}: ${r.failures.join("; ")}`); + + const artifact = { + runId, + lane: "adversarial-limit-matrix", + verdict: failures.length === 0 ? "pass" : "fail", + failures, + provenance: runtimeProvenance(), + results, + sentinel: { + probes: sentinelProbes.length, + failures: sentinelFailures, + p99Ms: percentile(sentinelProbes.filter((p) => p.ok).map((p) => p.durationMs), 0.99), + }, + processTree: { before, final }, + cgroupFinal: cgroupSnapshot(), + }; + const path = writeArtifact("limit-matrix", runId, artifact); + console.log( + JSON.stringify({ + verdict: artifact.verdict, + failures: failures.slice(0, 8), + path, + probes: results.map((r) => ({ name: r.name, verdict: r.verdict, spawned: r.spawned, rejected: r.rejected })), + }), + ); + if (failures.length > 0) process.exitCode = 1; +} diff --git a/packages/load-tests/src/local/limit-survival.ts b/packages/load-tests/src/local/limit-survival.ts new file mode 100644 index 0000000000..7d6fec773e --- /dev/null +++ b/packages/load-tests/src/local/limit-survival.ts @@ -0,0 +1,238 @@ +import { + AgentOs, + type AgentOsSidecar, + type LimitWarning, +} from "@rivet-dev/agentos-core"; +import { + cgroupSnapshot, + errorText, + integerEnv, + newRunId, + percentile, + runtimeProvenance, + sampleProcessTree, + sleep, + type TimedProbe, + withTimeout, + writeArtifact, +} from "../common.js"; + +const PROCESS_LIMIT_PATTERN = + /EAGAIN|resource temporarily unavailable|maxProcesses|process(?:es)?[^\n]*limit|ERR_AGENTOS/i; + +// CommonJS on purpose: AgentOS's `node` runtime does not parse +// `--input-type=module` as a flag (it treats it as the entry module path, see +// LT-006), so the guest workload uses `require` + an async IIFE and is passed +// as a plain `node -e