From acd331f5fecf9ba8d3a4d49eac42aa2ef54fefdc Mon Sep 17 00:00:00 2001 From: Nathan Flurry Date: Mon, 20 Jul 2026 01:22:05 -0700 Subject: [PATCH] docs: document sidecar stderr forwarding and host-backed filesystem budget Both are user-facing behavior changes that shipped without docs. - debugging.mdx: sidecar logs are forwarded to the host's stderr as they arrive, so a still-running sidecar's warnings are visible rather than only replayed if it dies. Notes the bounded postmortem excerpt (first 16 KiB + last 48 KiB, truncation stated inline) and points at AGENTOS_LOG_FILE for the complete log. - resource-limits.mdx: maxFilesystemBytes now also governs host-backed paths, which never reach the virtual filesystem and so are checked separately on write. Documents the honest caveat that this separate check is per-file while the VFS side is per-VM total, so a guest can still exceed its budget by spreading data across many host-backed files, and recommends bounding the host directory until that check aggregates per VM. The interim AGENTOS_SIDECAR_STDERR knob is intentionally not documented: it was removed in favor of unconditional forwarding to match the Rust client. Not validated with `pnpm --dir website build`: `website` is excluded from the workspace in this checkout by a pre-existing local workaround, so the build cannot run here. MDX table/fence structure was checked instead. --- docs-internal/load-testing-issues.md | 2 +- website/src/content/docs/docs/debugging.mdx | 11 ++++++ .../src/content/docs/docs/resource-limits.mdx | 19 ++++++++-- website/src/generated/registry.json | 36 +++++++++++++++++++ 4 files changed, 65 insertions(+), 3 deletions(-) diff --git a/docs-internal/load-testing-issues.md b/docs-internal/load-testing-issues.md index b9fa7689fd..055087ae49 100644 --- a/docs-internal/load-testing-issues.md +++ b/docs-internal/load-testing-issues.md @@ -31,7 +31,7 @@ logged in `~/.agents/friction/`. | LT-015 | Open (by design) | Runtime | Excess concurrent guest executions are REJECTED, not queued. | Creating 50–200 VMs on one sidecar (8-CPU container) and firing `execArgv` in all concurrently: only 8 execute; the rest get `sidecar rejected request N: execution_error: ... ERR_AGENTOS_VM_EXECUTOR_LIMIT: active V8 executors reached limit of 8; raise runtime.executor.maxActiveVms`. So the executor pool (= CPU count, LT-008) is a hard concurrency cap and surplus executions fail immediately rather than waiting in a queue. The sidecar SURVIVES (reconciles fd/thread/pid to baseline, no OOM), and the rejection is typed/actionable — but a caller must bound execution concurrency to the CPU count or handle rejections. Also observed: a possible concurrent-VM admission cap (~105 coexisting VMs at the 200-VM test) — under investigation. | | LT-021 | Info | Compute/GCP | Rivet Compute runs on **GKE Autopilot**, multi-region shared clusters. | Found via gcloud (`nathan@rivet.gg`, project `cloud-prod-474518`): clusters `ap-southeast-1-engine-autopilot` (18 nodes), `eu-central-1` (12), `us-east-1` (7), `us-west-1` (16), `cloud-api-autopilot` (4). The `rivet-engine` namespace holds the control plane (nats/guard/worker/pegboard); serverless actor runners are ephemeral (no per-deployment persistent pods). So the "autoscaler" is GKE Autopilot node autoscaling (pod-request-driven bin-packing), and node-count is the scale signal — but clusters are shared across all Rivet customers, so isolating one deployment's impact requires watching for its runner pods live during a ramp. | | LT-022 | **Open (HIGH — soak/leak)** | Runtime | Sustained VM churn grows sidecar memory ~0.5 MiB per VM lifecycle without plateauing; reclaimed only on full sidecar teardown. | 200-cycle soak: sequential RSS climbed 226→323 MiB monotonically; steady-state slope was ~0.53 MB/cycle at 30 cycles AND ~0.50 MB/cycle at 200 cycles — a plateauing warmup's average slope SHRINKS with run length, so a slope that stays constant across 30→200 cycles indicates a real slow leak of per-lifecycle state (generation records / caches / ledgers) that VM dispose does not reclaim. Post-teardown slope is 0 (freed when the sidecar exits), so it is not a classic unbounded leak — but a long-lived shared sidecar servicing millions of VM lifecycles would grow until restart. (The soak's op/sentinel failures were separate executor-contention noise — LT-015 — from a burst wider than the executor pool; a clean batch=1 soak is running to confirm the slope in isolation.) Owner: audit per-VM-lifecycle state not released on dispose. **CONFIRMED as a real leak by a clean soak** (batch=1 so no executor contention, 300 cycles/shape ≈ 900 VM lifecycles, zero op errors): steady slope 0.495 MB/cycle vs 0.50 at 200 cycles and 0.53 at 30 — **constant across run lengths**, which is the leak signature (a plateauing warmup's average slope decreases). 461 MiB retained over ~900 lifecycles ≈ 0.51 MiB per VM lifecycle; fd/thread/pid flat; released only on sidecar exit. | -| LT-023 | **Open (MEDIUM — observability)** — mitigated behind a flag | Client (`packages/runtime-core/src/process.ts`) | A LIVE sidecar's stderr is invisible to the host: it is buffered into `stderrChunks` and only replayed via `SidecarProcessExited`/`SidecarProcessError`, i.e. **only if the sidecar dies**. Every `tracing::error!`/`WARN_*` emitted by a sidecar that survives a guest-triggered failure is silently discarded. | Directly blocked this investigation: a guest write failing inside the sidecar produced ZERO host-visible output (`RUST_LOG=debug` changed nothing) — the only symptom was a 31 s client-side timeout. This violates the repo rule that host-visible warnings/errors must reach stderr/log rather than staying trapped. **Mitigation applied:** `AGENTOS_SIDECAR_STDERR=1` now forwards sidecar stderr to host stderr as it arrives (still buffered for the exit path). That immediately surfaced the LT-024 root cause. A flag is a stopgap — the right fix is a structured, always-on host log/trace sink for sidecar diagnostics. | +| LT-023 | **FIXED** (no longer behind a flag) | Client (`packages/runtime-core/src/process.ts`) | A LIVE sidecar's stderr is invisible to the host: it is buffered into `stderrChunks` and only replayed via `SidecarProcessExited`/`SidecarProcessError`, i.e. **only if the sidecar dies**. Every `tracing::error!`/`WARN_*` emitted by a sidecar that survives a guest-triggered failure is silently discarded. | Directly blocked this investigation: a guest write failing inside the sidecar produced ZERO host-visible output (`RUST_LOG=debug` changed nothing) — the only symptom was a 31 s client-side timeout. This violates the repo rule that host-visible warnings/errors must reach stderr/log rather than staying trapped. **Fixed:** sidecar stderr is now forwarded to host stderr unconditionally, matching the Rust client which already spawned with `Stdio::inherit()` (the interim `AGENTOS_SIDECAR_STDERR` flag was removed — gating it left the two clients behaviorally different for identical code). The retained postmortem copy is bounded (first 16 KiB + last 48 KiB, truncation stated inline) because an append-only buffer fed by guest-triggered diagnostics is a host memory leak. Documented in `website/src/content/docs/docs/debugging.mdx`. | | LT-024 | **Open (CRITICAL — guest file writes are broken; supersedes the LT-011 "fix")** | Runtime/VFS + native-sidecar | **Every** guest file write hangs for 31 s and fails with `ERR_AGENTOS_BRIDGE_CALL_TIMEOUT` instead of writing. The underlying `EBADF: bad file descriptor 1000000000` (LT-011's root cause) was never fixed — F-4 only stopped it from killing the process, converting a loud crash into a silent hang. | **Directly evidenced** (not inferred) once LT-023 was mitigated. Sidecar log during a single 4 KiB `fs.writeSync`: `level=error message="process-event pump failed for session; continuing to serve other sessions" session=session-1 error="EBADF: bad file descriptor 1000000000"`. Chain: guest write → process-event pump raises EBADF on `MAPPED_HOST_FD_START` → **F-4 catches it, logs, and `continue`s** → the guest's pending sync-RPC response is *never delivered* → guest blocks until `limits.reactor.operationDeadlineMs` (31 s) → `ERR_AGENTOS_BRIDGE_CALL_TIMEOUT`. Before F-4 the identical error propagated to `main` → `exit(1)` → the cross-tenant crash. **Size- and path-independent**, so this is not a quota/large-write issue: a sweep of 4 KiB / 64 KiB / 256 KiB / 512 KiB / 1 MiB writes to BOTH `/tmp` and `/workspace` stalled ~31 000 ms in all 10 combinations. This also fully explains the previously-unexplained `filesystem-bytes` probe result (`exit=137`, "maxFilesystemBytes did not fire"): the guest never completes a single write, so it is killed by the harness timeout long before any byte cap could be reached — the cap was never the issue. **Correction to my earlier claim:** reporting LT-011 as "fixed" was correct only about *blast radius* (verified: co-located actors survive, 8→8 on Compute) and wrong to imply the defect was resolved — swallowing the error also violates the repo's "never swallow errors silently" rule. Correct fix has two parts: (1) stop routing mapped host fds (>= `MAPPED_HOST_FD_START`) into the kernel fd table, and (2) make pump failures deliver a typed errno to the waiting caller instead of dropping its response. | | LT-020 | Open (by design) | Compute | On a `--cpu 1` runner, per-runner AgentOS execution concurrency is 1 (executor pool = CPU count, LT-008/015), so co-located actors calling `execArgv` serialize and time out under contention. | A single-actor gateway `execArgv` works (`POST /gateway//action/execArgv` `{"args":["node",["-e",...]]}` → `{"output":{"exitCode":0,"stdout":...}}`). But 25 co-located actors each calling `execArgv` on a `--cpu 1` runner all timed out (25 s) — the noisy-neighbor blast-radius test couldn't even establish a baseline (`survivorsBefore=0`). Implication: per-runner guest execution throughput scales with `--cpu`, not actor count; a realistic runner needs `--cpu >= 2` (or fewer actors/runner) for concurrent AgentOS work. Retried with 6 co-located actors. | | LT-018 | **Open (HIGH — scaling)** | Compute | A fast burst to hundreds of actors overwhelms Compute scale-up: a large fraction fail the health timeout. | With `--max-scale 8 --max-concurrent-actors 25` and a create-storm at concurrency 48, ramp 25→50→100→150→200: step 25 = all ready; step 50 = 25 ready / **25 failed**; 100 = 50/50; 150 = 56 ready / **94 failed**, create-to-ready **P99 57 s**; 200 = 76 ready / **124 failed**. `peakConnectable` = 25 (one runner's slot count). Deployment logs confirm runners DID scale across regions (`ap-southeast-1`, `eu-central-1`, `us-east-1`, `us-west-1` booted mid-ramp), so scale-up happens — but too slowly to absorb a burst: actors wait >60 s for a slot and time out. Cleanup was clean (0 leaked actors). Net: the managed pool scales, but burst admission of hundreds of actors is not absorbed — high failure rate + tens-of-seconds cold-start. Needs either slower client ramp, higher `--max-concurrent-actors`, or faster Rivet scale-up. **Characterized (gentle ramp — concurrency 6, 150 s timeout, 45 s holds): 25→all ready, 50→44/50, 100→91/100 ready (only ~9% fail), but slowest create-to-ready P99 = 108 s.** So it is "can't scale FAST", not "can't scale": Compute absorbs ~90% of 100 actors when the client is patient, but new-runner/region cold-start is ~100+ s, so a fast burst (conc 48, 60 s timeout) causes majority failures. Owner action: reduce cold-start latency and/or document a client back-pressure/ramp guidance for large actor fleets. | diff --git a/website/src/content/docs/docs/debugging.mdx b/website/src/content/docs/docs/debugging.mdx index 7e5513315b..d09338dbbc 100644 --- a/website/src/content/docs/docs/debugging.mdx +++ b/website/src/content/docs/docs/debugging.mdx @@ -41,6 +41,17 @@ The agentOS sidecar emits structured **logfmt** logs for request handling, netwo | `AGENTOS_LOG_FILE` | append logs to this file instead of stderr (never stdout, which carries the wire protocol) | | `RUST_LOG_{SPAN_NAME,SPAN_PATH,TARGET,LOCATION,MODULE_PATH,ANSI_COLOR}` | per-field toggles (`=1` to enable) | +Sidecar logs are written to the sidecar's stderr, and **both clients forward that +stream to the host process's stderr as it arrives** — so warnings and errors from +a sidecar that is still running are visible immediately, not only if it dies. + +The clients additionally retain a **bounded excerpt** of sidecar stderr (the +first 16 KiB and the most recent 48 KiB) to attach to `SidecarProcessExited` for +postmortem. Both ends are kept because the root cause is usually the *first* +error while the fatal one is the *last*. If the excerpt is truncated, the omitted +byte count is stated inline rather than silently shortening the output. Use +`AGENTOS_LOG_FILE` when you need the complete, untruncated log. + ```bash AGENTOS_LOG_LEVEL=debug AGENTOS_LOG_FILE=./sidecar.log RUST_LOG_FORMAT=logfmt node app.mjs ``` diff --git a/website/src/content/docs/docs/resource-limits.mdx b/website/src/content/docs/docs/resource-limits.mdx index 1f6d5f59a6..daf4a97b89 100644 --- a/website/src/content/docs/docs/resource-limits.mdx +++ b/website/src/content/docs/docs/resource-limits.mdx @@ -24,7 +24,7 @@ Set caps on the `limits` object in the `agentOS` config. Limits are grouped by s | `resources.maxProcesses` | Concurrent processes in the VM process table | Caps fork bombs and runaway spawning. New spawns fail with `EAGAIN`. | | `resources.maxOpenFds` | Open file descriptors | Exhausting the table fails with `EMFILE` / `ENFILE`. | | `resources.maxSockets` | Open sockets in the socket table | Bounds concurrent connections; excess `connect`/`accept` fail. | -| `resources.maxFilesystemBytes` | Total bytes stored in the virtual filesystem | Bounds VFS storage; writes past the budget fail with a no-space error. | +| `resources.maxFilesystemBytes` | Bytes stored by the VM, in the virtual filesystem **and** on host-backed paths | Default is `64 MiB`. Writes past the budget fail with a no-space error naming this limit. Host-backed paths (such as a writable `host_dir` mount) never reach the kernel VFS, so they are checked separately on write — see the caveat below. | | `resources.maxInodeCount` | Inodes retained by the virtual filesystem | Default is `16384`; creating another file or directory fails with a no-space error. This is the expected upper bound for filesystem-schema sizing and benchmarks. | | `resources.maxWasmFuel` | WASM execution budget | Bounds WASM execution work; unset means no explicit fuel budget. | | `resources.maxWasmMemoryBytes` | WASM linear memory, in bytes | Default is `128 MiB`. | @@ -58,10 +58,25 @@ Set caps on the `limits` object in the `agentOS` config. Limits are grouped by s - **WASM stack**: deep recursion throws a stack-overflow error in the guest, never a host crash. - **JavaScript CPU time**: CPU-bound loops terminate with a CPU-budget error once active JS CPU exceeds `jsRuntime.cpuTimeLimitMs`. - **JavaScript wall time**: awaiting or blocked JS terminates only when you set `jsRuntime.wallClockLimitMs`; the default is disabled for long-lived adapters. -- **Filesystem bytes**: writing past the VFS budget fails with a no-space error to the guest. +- **Filesystem bytes**: writing past the budget fails with a no-space error to the guest, naming `limits.resources.maxFilesystemBytes`. - **Counts (fds / processes / sockets)**: hitting a table cap returns the standard POSIX errno appropriate to that cap (`EMFILE`/`ENFILE`, `EAGAIN`, etc.). - **Durable ACP collections**: session, prompt, pending-permission, and terminal-outcome caps fail atomically with typed errors naming the exact `limits.acp.*` field to raise. Per-session caps are validated not to exceed their corresponding per-VM cap. +### Host-backed paths and `maxFilesystemBytes` + +Writes to host-backed paths — a writable `host_dir` mount, and other paths the +runtime maps straight to a host file — do not pass through the virtual +filesystem, so the VFS accounting that enforces `maxFilesystemBytes` never sees +them. Those writes are checked separately, at the point the guest writes. + +That separate check is currently **per file**, while the VFS side compares +against the VM's *total* usage. A guest can therefore stay under the cap on +every individual host-backed file and still exceed its overall budget by +spreading data across many files. If you rely on `maxFilesystemBytes` as a hard +storage bound for untrusted code, also bound the underlying host directory (for +example a dedicated filesystem or quota on the mount source) until the +host-backed check aggregates per VM. + ### WASM memory residency calls V8 owns the physical backing pages for WASM linear memory, and the runtime diff --git a/website/src/generated/registry.json b/website/src/generated/registry.json index 5f4e39b76d..4b656488e9 100644 --- a/website/src/generated/registry.json +++ b/website/src/generated/registry.json @@ -254,6 +254,42 @@ "status": "available", "image": "/images/registry/coreutils.svg" }, + { + "slug": "xfsprogs", + "title": "xfs_io", + "description": "Scriptable low-level filesystem I/O commands used by xfstests.", + "types": [ + "software" + ], + "category": "core", + "priority": 43, + "package": "@agentos-software/xfsprogs", + "status": "available" + }, + { + "slug": "acl", + "title": "POSIX ACL tools", + "description": "Inspect and modify POSIX access and default ACLs.", + "types": [ + "software" + ], + "category": "core", + "priority": 42, + "package": "@agentos-software/acl", + "status": "available" + }, + { + "slug": "attr", + "title": "attr", + "description": "Inspect and modify filesystem extended attributes.", + "types": [ + "software" + ], + "category": "core", + "priority": 41, + "package": "@agentos-software/attr", + "status": "available" + }, { "slug": "grep", "title": "grep",