docs: document sidecar stderr forwarding and host-backed filesystem budget

docs-internal/load-testing-issues.md

@@ -31,7 +31,7 @@ logged in `~/.agents/friction/`.
 | LT-015 | Open (by design) | Runtime | Excess concurrent guest executions are REJECTED, not queued. | Creating 50–200 VMs on one sidecar (8-CPU container) and firing `execArgv` in all concurrently: only 8 execute; the rest get `sidecar rejected request N: execution_error: ... ERR_AGENTOS_VM_EXECUTOR_LIMIT: active V8 executors reached limit of 8; raise runtime.executor.maxActiveVms`. So the executor pool (= CPU count, LT-008) is a hard concurrency cap and surplus executions fail immediately rather than waiting in a queue. The sidecar SURVIVES (reconciles fd/thread/pid to baseline, no OOM), and the rejection is typed/actionable — but a caller must bound execution concurrency to the CPU count or handle rejections. Also observed: a possible concurrent-VM admission cap (~105 coexisting VMs at the 200-VM test) — under investigation. |
 | LT-021 | Info | Compute/GCP | Rivet Compute runs on **GKE Autopilot**, multi-region shared clusters. | Found via gcloud (`nathan@rivet.gg`, project `cloud-prod-474518`): clusters `ap-southeast-1-engine-autopilot` (18 nodes), `eu-central-1` (12), `us-east-1` (7), `us-west-1` (16), `cloud-api-autopilot` (4). The `rivet-engine` namespace holds the control plane (nats/guard/worker/pegboard); serverless actor runners are ephemeral (no per-deployment persistent pods). So the "autoscaler" is GKE Autopilot node autoscaling (pod-request-driven bin-packing), and node-count is the scale signal — but clusters are shared across all Rivet customers, so isolating one deployment's impact requires watching for its runner pods live during a ramp. |
 | LT-022 | **Open (HIGH — soak/leak)** | Runtime | Sustained VM churn grows sidecar memory ~0.5 MiB per VM lifecycle without plateauing; reclaimed only on full sidecar teardown. | 200-cycle soak: sequential RSS climbed 226→323 MiB monotonically; steady-state slope was ~0.53 MB/cycle at 30 cycles AND ~0.50 MB/cycle at 200 cycles — a plateauing warmup's average slope SHRINKS with run length, so a slope that stays constant across 30→200 cycles indicates a real slow leak of per-lifecycle state (generation records / caches / ledgers) that VM dispose does not reclaim. Post-teardown slope is 0 (freed when the sidecar exits), so it is not a classic unbounded leak — but a long-lived shared sidecar servicing millions of VM lifecycles would grow until restart. (The soak's op/sentinel failures were separate executor-contention noise — LT-015 — from a burst wider than the executor pool; a clean batch=1 soak is running to confirm the slope in isolation.) Owner: audit per-VM-lifecycle state not released on dispose. **CONFIRMED as a real leak by a clean soak** (batch=1 so no executor contention, 300 cycles/shape ≈ 900 VM lifecycles, zero op errors): steady slope 0.495 MB/cycle vs 0.50 at 200 cycles and 0.53 at 30 — **constant across run lengths**, which is the leak signature (a plateauing warmup's average slope decreases). 461 MiB retained over ~900 lifecycles ≈ 0.51 MiB per VM lifecycle; fd/thread/pid flat; released only on sidecar exit. |
-| LT-023 | **Open (MEDIUM — observability)** — mitigated behind a flag | Client (`packages/runtime-core/src/process.ts`) | A LIVE sidecar's stderr is invisible to the host: it is buffered into `stderrChunks` and only replayed via `SidecarProcessExited`/`SidecarProcessError`, i.e. **only if the sidecar dies**. Every `tracing::error!`/`WARN_*` emitted by a sidecar that survives a guest-triggered failure is silently discarded. | Directly blocked this investigation: a guest write failing inside the sidecar produced ZERO host-visible output (`RUST_LOG=debug` changed nothing) — the only symptom was a 31 s client-side timeout. This violates the repo rule that host-visible warnings/errors must reach stderr/log rather than staying trapped. **Mitigation applied:** `AGENTOS_SIDECAR_STDERR=1` now forwards sidecar stderr to host stderr as it arrives (still buffered for the exit path). That immediately surfaced the LT-024 root cause. A flag is a stopgap — the right fix is a structured, always-on host log/trace sink for sidecar diagnostics. |
+| LT-023 | **FIXED** (no longer behind a flag) | Client (`packages/runtime-core/src/process.ts`) | A LIVE sidecar's stderr is invisible to the host: it is buffered into `stderrChunks` and only replayed via `SidecarProcessExited`/`SidecarProcessError`, i.e. **only if the sidecar dies**. Every `tracing::error!`/`WARN_*` emitted by a sidecar that survives a guest-triggered failure is silently discarded. | Directly blocked this investigation: a guest write failing inside the sidecar produced ZERO host-visible output (`RUST_LOG=debug` changed nothing) — the only symptom was a 31 s client-side timeout. This violates the repo rule that host-visible warnings/errors must reach stderr/log rather than staying trapped. **Fixed:** sidecar stderr is now forwarded to host stderr unconditionally, matching the Rust client which already spawned with `Stdio::inherit()` (the interim `AGENTOS_SIDECAR_STDERR` flag was removed — gating it left the two clients behaviorally different for identical code). The retained postmortem copy is bounded (first 16 KiB + last 48 KiB, truncation stated inline) because an append-only buffer fed by guest-triggered diagnostics is a host memory leak. Documented in `website/src/content/docs/docs/debugging.mdx`. |
 | LT-024 | **Open (CRITICAL — guest file writes are broken; supersedes the LT-011 "fix")** | Runtime/VFS + native-sidecar | **Every** guest file write hangs for 31 s and fails with `ERR_AGENTOS_BRIDGE_CALL_TIMEOUT` instead of writing. The underlying `EBADF: bad file descriptor 1000000000` (LT-011's root cause) was never fixed — F-4 only stopped it from killing the process, converting a loud crash into a silent hang. | **Directly evidenced** (not inferred) once LT-023 was mitigated. Sidecar log during a single 4 KiB `fs.writeSync`: `level=error message="process-event pump failed for session; continuing to serve other sessions" session=session-1 error="EBADF: bad file descriptor 1000000000"`. Chain: guest write → process-event pump raises EBADF on `MAPPED_HOST_FD_START` → **F-4 catches it, logs, and `continue`s** → the guest's pending sync-RPC response is *never delivered* → guest blocks until `limits.reactor.operationDeadlineMs` (31 s) → `ERR_AGENTOS_BRIDGE_CALL_TIMEOUT`. Before F-4 the identical error propagated to `main` → `exit(1)` → the cross-tenant crash. **Size- and path-independent**, so this is not a quota/large-write issue: a sweep of 4 KiB / 64 KiB / 256 KiB / 512 KiB / 1 MiB writes to BOTH `/tmp` and `/workspace` stalled ~31 000 ms in all 10 combinations. This also fully explains the previously-unexplained `filesystem-bytes` probe result (`exit=137`, "maxFilesystemBytes did not fire"): the guest never completes a single write, so it is killed by the harness timeout long before any byte cap could be reached — the cap was never the issue. **Correction to my earlier claim:** reporting LT-011 as "fixed" was correct only about *blast radius* (verified: co-located actors survive, 8→8 on Compute) and wrong to imply the defect was resolved — swallowing the error also violates the repo's "never swallow errors silently" rule. Correct fix has two parts: (1) stop routing mapped host fds (>= `MAPPED_HOST_FD_START`) into the kernel fd table, and (2) make pump failures deliver a typed errno to the waiting caller instead of dropping its response. |
 | LT-020 | Open (by design) | Compute | On a `--cpu 1` runner, per-runner AgentOS execution concurrency is 1 (executor pool = CPU count, LT-008/015), so co-located actors calling `execArgv` serialize and time out under contention. | A single-actor gateway `execArgv` works (`POST /gateway/<id>/action/execArgv` `{"args":["node",["-e",...]]}` → `{"output":{"exitCode":0,"stdout":...}}`). But 25 co-located actors each calling `execArgv` on a `--cpu 1` runner all timed out (25 s) — the noisy-neighbor blast-radius test couldn't even establish a baseline (`survivorsBefore=0`). Implication: per-runner guest execution throughput scales with `--cpu`, not actor count; a realistic runner needs `--cpu >= 2` (or fewer actors/runner) for concurrent AgentOS work. Retried with 6 co-located actors. |
 | LT-018 | **Open (HIGH — scaling)** | Compute | A fast burst to hundreds of actors overwhelms Compute scale-up: a large fraction fail the health timeout. | With `--max-scale 8 --max-concurrent-actors 25` and a create-storm at concurrency 48, ramp 25→50→100→150→200: step 25 = all ready; step 50 = 25 ready / **25 failed**; 100 = 50/50; 150 = 56 ready / **94 failed**, create-to-ready **P99 57 s**; 200 = 76 ready / **124 failed**. `peakConnectable` = 25 (one runner's slot count). Deployment logs confirm runners DID scale across regions (`ap-southeast-1`, `eu-central-1`, `us-east-1`, `us-west-1` booted mid-ramp), so scale-up happens — but too slowly to absorb a burst: actors wait >60 s for a slot and time out. Cleanup was clean (0 leaked actors). Net: the managed pool scales, but burst admission of hundreds of actors is not absorbed — high failure rate + tens-of-seconds cold-start. Needs either slower client ramp, higher `--max-concurrent-actors`, or faster Rivet scale-up. **Characterized (gentle ramp — concurrency 6, 150 s timeout, 45 s holds): 25→all ready, 50→44/50, 100→91/100 ready (only ~9% fail), but slowest create-to-ready P99 = 108 s.** So it is "can't scale FAST", not "can't scale": Compute absorbs ~90% of 100 actors when the client is patient, but new-runner/region cold-start is ~100+ s, so a fast burst (conc 48, 60 s timeout) causes majority failures. Owner action: reduce cold-start latency and/or document a client back-pressure/ramp guidance for large actor fleets. |

website/src/content/docs/docs/debugging.mdx

@@ -41,6 +41,17 @@ The agentOS sidecar emits structured **logfmt** logs for request handling, netwo
 | `AGENTOS_LOG_FILE` | append logs to this file instead of stderr (never stdout, which carries the wire protocol) |
 | `RUST_LOG_{SPAN_NAME,SPAN_PATH,TARGET,LOCATION,MODULE_PATH,ANSI_COLOR}` | per-field toggles (`=1` to enable) |
 
+Sidecar logs are written to the sidecar's stderr, and **both clients forward that
+stream to the host process's stderr as it arrives** — so warnings and errors from
+a sidecar that is still running are visible immediately, not only if it dies.
+
+The clients additionally retain a **bounded excerpt** of sidecar stderr (the
+first 16 KiB and the most recent 48 KiB) to attach to `SidecarProcessExited` for
+postmortem. Both ends are kept because the root cause is usually the *first*
+error while the fatal one is the *last*. If the excerpt is truncated, the omitted
+byte count is stated inline rather than silently shortening the output. Use
+`AGENTOS_LOG_FILE` when you need the complete, untruncated log.
+
 ```bash
 AGENTOS_LOG_LEVEL=debug AGENTOS_LOG_FILE=./sidecar.log RUST_LOG_FORMAT=logfmt node app.mjs
 ```

website/src/content/docs/docs/resource-limits.mdx

@@ -24,7 +24,7 @@ Set caps on the `limits` object in the `agentOS` config. Limits are grouped by s
 | `resources.maxProcesses` | Concurrent processes in the VM process table | Caps fork bombs and runaway spawning. New spawns fail with `EAGAIN`. |
 | `resources.maxOpenFds` | Open file descriptors | Exhausting the table fails with `EMFILE` / `ENFILE`. |
 | `resources.maxSockets` | Open sockets in the socket table | Bounds concurrent connections; excess `connect`/`accept` fail. |
-| `resources.maxFilesystemBytes` | Total bytes stored in the virtual filesystem | Bounds VFS storage; writes past the budget fail with a no-space error. |
+| `resources.maxFilesystemBytes` | Bytes stored by the VM, in the virtual filesystem **and** on host-backed paths | Default is `64 MiB`. Writes past the budget fail with a no-space error naming this limit. Host-backed paths (such as a writable `host_dir` mount) never reach the kernel VFS, so they are checked separately on write — see the caveat below. |
 | `resources.maxInodeCount` | Inodes retained by the virtual filesystem | Default is `16384`; creating another file or directory fails with a no-space error. This is the expected upper bound for filesystem-schema sizing and benchmarks. |
 | `resources.maxWasmFuel` | WASM execution budget | Bounds WASM execution work; unset means no explicit fuel budget. |
 | `resources.maxWasmMemoryBytes` | WASM linear memory, in bytes | Default is `128 MiB`. |
@@ -58,10 +58,25 @@ Set caps on the `limits` object in the `agentOS` config. Limits are grouped by s
 - **WASM stack**: deep recursion throws a stack-overflow error in the guest, never a host crash.
 - **JavaScript CPU time**: CPU-bound loops terminate with a CPU-budget error once active JS CPU exceeds `jsRuntime.cpuTimeLimitMs`.
 - **JavaScript wall time**: awaiting or blocked JS terminates only when you set `jsRuntime.wallClockLimitMs`; the default is disabled for long-lived adapters.
-- **Filesystem bytes**: writing past the VFS budget fails with a no-space error to the guest.
+- **Filesystem bytes**: writing past the budget fails with a no-space error to the guest, naming `limits.resources.maxFilesystemBytes`.
 - **Counts (fds / processes / sockets)**: hitting a table cap returns the standard POSIX errno appropriate to that cap (`EMFILE`/`ENFILE`, `EAGAIN`, etc.).
 - **Durable ACP collections**: session, prompt, pending-permission, and terminal-outcome caps fail atomically with typed errors naming the exact `limits.acp.*` field to raise. Per-session caps are validated not to exceed their corresponding per-VM cap.
 
+### Host-backed paths and `maxFilesystemBytes`
+
+Writes to host-backed paths — a writable `host_dir` mount, and other paths the
+runtime maps straight to a host file — do not pass through the virtual
+filesystem, so the VFS accounting that enforces `maxFilesystemBytes` never sees
+them. Those writes are checked separately, at the point the guest writes.
+
+That separate check is currently **per file**, while the VFS side compares
+against the VM's *total* usage. A guest can therefore stay under the cap on
+every individual host-backed file and still exceed its overall budget by
+spreading data across many files. If you rely on `maxFilesystemBytes` as a hard
+storage bound for untrusted code, also bound the underlying host directory (for
+example a dedicated filesystem or quota on the mount source) until the
+host-backed check aggregates per VM.
+
 ### WASM memory residency calls
 
 V8 owns the physical backing pages for WASM linear memory, and the runtime

website/src/generated/registry.json

@@ -254,6 +254,42 @@
 			"status": "available",
 			"image": "/images/registry/coreutils.svg"
 		},
+		{
+			"slug": "xfsprogs",
+			"title": "xfs_io",
+			"description": "Scriptable low-level filesystem I/O commands used by xfstests.",
+			"types": [
+				"software"
+			],
+			"category": "core",
+			"priority": 43,
+			"package": "@agentos-software/xfsprogs",
+			"status": "available"
+		},
+		{
+			"slug": "acl",
+			"title": "POSIX ACL tools",
+			"description": "Inspect and modify POSIX access and default ACLs.",
+			"types": [
+				"software"
+			],
+			"category": "core",
+			"priority": 42,
+			"package": "@agentos-software/acl",
+			"status": "available"
+		},
+		{
+			"slug": "attr",
+			"title": "attr",
+			"description": "Inspect and modify filesystem extended attributes.",
+			"types": [
+				"software"
+			],
+			"category": "core",
+			"priority": 41,
+			"package": "@agentos-software/attr",
+			"status": "available"
+		},
 		{
 			"slug": "grep",
 			"title": "grep",