fix(deploy): pass capacity.env values through Compose and prepare runtime dirs

[ttbombadil]

Summary

Fixes the two deployment defects found on a fresh Ubuntu 26.04 install after the calibration run itself had succeeded. No change to capacity policy, the security model, the gateway/trust semantics, or production defaults (with the one clearly marked exception below, where an existing value was not passed through at all).

Based on current main (rebased after #248 was merged); the diff contains only the three commits below.

3. capacity.env values were not reaching the backend container

docker-compose.yml did not forward SIMULATION_QUEUE_TIMEOUT_MS at all, so a calibrated 151000 was silently replaced by the application default (Queue timeout: 60 s).

• SIMULATION_QUEUE_TIMEOUT_MS=${SIMULATION_QUEUE_TIMEOUT_MS:-}: an empty value keeps the application default (60000 ms) from server/config.ts, which already treats an empty string as unset. No second default is introduced.
• Same defect class: SIMULATION_MAX_CONCURRENT=200 was hard-coded although docs/INSTALL_SERVER.md lists it as an .env setting and calibration proposes it. It is now ${SIMULATION_MAX_CONCURRENT:-200}: the default stays 200, a reviewed value now takes effect. The other five variables of capacity.env were already forwarded.
• Docs (INSTALL_SERVER.md, CAPACITY_MODEL.md) say that capacity.env lines only take effect once they are in .env and the backend container has been recreated, and where to verify them (/api/status → capacity).

4. Bind-mounted runtime directories were not writable for the container user

The backend runs as node (UID/GID 1000). server/arduino-cache, temp and storage were root:root (created by Docker for a missing mount source) or owned by the installing host user (UID 1001 in the field), so every compile failed with EACCES → POST /api/compile 500.

• New scripts/prepare-runtime-dirs.sh: creates the three directories and hands them to UID/GID 1000 (chown -hR, mode 0750). It targets the container user, not the host UID. No chmod 777, the backend still does not run as root, the container's security settings are untouched.
• Idempotent; repairs content an earlier container start left as root:root; refuses symlinks and non-directories before changing anything; requires sudo unless the target already is the running account.
• scripts/install-ubuntu-vbox.sh calls it instead of a plain mkdir -p as the installing user.
• The manual Compose path in docs/INSTALL_SERVER.md has the one-time sudo ./scripts/prepare-runtime-dirs.sh step; docs/INSTALL_UBUNTU_VBOX.md describes the resulting ownership.

Build context: storage added to .dockerignore

After the script ran, the host account can no longer read these directories. server/arduino-cache and temp were already in .dockerignore; storage was not.

What I measured (Docker CLI as UID 1001 in a Linux container, directories 1000:1000 mode 0750):

Dockerfile step	storage not ignored	storage ignored
current COPY lines (BuildKit)	builds	builds
COPY . /all	open /ctx/storage: permission denied	builds
COPY server /server (arduino-cache inside the copied directory)	open /ctx/server/arduino-cache: permission denied	builds

So with today's Dockerfile BuildKit never reads storage, and the ignore is a safeguard, not a repair of an observed failure: storage is runtime state that never belongs in an image, and the build must keep working if a step ever copies the whole context. The existing server/arduino-cache entry is the one that matters today because that directory sits inside the copied server/. Two things the ignore does not do, found while testing: the legacy builder (DOCKER_BUILDKIT=0) and Docker Desktop for macOS stat every directory before applying excludes and fail on an unreadable one regardless of .dockerignore. Neither is the supported path (Ubuntu installer: BuildKit via docker-buildx).

How the failures were reproduced

On the fresh Ubuntu 26.04 / x86_64 host (as reported): after a successful calibration, SIMULATION_QUEUE_TIMEOUT_MS=151000 set in .env still showed Queue timeout: 60 s; adding it to the compose environment locally showed 151 s. Compile requests returned 500 with EACCES while the mount sources were root:root / UID 1001; after chown -R 1000:1000 server/arduino-cache temp storage compile worked. I did not have access to that host.

Locally: with the old docker-compose.yml the new compose tests fail (SIMULATION_MAX_CONCURRENT pinned to 200, SIMULATION_QUEUE_TIMEOUT_MS absent); with the fix they pass. The EACCES itself cannot be reproduced on macOS (Docker Desktop emulates bind-mount ownership), so it is covered by the Linux CI job below.

Tests

• tests/server/compose-capacity-passthrough.test.ts (new): the variable list is derived from the capacity.env renderer; real docker compose config interpolation checks that every variable set in .env lands in the backend environment and that defaults are unchanged (200 / 25 / empty queue timeout → application default).
• tests/server/prepare-runtime-dirs.test.ts (new): real filesystem permissions with the current account as target (never assumes UID 1000; the CI runner is 1001): ownership, 0750, writable, no world access, idempotent, keeps content, refuses symlink/file, asks for sudo instead of failing halfway, directory list equals the compose bind mounts.
• tests/server/docker-build-context.test.ts (new): every compose bind mount is in the real .dockerignore and no negation re-includes one; the Dockerfile copies none of them; a COPY . /all build as UID 1001 against 1000:1000/0750 directories succeeds with the real .dockerignore and, as a control, fails with permission denied once the runtime entries are removed. Needs the Docker daemon and the docker:27-cli image the Dockerfile already uses; skipped otherwise.
• scripts/deployment/installer-deployment.ts (npm run test:deployment:installer, CI job "Production Gateway Deployment"): replaced chmod 0o777 on the runtime directories with the real script run as root in the server image, checks that the script's default UID:GID equals id of the image's user, asserts the backend user (not root) can write all three paths, and that SIMULATION_QUEUE_TIMEOUT_MS / SIMULATION_MAX_CONCURRENT appended to .env appear in /api/status.

Verification

• Local: npm run check, ESLint --max-warnings=0, npm run check:docs, npm run build, and the full ./run-tests.sh (static analysis, knip, unit, toolchain integration, Docker, Playwright E2E, production build, security audit, Sonar gate with 0 violations) on the final head; pre-push hook.
• Documented flow on a clean copy of the committed tree: prepare-runtime-dirs.sh, then docker compose up --build -d with SIMULATION_QUEUE_TIMEOUT_MS=151000 in .env: readiness 200, /api/status showed queue.timeoutMs: 151000 and maxConcurrent: 200 (unchanged default), POST /api/compile returned 200 as uid=1000(node).
• Real Dockerfile built as host UID 1001 in a Linux container against directories prepared by the script (1000:1000, 0750): image built.
• Not runnable on macOS: the installer deployment test stops at its Nginx gateway step (network_mode: host is not reachable on Docker Desktop for Mac). This happens identically on unchanged main. Everything the new code asserts before that step passed locally; the rest is exercised by the Linux CI job.

Remaining risks

• The UID/GID 1000 default is tied to the node user of the base image; the deployment test fails if the image's user changes.
• Existing installations must run the script once (it is idempotent) or have the directories owned by 1000 already.
• With mode 0750 and owner 1000, other host accounts cannot read the cache or temp contents (intentional: temp holds learners' sketches); administrators use sudo.
• Docker Desktop for macOS and the legacy builder cannot build from a context that holds an unreadable directory, with or without .dockerignore (see above); they are outside the supported Ubuntu path.
• tests/deployment/docker-compose.gateway.yml (separate gateway test topology, not the production compose file) still chmod 0777s its own throwaway temp dir; left unchanged to keep this PR minimal.

🤖 Generated with Claude Code