test(distributed): run distributed mode in CI, and test replica failover

.agents/building-and-testing.md

@@ -45,3 +45,23 @@ Rules (both gates):
 - **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
 - If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
 - The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.
+
+## Distributed-mode test suites
+
+Two suites cover distributed mode (frontend replicas, worker nodes, PostgreSQL, NATS), split by a Ginkgo label:
+
+- `make test-e2e-distributed` runs `Distributed && !VLLMMultinode && !Cluster` over `./tests/e2e/distributed` recursively. Services are wired directly into the test binary. ~240 specs, ~75s.
+- `make test-e2e-cluster` runs `Cluster` and spawns real `local-ai` child processes through the `tests/e2e/distributed/cluster` helper package. 6 specs, about 8m30s measured over three consecutive runs (509.1s / 509.8s / 512.3s, so 8m29s to 8m32s).
+
+Both jobs live in `.github/workflows/tests-e2e-distributed.yml`, with `timeout-minutes: 45` each. They trigger on pull requests *and* on every push to `master`; the `paths-ignore` filter (see [.agents/ci-caching.md](ci-caching.md)) sits on the pull-request trigger only, so a master push always runs both. They are advisory only because `master` carries no branch protection, which is a repository setting and not a YAML key: `continue-on-error: true` would flip the run's *conclusion* to success and hide the failure, so it is not used.
+
+- **Containers are suite-scoped, not spec-scoped.** `SetupInfra` used to start a PostgreSQL (~10s) and a NATS (~3.5s) per spec. Across the 213 specs behind it that was roughly **48 minutes of pure container startup per run**, which is why this suite was never in CI. (213 rather than the ~240 above: the larger number is everything the label filter selects, the smaller one is just the specs that call `SetupInfra`.) Containers now start once in `BeforeSuite` and each spec gets its own database via `CREATE DATABASE` (~67ms), which is what the `dbName` argument was always describing. Adding a spec needs no change: call `SetupInfra("some-name")` as before, the name is a prefix and a counter keeps it unique.
+- **Consequence for new specs:** the NATS bus is now *shared* within a Ginkgo process, so a wildcard subscriber can observe another spec's traffic. Filter assertions on an identifier your spec owns (a node ID, a job ID) instead of counting everything on `jobs.*.progress`, and verify the spec with `--randomize-all`.
+- **`BeforeSuite`, not `SynchronizedBeforeSuite`.** Under `ginkgo -p` each process then gets its own container pair, keeping NATS subjects isolated per process. A single shared NATS across parallel processes would let specs on different processes see each other's messages on the same subject.
+- **The label split.** The 8 argument-validation specs under `tests/e2e/distributed/cluster/` carry `Label("Distributed")` only, on purpose: they need no binary, no PostgreSQL and no NATS, so they belong in the fast job. That is why `test-e2e-distributed` keeps `-r` (it must reach the subpackage) and `test-e2e-cluster` deliberately does **not** (the subpackage is out of its scope).
+- **`--fail-on-empty` is load-bearing on both targets.** Ginkgo exits 0 when a label filter selects nothing, so without it a refactor that renames or drops `Label("Cluster")` leaves the target reporting "Test Suite Passed" having started no cluster at all. `LOCALAI_E2E_REQUIRE_BINARIES` does not cover this case: it only fires inside a spec that is actually running.
+- **The binary gate.** `localAIBinary()` and `mockBackendBinary()` **fail** rather than skip when `CI` is set, or when `LOCALAI_E2E_REQUIRE_BINARIES` is truthy; `LOCALAI_E2E_REQUIRE_BINARIES=0` (also `off`, `no`, `n`, `disabled`, and anything `strconv.ParseBool` reads as false) forces skipping even under CI. **Any value that parses as neither reads as ON**, not off: setting the variable to something meaningless means someone meant to turn the gate on, and reading it as false would quietly restore the silent skip the flag exists to remove. The whole polarity is deliberate, because in CI a skipped cluster spec is indistinguishable from a passing one: Ginkgo exits 0 on skips. Locally a missing binary still just skips, since `CI` is unset in an ordinary shell.
+- **Flake budget: no retries at all.** `--flake-attempts` is *total attempts*, not retries (ginkgo v2.29.0 `internal/group.go` sets `maxAttempts = FlakeAttempts` and loops `attempt < maxAttempts`; the flag's own usage string reads "0 - failed tests are not retried"). `DISTRIBUTED_TEST_FLAKES` defaults to **1**, so each spec runs once and a failure is a failure, and `test-e2e-cluster` pins `--flake-attempts 1` outright rather than reading the variable. The repo-wide `TEST_FLAKES=5` means up to five attempts, so up to four retries. These suites exist to surface nondeterminism, and a retry converts exactly that signal into a green run. Raise it locally when bisecting something unrelated, not in the Makefile.
+- **Coverage:** `tests/e2e/distributed` is excluded from the coverage roots (`COVERAGE_E2E_ROOTS = ./tests/e2e`, run non-recursively), and so is the `cluster` helper package beneath it. Neither suite moves the baseline, so production code that these suites are the only cover for reads as **uncovered**. Unit tests for such code belong under `./core/...` with `testutil.SetupTestDB()`.
+- **The cluster job builds against a stubbed React UI.** `core/http/react-ui/dist` is gitignored and built by Node, so the workflow writes a one-line `index.html` there to satisfy the `//go:embed react-ui/dist/*` in `core/http/app.go` and skips a full Node and Vite install. That holds only while the suite drives the HTTP API and never the UI, which has its own e2e suite. A spec that ever asserts on a UI asset would pass locally, where a real `dist/` exists, and be served the stub in CI: if you write one, the stub step has to go and the real build come back.
+- **Do not shorten the cluster suite's waits.** Three of its six specs sit at ~167s each because they wait out a 60s staleness threshold plus a 15s health-check tick. That wait is what stops the assertions from passing before the system could have reacted, which was a real false green earlier on. If the job has to get faster, the levers are CI concurrency or making the thresholds configurable, not shorter waits.

.agents/ci-caching.md

@@ -153,7 +153,7 @@ This is worth more than it looks. Measured over the week to 2026-07-30, **97% of
 
 The volume is real: 13 gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs opened were bot-generated.
 
-`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20. The excluded set:
+`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20, measured before `tests-e2e-distributed.yml` (2 jobs) landed. That workflow carries the same exclusion set for the same reason: its dependency graph is 99 packages, so an allowlist of paths would silently stop guarding the moment code moved, while a diff confined to the paths below provably cannot reach it. The excluded set:
 
 | Path | Why no image or Go build can see it |
 |---|---|
@@ -192,7 +192,7 @@ What still runs, and why it has to:
 Two properties this relies on:
 
 - `paths-ignore` skips a run only when **every** changed file matches, so a PR touching the gallery *and* Go code still runs everything. That is what makes the exclusion safe rather than a hole.
-- `master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these four entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
+- `master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these five entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
 
 ### `image.yml` on master push is gated too, by a job rather than a path filter
 

.github/workflows/tests-e2e-distributed.yml

@@ -0,0 +1,204 @@
+---
+name: 'E2E Distributed Tests'
+
+on:
+  pull_request:
+    # The suite's dependency graph is 99 packages, so an allowlist of paths
+    # silently stops guarding the moment code moves. At ~75s the job is cheap
+    # enough to run unless the diff is confined to paths it provably cannot
+    # reach. See .agents/ci-caching.md.
+    paths-ignore:
+      - 'gallery/**'
+      - 'docs/**'
+      - 'examples/**'
+      - '**/*.md'
+  push:
+    branches:
+      - master
+
+concurrency:
+  group: ci-tests-e2e-distributed-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  tests-e2e-distributed:
+    runs-on: ubuntu-latest
+    # Advisory because it is deliberately not in branch protection, so a failure
+    # is a visible red X rather than a blocked merge. Promoting it to a required
+    # check is a repository-settings change, to be made once it has a track
+    # record; a heavy suite made required on day one gets disabled instead of
+    # fixed.
+    timeout-minutes: 45
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+        with:
+          submodules: true
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.0'
+          cache: false
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential libopus-dev
+      - name: Proto Dependencies
+        run: |
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Pre-pull test images
+        # Pulling here rather than inside the suite keeps container-start timing
+        # out of the spec timeouts and makes a registry outage read as a
+        # setup failure instead of a test failure. These two are the only images
+        # the suite needs once the testcontainers reaper is disabled below.
+        run: |
+          docker pull postgres:16-alpine
+          docker pull nats:2-alpine
+      - name: Distributed E2E
+        # TESTCONTAINERS_RYUK_DISABLED keeps the pre-pull above meaningful. The
+        # reaper exists to clean up leaked containers on a long-lived host, but
+        # this runner is ephemeral and every container dies with the VM. Leaving
+        # it enabled would pull a third, unpinned image (testcontainers/ryuk)
+        # from Docker Hub mid-suite: exactly the registry dependency the
+        # pre-pull step exists to remove.
+        env:
+          TESTCONTAINERS_RYUK_DISABLED: "true"
+        run: |
+          PATH="$PATH:$HOME/go/bin" make test-e2e-distributed
+      - name: Setup tmate session if tests fail
+        if: ${{ failure() }}
+        uses: mxschmitt/action-tmate@v3.23
+        with:
+          detached: true
+          connect-timeout-seconds: 180
+          limit-access-to-actor: true
+
+  tests-e2e-cluster:
+    runs-on: ubuntu-latest
+    # Advisory for the same reason as the job above: master has no branch
+    # protection, so a failure here is a visible red X rather than a blocked
+    # merge. That is a repository-settings property, not a YAML key. The key
+    # that looks like it says "advisory" instead flips the run's conclusion to
+    # success, which hides the failure rather than flagging it, so it appears in
+    # none of this repo's workflows and must not be added here.
+    #
+    # Separate job from tests-e2e-distributed so the fast in-process suite is
+    # not held behind a Go build of local-ai. Serial on purpose: each Ginkgo
+    # process would get its own PostgreSQL and NATS container and each spec
+    # spawns two or three local-ai children, so --procs on an unmeasured runner
+    # is a change to make with numbers, not by default.
+    #
+    # The two timeouts bound different things and are not alternatives. Ginkgo's
+    # --timeout=20m bounds the SUITE only; this job timeout must additionally
+    # cover setup, which here is the larger and more variable half: submodule
+    # checkout, apt, protoc plus two go installs plus protogen-go, a cold-cache
+    # module download (cache: false), a full go build of ./cmd/local-ai, and a
+    # separate ginkgo test compile. That build alone is ~316s of CPU, so on a
+    # 4-vCPU runner setup is realistically 8-12 minutes.
+    #
+    # 45 minutes therefore, matching the sibling job. A tighter number does not
+    # make a hang fail faster, it just moves the kill from Ginkgo, which prints
+    # which spec hung, to the runner, which prints nothing: a red job with no
+    # evidence, which is how a suite gets disabled rather than fixed.
+    #
+    # The suite itself is about 8m30s over three consecutive runs (509.1s /
+    # 509.8s / 512.3s, so 8m29s to 8m32s) on a developer box, and will be slower
+    # here. Three specs sit at ~167s each because they wait out a 60s staleness
+    # threshold plus a 15s health-check tick (HealthCheckInterval, in
+    # core/config/distributed_config.go; core/services/nodes/health.go runs the
+    # ticker on the unexported checkInterval, not one of the reconcilers). Do
+    # not shorten those windows to make this job faster: the wait is what stops
+    # the assertions from passing before the system could have reacted, which
+    # was a real false green earlier on.
+    timeout-minutes: 45
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+        with:
+          submodules: true
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.26.0'
+          cache: false
+      - name: Dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential libopus-dev
+      - name: Proto Dependencies
+        run: |
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
+          unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+      - name: Stub the embedded React UI
+        # core/http/react-ui/dist is gitignored and built by Node, but this
+        # suite drives the HTTP API and never the UI, which has its own e2e
+        # suite. A single index.html satisfies the //go:embed react-ui/dist/*
+        # in core/http/app.go, so the job skips a full Node and Vite install.
+        # If a cluster spec ever asserts on a UI asset, this step must go and
+        # the real build come back: a developer box has a real dist/, so such a
+        # spec would pass locally and fail only here, or worse be served the
+        # stub and pass in both places.
+        run: |
+          mkdir -p core/http/react-ui/dist
+          printf '<!doctype html><title>stub</title>\n' > core/http/react-ui/dist/index.html
+      - name: Build local-ai
+        # Not `make build`: that target pulls in the React UI build. The specs
+        # exec this binary directly via LOCALAI_E2E_BINARY.
+        run: |
+          PATH="$PATH:$HOME/go/bin" go build -o local-ai ./cmd/local-ai
+      - name: Pre-pull test images
+        # Same reasoning as the job above: pulling here keeps container-start
+        # timing out of the spec timeouts and makes a registry outage read as a
+        # setup failure rather than a test failure.
+        run: |
+          docker pull postgres:16-alpine
+          docker pull nats:2-alpine
+      - name: Cluster E2E
+        env:
+          LOCALAI_E2E_BINARY: ${{ github.workspace }}/local-ai
+          # Must live under the workspace so the upload step below can reach it.
+          # The harness defaults to GinkgoT().TempDir(), which lands under
+          # TMPDIR and would leave the artifact glob matching nothing.
+          LOCALAI_E2E_LOG_DIR: ${{ github.workspace }}/cluster-logs
+          # Belt and braces: the harness already fails rather than skips when CI
+          # is set, and GitHub Actions always sets CI. Stating it here means a
+          # future edit to that default cannot silently turn this job into one
+          # that passes without ever starting a cluster, since a skipped cluster
+          # spec is indistinguishable from a passing one.
+          LOCALAI_E2E_REQUIRE_BINARIES: "true"
+          # See the job above: the runner is ephemeral, so the reaper buys
+          # nothing and would pull a third, unpinned Docker Hub image mid-suite.
+          TESTCONTAINERS_RYUK_DISABLED: "true"
+        run: |
+          PATH="$PATH:$HOME/go/bin" make test-e2e-cluster
+      - name: Upload process logs
+        # The per-process logs are the only way to read a cluster failure: the
+        # Ginkgo output says which assertion failed, not what the four child
+        # processes were doing. Without this a red job is undebuggable.
+        if: ${{ failure() }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: cluster-process-logs
+          path: cluster-logs/**/*.log
+          if-no-files-found: ignore
+          retention-days: 7
+      - name: Setup tmate session if tests fail
+        if: ${{ failure() }}
+        uses: mxschmitt/action-tmate@v3.23
+        with:
+          detached: true
+          connect-timeout-seconds: 180
+          limit-access-to-actor: true