Build Debian indexer boxes, starting with indexers-chic1

AGENTS.md

@@ -24,6 +24,12 @@ ansible-playbook cve.yml -K --check --diff -l sf   # dry run first
 ansible infra_boxes -m ping                         # no sudo needed
 ```
 
+**indexers** (`indexers-chic1`, 84.32.71.3, the indexer test box) are built by this repo
+but are not `ipfs_nodes`: Debian, `site.yml` and `k3s-upgrade.yml` only. Shared
+settings are in `group_vars/managed_boxes/`. See "Indexer boxes" in the README.
+Merge a new box only after it is built and its host key is pinned, or CI's
+`site.yml` runs fail on it.
+
 ## Getting in
 
 SSH is key-only as the hardened admin user `ipni`; root login is disabled and

README.md

@@ -46,8 +46,9 @@ The playbook is idempotent and safe to re-run.
 **Controller prerequisites.** `ansible-core` with the `community.general` and
 `ansible.posix` collections, and `kubectl` for the tunnel scripts. A **first**
 run against a freshly provisioned box also needs `sshpass` installed on the
-controller: site.yml falls back to root password authentication when the admin
-key does not work yet, and Ansible shells out to `sshpass` for that
+controller: site.yml falls back to root when the admin key does not work yet,
+by key from your ssh-agent if the box takes it (no `sshpass` needed), otherwise
+with the root password, and Ansible shells out to `sshpass` for that
 (`ansible.cfg` keeps `password` in `PreferredAuthentications` for the same
 reason). Once hardening has run, every later connection is key-based and
 `sshpass` is no longer used.
@@ -217,7 +218,7 @@ those with this box's own IP (the same machine under its old name) and any
 that are not Ready. A Ready node on another IP is never deleted.
 
 **Upgrading k3s.** Add the latest patch of each minor version you need, up to
-the stable channel, to `k3s_upgrade_path` in `group_vars/ipfs_nodes/main.yml`,
+the stable channel, to `k3s_upgrade_path` in `group_vars/managed_boxes/main.yml`,
 and set `k3s_version` there to the last entry — the playbook refuses to run
 unless the two agree, so the target and the route to it stay in step. Then run
 `k3s-upgrade.yml`. Look versions up at
@@ -231,6 +232,129 @@ unless the two agree, so the target and the route to it stay in step. Then run
   k3s itself, it first flushes every `KUBE-*`/flannel iptables rule.
 - Each step waits for the node version, Ready, and every deployment.
 - Rolling back to an older minor needs the backup taken on that minor.
+- Each step records every container ID first and compares afterwards. A
+  container replaced outside `kube-system` fails the box, so the next box is
+  not touched; one replaced inside it is reported.
+
+**Do containers restart on a k3s upgrade?** Normally not, from 1.31 on. This was
+checked against k3s's docs and the source of k3s and Kubernetes, not measured on
+a box, so the check above is what proves it on each hop.
+
+Why they survive: the unit has `KillMode=process`, so stopping k3s kills only
+k3s itself. Its containerd child dies with it (`Pdeathsig`), but each container
+has its own shim process, which keeps running and which the new containerd
+reconnects to. A running container keeps its old shim and runc until it is
+replaced. k3s's docs say containers continue while k3s is stopped. The kubelet
+then decides from two checks, and neither looks at its own version or at the
+pause image: the container hash (since 1.31 only the name and image, unchanged
+through 1.37) and `PodSandboxChanged`. Readiness is carried over a kubelet
+restart for containers that were already ready.
+
+Where containers **do** restart, or traffic is otherwise touched:
+
+| Case | What happens |
+|---|---|
+| Crossing 1.30 to 1.31 | The hash changed, so every container on the node restarts. Any box still below 1.31 hits this once (sf and sf2, on 1.28, would). |
+| A later Kubernetes changes the hash again | The same, with no warning. Upstream only supports draining before a minor kubelet upgrade, and declined to treat this as a breaking change. |
+| k3s's packaged components get a new image | Those pods roll: CoreDNS (1.36.4 to 1.37.2 moves it from 1.14.6 to 1.14.7) and, where enabled, Traefik, metrics-server, local-path and servicelb's `svclb` pods. |
+| A new pause image | Existing pods keep theirs; only new pods get it. Not a restart. |
+| A containerd bug | Has happened: k3s issue 6809 restarted non-host-network pods because CNI state was lost. Read the release notes. |
+| The API server and kubelet are down for the backup and restart | Containers run on, but nothing is restarted if one crashes, and anything calling the API sees errors: Argo CD, operators, pods that watch. |
+| A reboot or `k3s-killall.sh` | Everything restarts. |
+
+## Indexer boxes
+
+The `indexers` group holds indexers built by this repo: Debian stable, standalone
+k3s, built by `site.yml` and upgraded by `k3s-upgrade.yml`. No other playbook
+reaches them. They share the fleet's settings through `managed_boxes`, the parent
+of `ipfs_nodes` and `indexers` (`group_vars/managed_boxes/main.yml`); a value in
+`group_vars/indexers/` or `host_vars/<box>/` overrides it. What runs on the box
+comes from ipni/infra through Argo CD. sf and sf2 are not part of this.
+
+**Storage** is per box, in `host_vars/<box>/main.yml`. `storage_mode` is
+`ext4_blank_nvme` (as on the IPFS boxes), `zfs` (the pools in
+`storage_zfs_pools`, with disks by `/dev/disk/by-id`) or `md_xfs` (one md array
+over a partition on each disk, XFS on it, from `storage_md`). Indexers mount
+their data at `/ipni`, the path ipni/infra's indexer manifests use. The `zfs` and
+`md_xfs` modes never recreate anything: an existing pool, array or filesystem is left alone, a disk
+or member with any signature stops the run, `zpool create` and `mdadm --create`
+get no force flag, and a run from CI refuses to create a pool or array at all.
+k3s will not start until the data volume is mounted, so pods cannot write to the
+OS disk after a failed import or assembly.
+
+**The indexer test layout** (`indexers-chic1`, three 15.36 TB NVMe), meant to become
+the layout for sf and sf2 once it has proved itself:
+
+| Part | Where | Layout | Filesystem |
+|------|-------|--------|------------|
+| EFI | partition 1 on each disk | the installer's | vfat |
+| OS, 300 GB | partition 2 on each disk | md RAID1 over the three | ext4, `/` |
+| Data, the rest (about 27 TiB) | partition 3 on each disk | md RAID5 over the three | XFS, `/ipni` |
+
+The OS part is the install: the provider's custom partitioning or a Debian
+install from the console, with the rest of each disk left unallocated. Ansible
+cannot repartition a running root. `site.yml` then does the data part: it creates
+partition 3 after the last partition on each disk (only if every disk has at
+least `storage_md_min_free_gib` free and a GPT table), checks the three are
+blank, creates the array, records it in `mdadm.conf` and the initramfs, and
+formats XFS with 4 KiB sectors, which `mkfs.xfs` aligns to the RAID stripe by
+itself. The array's first resync runs for hours afterwards; it is usable
+meanwhile.
+
+**Switch the drives to 4096-byte sectors before the install.** Drives often
+ship on 512-byte sectors and offer 4096, which their makers rate faster (the
+Micron 7450s here: 512 "Good", 4096 "Best"). Switching erases the drive, so it
+is done from the provider's rescue system before the OS goes on, never from the
+installed OS:
+
+```bash
+nvme id-ns -H /dev/nvme0n1 | grep 'LBA Format'   # which format number is 4096
+mdadm --stop --scan                              # if the rescue system assembled the old RAID
+nvme format /dev/nvme0n1 --lbaf=1                # each drive; erases it
+lsblk -d -o NAME,PHY-SEC,LOG-SEC                 # LOG-SEC 4096 on each
+```
+
+Cherry's Debian 13 reinstall boots on 4096-byte drives (indexers-chic1, 2026-11). A
+`PHY-SEC` of 262144 on the 7450s is their preferred write size, not a sector
+size; partitions on whole MiB and the 512 KiB chunk are multiples of it.
+
+To try a different data layout on the same box, stop k3s, unmount `/ipni`, and
+stop and clear the array by hand; the role will not do that.
+
+**Building a box.** Add it to `indexers` with its `host_vars`, on a branch. With
+the root key in your agent (`ssh-add`), and an admin key from `keys/` too:
+
+```bash
+ansible-playbook site.yml -l <box>
+```
+
+A dry run cannot get far on a box this new: the access gate needs the admin user
+to exist and later tasks need packages installed. So check the box with read-only
+commands first. Then pin its host key in `inventory/known_hosts` and merge. **Do
+not merge before the build**: CI refuses an unhardened box, and its `site.yml` runs
+would fail until the box is built and pinned. If root takes neither your key nor
+`vault_root_password` (which only `ipfs_nodes` can see), the run stops there.
+
+### Package updates and reboots
+
+| | Policy on an indexer |
+|---|---|
+| What updates | `unattended-upgrades`, daily: Debian stable, its security archive and `-updates`. Never backports. |
+| Reboots | Never automatic. |
+| ZFS packages | Held out of unattended upgrades on a zfs box: a DKMS build that fails would only show at the next reboot, as a pool that does not import. Upgrade them by hand. |
+| Services | Packages that restart their own service on upgrade still do (ssh, chrony, fail2ban). Established SSH sessions survive. k3s is not an apt package, so apt never restarts k3s or containers. `needrestart` only lists what is left running on old libraries. |
+| Kernels | Installed but inactive until a reboot. |
+
+A pending reboot shows in three places: `/var/run/reboot-required`, the `reboot:`
+line in every `site.yml` summary, and the `node_reboot_required` metric, which an
+hourly timer writes to `/var/lib/node_exporter/textfile/reboot.prom`. The metric
+reaches Prometheus only once ipni/infra's node-exporter is given that directory
+(`--collector.textfile.directory`, with a host mount) and an alert is added.
+
+To reboot: on a zfs box run `dkms status` first and check ZFS is built for the new
+kernel, then reboot one box at a time. Every container restarts.
+
+The IPFS boxes keep their existing update policy. They only gain the `reboot:` summary line.
 
 ## needle
 
@@ -1772,8 +1896,11 @@ turn an unrelated pull request red.
 
 ```
 ansible.cfg               inventory, vault and SSH defaults
-inventory/hosts.yml       the three hosts
-group_vars/ipfs_nodes/    tunables (admin user, firewall, k3s, sysctl)
+inventory/hosts.yml       the boxes and their groups
+group_vars/managed_boxes/ what every built box shares (admin user, firewall, k3s, sysctl)
+group_vars/ipfs_nodes/    the IPFS boxes' own settings (rollout, bootstrappers, Argo CD)
+group_vars/indexers/      the indexers' own settings
+host_vars/<box>/main.yml  an indexer's site and storage
 host_vars/<box>/vault.yml   encrypted per-box credentials (gitignored, see Secrets)
 roles/{common,storage,hardening,k3s}/   base preparation
 roles/needle/            needle firewall + deploy

group_vars/indexers/main.yml

@@ -0,0 +1,26 @@
+---
+# Indexer boxes. Everything not set here comes from group_vars/managed_boxes.
+
+# Where the data volume is mounted, as on sf and sf2: ipni/infra's indexer
+# manifests use /ipni/indexer/... host paths. k3s and local-path live here too.
+data_mount: /ipni
+
+# `debian` is the Debian cloud image's login account, the same uid 1000 risk as
+# Ubuntu's (see roles/hardening/tasks/main.yml).
+hardening_removed_accounts:
+  - ubuntu
+  - debian
+
+# Point releases as well as security fixes, so packages keep up with Debian
+# stable and not only with its advisories. Debian's own defaults already allow
+# the release itself and Debian-Security; this adds -updates. Backports are
+# never matched. See "Package updates and reboots" in the README.
+common_unattended_extra_origins:
+  - "origin=Debian,codename=${distro_codename}-updates"
+
+# needrestart reports and restarts nothing, and the reboot state goes to a
+# file node-exporter can read.
+common_reboot_reporting: true
+
+# To upgrade k3s here ahead of the fleet (a canary), set k3s_version and
+# k3s_upgrade_path in this file: they override group_vars/managed_boxes.