docs(cluster-upgrade): add DRBD-aware Talos node-upgrade runbook

plugins/cozystack/skills/cluster-upgrade/SKILL.md

@@ -140,6 +140,12 @@ Run both general **and** targeted-per-change checks from Step 1.
 
 Show user: result (success/partial/failed), before→after version, HR/Package totals, any warnings, one-line outcome per targeted check.
 
+## Node / Talos OS upgrades
+
+The steps above cover only the `cozy-installer` Helm upgrade. When a release also moves the **Talos OS** or **Kubernetes** version, that is a separate node-by-node procedure — and on DRBD/LINSTOR-backed clusters the sequencing (one node at a time, wait for each node's DRBD resources to return `UpToDate` before the next) is what keeps VMs alive. Order the layers Talos → Kubernetes → Cozystack, since some fixes ship only in a newer Talos extension bundle (e.g. a DRBD kernel-module fix) and must land first.
+
+**Full node-upgrade runbook:** read `references/talos-node-upgrade.md`.
+
 ## Rollback
 
 `helm --kube-context $CTX rollback cozystack <rev> --namespace cozy-system` is possible but has caveats (data migrations don't reverse). Before rolling back, snapshot: `kubectl --context $CTX get packages.cozystack.io -A -o yaml > pre-rollback.yaml`.
@@ -162,6 +168,7 @@ High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore c
 | HR `UninstallFailed, failed to delete release` | Stuck helm history (known-failures #1) |
 | TCP `INSTALLED VERSION` diverges from `VERSION` | Kamaji upgrade stuck (known-failures #4) |
 | `cozy-system` namespace gone | Missing `helm.sh/resource-policy=keep` (known-failures #7); restore from backup |
+| VM `PausedIOError` loop after a node reboot / `auto-diskful` | DRBD block-size mismatch on 9.2.16, Talos ≤ 1.12 (known-failures #8) |
 
 ## Common mistakes
 
@@ -174,7 +181,7 @@ High-blast-radius stuck states — stuck helm `uninstalling`, Kamaji datastore c
 
 ## References
 
-- Skill files: `references/release-notes-analysis.md`, `references/preflight-checks.md`, `references/post-upgrade-checks.md`, `references/rollback.md`, `references/known-failures.md`
+- Skill files: `references/release-notes-analysis.md`, `references/preflight-checks.md`, `references/post-upgrade-checks.md`, `references/rollback.md`, `references/known-failures.md`, `references/talos-node-upgrade.md`
 - Upstream (pick the version matching your target from the docs site version selector): `https://cozystack.io/docs/<vX.Y>/operations/cluster/upgrade/`
 - Troubleshooting checklist: `https://cozystack.io/docs/<vX.Y>/operations/troubleshooting/`
 - Releases: `https://github.com/cozystack/cozystack/releases`

plugins/cozystack/skills/cluster-upgrade/references/known-failures.md

@@ -253,6 +253,44 @@ Restore from backup. There is no clean in-cluster recovery for a deleted `cozy-s
 3. Re-apply the Platform Package from rescue.yaml (manual review required; CRD schemas may have moved).
 4. Expect tenant disruption; communicate to users.
 
+## 8. VM `PausedIOError` loop after LINSTOR `auto-diskful` (DRBD block-size mismatch)
+
+### Symptom
+
+A running KubeVirt VM on a DRBD volume flips into `PausedIOError` and loops pause↔unpause every ~3 min. No data loss — all replicas stay `UpToDate`. dmesg on the node shows:
+
+```text
+drbd <res>: setting new queue limits failed
+```
+
+and the VM's qemu log shows an EINVAL on the disk:
+
+```text
+IO error device='ua-disk-boot' ... reason='Invalid argument'
+```
+
+Trigger: the VM was running on a node **without** a local replica (DRBD *diskless* + Primary), and LINSTOR `auto-diskful` converted it to *diskful* online (`toggle-disk`) while the VM held the device open.
+
+### Root cause
+
+LINSTOR `Linstor/Drbd/auto-block-size` (default-on since LINSTOR 1.33.2) raises the DRBD logical block size to 4096 on the **diskful** side only; the **diskless** side stays 512. qemu opens the device diskless (512) and caches 512-byte O_DIRECT alignment. The `auto-diskful` toggle then changes the device geometry 512→4096 under the open qemu → qemu keeps submitting 512-aligned O_DIRECT (e.g. MariaDB/InnoDB) → the now-4096 device returns EINVAL → `errorPolicy=stop` → pause.
+
+Underneath, on **DRBD 9.2.16 + kernel 6.x** `queue_limits_commit_update()` fails (the `setting new queue limits failed` line); the freshly-computed limits are discarded and the queue keeps stale values. A diskless device has no backing device whose limits can be stacked in, so the computed 4096 never commits and it stays 512.
+
+- Affected: Talos ≤ 1.12.x (DRBD 9.2.16), kernel 6.x, LINSTOR ≥ 1.33.2 with `auto-block-size` on.
+- Fixed upstream in **DRBD ≥ 9.2.17** ("Correctly request BLK_FEAT_STABLE_WRITES from the kernel" + "always two-phase-commit for attach"). Talos `release-1.13` ships DRBD 9.3.2. The changelog does not name this diskless-block-size path explicitly — validate in a lab before banking on it: a diskless resource with `block-size=4096` should report `logical_block_size=4096` with no `setting new queue limits failed`.
+
+### Recovery / mitigation
+
+```bash
+# Immediate: reopen the device cleanly (recomputes limits from the current geometry)
+virtctl restart <vm> -n <ns>
+```
+
+- **Reduce exposure** without an OS upgrade: keep VMs on nodes that hold a replica (volume affinity), or raise the replica count so most nodes have a local copy — fewer diskless opens toggled under a live VM.
+- **New volumes only:** set `Linstor/Drbd/auto-block-size=False` before create → diskful and diskless both stay 512 (aligned), no mismatch. Existing volumes already carry the 4096 diskful geometry, so this does **not** fix them.
+- **Permanent:** upgrade Talos to 1.13.x (DRBD 9.3.2) — a node-by-node OS upgrade, see `talos-node-upgrade.md`.
+
 ## Diagnostic quick reference
 
 | Question | Command |

plugins/cozystack/skills/cluster-upgrade/references/talos-node-upgrade.md

@@ -0,0 +1,80 @@
+# Talos / node OS upgrade (DRBD-aware rolling)
+
+Read this when a Cozystack upgrade also moves the underlying **Talos OS** or **Kubernetes** version — the main `cluster-upgrade` flow only covers the `cozy-installer` Helm upgrade. Node upgrades are a **separate, node-by-node procedure**, and on DRBD/LINSTOR-backed clusters the sequencing is what keeps VMs alive.
+
+## Where it fits in the upgrade order
+
+When a release bumps Talos, treat each layer as its own window and run them in this order:
+
+1. **Talos OS** — one node at a time (this file).
+2. **Kubernetes** — per the release notes (control-plane first).
+3. **Cozystack platform** — the Helm flow in the main skill.
+
+Rationale: some fixes ship only in the newer Talos extension bundle (e.g. a DRBD kernel-module fix), so the OS bump has to land first. Talos pins every extension to its release branch — you **cannot** ship a newer DRBD without moving the Talos minor. Check the DRBD version a Talos release carries before planning: `release-1.12` ships DRBD 9.2.16, `release-1.13` ships 9.3.2.
+
+## The one rule that prevents outages
+
+**One node at a time — and never start the next node until the just-upgraded node's DRBD resources are back `UpToDate`.**
+
+DRBD replicates each volume across a small number of nodes (often 2–3). Rebooting a node takes its replicas offline; the volume keeps quorum from the survivors. Reboot a **second** replica-holding node before the first finished resyncing and the volume loses quorum → I/O suspends → every VM on it stalls. The resync wait is not optional.
+
+## Pre-flight (storage must be clean)
+
+```bash
+alias linstor='kubectl --context $CTX exec -n cozy-linstor deploy/linstor-controller -ti -- linstor'
+linstor node list                 # every node Online
+linstor resource list --faulty    # empty
+linstor storage-pool list         # no Err
+# no lost-quorum taints stuck on any node:
+kubectl --context $CTX get nodes -o json \
+  | jq -r '.items[] | select(.spec.taints[]?.key=="drbd.linbit.com/lost-quorum") | .metadata.name'
+```
+
+Any faulty resource, pending resync, or lost-quorum taint blocks the upgrade — fix it first (the `linstor:recover` skill covers the recovery paths).
+
+## Per-node loop
+
+Do control-plane nodes first (one at a time, so etcd quorum holds), then storage/worker nodes one at a time.
+
+```bash
+NODE=<node-name>
+
+# 1. Cordon
+kubectl --context $CTX cordon "$NODE"
+
+# 2. Drain — live-migrate VMs off, respect PDBs.
+#    KubeVirt VMs with evictionStrategy=LiveMigrate migrate on drain; ones without
+#    it are stopped instead, so check before draining a node that hosts VMs.
+kubectl --context $CTX get vmi -A --field-selector status.nodeName="$NODE"
+kubectl --context $CTX drain "$NODE" --ignore-daemonsets --delete-emptydir-data
+
+# 3. Upgrade Talos. talm resolves the installer image from values.yaml::image —
+#    bump that to the target release first, then:
+talm upgrade -f nodes/"$NODE".yaml
+#    talm's post-upgrade verify gate confirms the node booted the new version
+#    (catches the silent A/B rollback case). Do NOT pass --skip-post-upgrade-verify.
+
+# 4. Wait for the node to come back Ready
+kubectl --context $CTX wait --for=condition=Ready node/"$NODE" --timeout=15m
+
+# 5. WAIT FOR DRBD RESYNC on this node before touching the next one.
+#    All resources back UpToDate, nothing SyncTarget / Inconsistent / Connecting:
+linstor resource list --faulty    # must be empty again
+linstor exec -- drbdsetup status | grep -E 'SyncTarget|Inconsistent|Connecting' || echo clean
+
+# 6. Uncordon
+kubectl --context $CTX uncordon "$NODE"
+```
+
+Only after step 5 reports clean do you move to the next node. Large volumes can take a while to resync — that wait is the whole point.
+
+## After a node reboots: watch for the auto-diskful pause
+
+A VM that restarts onto a node **without a local replica** opens its disk *diskless*, and LINSTOR `auto-diskful` may then convert it to *diskful* under the running VM. On DRBD 9.2.16 (Talos ≤ 1.12) that online toggle can leave the VM in a `PausedIOError` loop — see `known-failures.md` #8. Two ways to shrink the exposure during a node-upgrade window:
+
+- Prefer scheduling VMs onto nodes that already hold a replica (volume affinity), or run enough replicas that most nodes have a local copy — fewer diskless opens, fewer live toggles.
+- If a VM does get stuck, `virtctl restart <vm>` reopens the device cleanly.
+
+## Rollback
+
+Talos upgrades are A/B: a failed boot rolls back to the previous partition automatically, and talm's post-upgrade verify catches the silent-rollback case. There is no "roll the whole cluster back" step — you fix the one node and re-run. Never proceed to the next node while any node is on the wrong version or any resource is still resyncing.