Straggler DetachVolume job clears attachment state of a newer attach on another VM, leaving an orphaned disk in libvirt (KVM)

[dloszewski-cni]

problem

A stale DetachVolume async job that completes after the same volume has already been re-attached (to another VM) clears the volume's attachment state in the database unconditionally. The result is a persistent split-brain: the DB shows the volume detached while the hypervisor still has the disk plugged into the original guest. Every subsequent AttachVolume to that guest is then allocated the "free" device id and fails in libvirt with XML error: target 'vdX' duplicated, permanently rejecting all volume attaches to that VM until an operator manually runs virsh detach-disk.

Root factors, as observed:

1. VM work jobs are serialized per VM, not per volume — so a detach of volume V from VM-A and an attach of volume V to VM-B run on independent queues with no ordering guarantee.
2. Clients that retry (Kubernetes CSI external-attacher, ~60 s retry) stack multiple identical DetachVolume jobs on the source VM's queue. The first succeeds; the stragglers also "succeed" later.
3. The straggler's completion path clears volumes.instance_id / device_id without verifying the row still refers to the attachment it detached — no compare-and-swap. If the volume was re-attached in the meantime, the new attachment's state is destroyed.
4. The corrupting event logs at INFO level as a successful detach — nothing flags the inconsistency until a later attach collides.

Observed occurrences (four in 20 days, production; all on 4.22.1.0, KVM, RBD primary storage)

Occurrence 1 — 2026-08-27/28, "worker7": straggler detach job-148750 cleared the state of an attachment established by job-148738; collisions observed via job-148762 et seq.

Occurrence 2 — 2026-09-08, "worker7"/"worker8" (management-server.log timeline, UTC):

Time | Job | Event -- | -- | -- 14:34:56 | job-210187 | DetachVolume #1 of volume e949a8c4 from worker1 (CSI-initiated) 14:35:56–14:37:57 | job-210205 / 210223 / 210235 | CSI ~60 s retries stack detaches #2–4 on worker1's queue 14:38:51 | job-210187 SUCCEEDED | real unplug from worker1 14:40:10→19 | job-210247 SUCCEEDED | volume attached to worker7, deviceid 2 (vdc), DB correct 14:42:40 | job-210235 SUCCEEDED | straggler completes 2m21s after the attach and clears the worker7 attachment row — orphan created; logged as a successful detach at INFO 14:43:04 onward | jobs 210253/210259/210265/210268/210277 FAILED 530 | attach collisions begin

Two points specific to this occurrence:

• The stale attachment's consumer kept running and doing I/O on worker2 for five days while the DB said the volume was detached — the DB lost track of a live, in-use attachment. That is a data-integrity hazard: the volume could have been offered to a second VM while mounted read-write in the first, and a live detach by an operator trusting the DB would have pulled a disk out from under a running workload.
• A stale detach that reaches the hypervisor path fails with an NPE in DiskTO.getDiskSeq() rather than a clean "volume is not attached to this VM" rejection; a stale detach that does not reach that path (job-185059) succeeds and clears state it does not own. Which one a straggler hits looks timing-dependent.

Occurrence 4 — attach-side variant, 2026-09-10 → 09-16, "worker4": the mirror image of the straggler-detach cases: here the DB lost a successful attach. Volume 1a6a7be4 (an RWO PVC backing a database StatefulSet member) was live at target vde / deviceid 4 on worker4 — libvirt and the Kubernetes VolumeAttachment both showed attached — while the volumes table showed it detached (state Ready, no instance). The corrupting job aged out of the async_job table before discovery (~6 days in), so the exact write-loss path is unrecovered, but the allocator consequence is identical: deviceid 4 is considered free, every subsequent AttachVolume to worker4 picks it, and fails target 'vde' duplicated — measured ~3,700 failed attach events/day for six days, with four unrelated PVCs unable to schedule onto the VM.

This occurrence also nearly realized the data-integrity hazard flagged in occurrence 3: when the consuming pod was later evicted, the CSI ControllerUnpublish "succeeded" as a no-op (the DB already said detached), and CloudStack then attached the volume to a second VM while the original guest's qemu still held the RBD image open read-write — a silent double-open of a database volume across two hypervisors, with no error or warning anywhere in the API path. No corruption resulted only because the first guest had unmounted the filesystem before the re-attach.

Actual results

Unconditional clear of the attachment row; persistent DB↔libvirt divergence; all subsequent attaches to the affected VM fail with libvirt duplicated-target errors (530) until manual operator intervention (virsh detach-disk <vm> <target> --live on the host).

versions

4.22.1.0 (KVM on Ubuntu, Ceph RBD primary storage, advanced networking)

Also reviewed 4.23.0 (2026-09-17): no change to the DetachVolume completion / volume-attachment state path that would address this behavior, and no existing upstream issue or PR describing it that we could find.

Component: Volume lifecycle / async job orchestration (DetachVolume completion), KVM Kubernetes CSI driver (csi.cloudstack.apache.org) drives the volume attach/detach traffic.

The steps to reproduce the bug

Environment: CloudStack 4.22.1.0, KVM, Ceph RBD primary storage, a Kubernetes cluster using the CloudStack CSI driver (csi.cloudstack.apache.org), where pod rescheduling moves an RWO PVC between worker VMs.

1. Have a data volume attached to worker VM-A.
2. Trigger rapid relocation of the volume (e.g. reschedule the consuming pod to VM-B). The CSI external-attacher submits DetachVolume; under load its ~60 s retries enqueue several duplicate detach jobs on VM-A's work queue.
3. First detach job completes (real unplug from VM-A); CSI attaches the volume to VM-B; DB is briefly correct (volume → VM-B, deviceid N).
4. A straggler detach job from step 2 completes minutes later and clears the volume's attachment row.
5. DB now: volume "Ready"/detached. Hypervisor: disk still defined in VM-B's domain at target vdN. (In one occurrence the stale disk remained in the previous VM's domain instead — both variants observed.)
6. Any subsequent AttachVolume to the affected VM picks deviceid N and fails: org.libvirt.LibvirtException: XML error: target 'vdN' duplicated for disk sources '' and '' (API errorcode 530). The VM rejects all further volume attaches.

What to do about it?

Expected behavior / suggested fix:

DetachVolume completion should only clear the volume's attachment state if it still refers to the attachment the job actually detached — i.e. a conditional update / compare-and-swap on (volume_id, instance_id[, device_id]) against the job's source VM. A stale detach whose volume has since been re-attached elsewhere should complete as a no-op (or fail), leaving the newer attachment state intact. Ideally, duplicate detach jobs for the same volume would also be coalesced, or volume-level serialization applied.

Workaround:

Manual: compare virsh domblklist against the volumes table per VM; live detach the orphaned target. For the attach-side variant (occurrence 4), a gentler remediation avoids touching a possibly-live disk: drain the guest's workloads, then stop/start the VM through CloudStack — on start the domain XML is rebuilt from the (now-consistent) DB and the stale target disappears. Mitigations deployed on our side: reconciliation alerting between libvirt and the DB, and increasing the CSI external-attacher retry backoff to reduce detach-job stacking

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!