Skip to content

fix(core): release inbound migration slot when VMI is deleted#2595

Draft
danilrwx wants to merge 1 commit into
mainfrom
fix/inbound-migration-slot-leak
Draft

fix(core): release inbound migration slot when VMI is deleted#2595
danilrwx wants to merge 1 commit into
mainfrom
fix/inbound-migration-slot-leak

Conversation

@danilrwx

@danilrwx danilrwx commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Description

The inbound migration limiter caps concurrent inbound live migrations per node with an in-memory slot registry (default limit 1). A slot was only released when the owning VMI reached a terminal MigrationState (Handle observing Completed/Failed), and the KVVMI watcher dropped delete events (DeleteFunc returned false). So a VMI deleted mid-migration leaked its slot forever.

This change:

  • Reconciles on VMI delete (DeleteFunctrue).
  • Releases any slot held by the deleted VMI in the reconciler when the VMI is absent, via a new InboundMigrationLimiter.ReleaseByVMI(namespace, name).
  • Wires the limiter into the reconciler.

Why do we need it, and what problem does it solve?

A VMI can disappear before its migration reaches a terminal state — test cleanup deletes the VM, or the source launcher goes unresponsive (No command socket found for vmi) and the VMI is removed. In those cases Release was never called, so the in-memory slot stayed occupied by a ghost owner.

With the default limit of 1 inbound migration per node, one leaked slot starves every subsequent inbound migration to that node. Since the migration target node is effectively fixed (source node excluded, control-plane tainted), all later migrations queue on the same node, wait for a slot past the target launcher's --qemu-timeout (~5 min), and the target panics with timed out waiting for domain to be definedmigration is not completed.

This was observed in the Replicated E2E nightly: dozens of migrations succeeded, then from one point on every migration failed with the VMI stuck at inbound-migration-slot: waiting on the same target node and no live VMI holding the slot — the ghost-slot signature.

Releasing on delete is safe: a deleted VMI cannot have an active migration — the migration controller fails the migration when vmi == nil (FailedReasonVMIDoesNotExist), and the limiter's own invariant (isInActiveMigration, Restore) only ever tracks a live VMI with a non-terminal MigrationState.

What is the expected result?

A VMI deleted mid-migration frees its inbound slot immediately, so subsequent inbound migrations to that node are no longer starved. Covered by unit tests for ReleaseByVMI (release by VMI when the migration UID is unknown; must not release a different VMI that shares a name prefix).

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: core
type: fix
summary: Release the inbound migration slot when a VMI is deleted mid-migration to avoid starving further migrations to the node.
impact_level: low

The inbound migration limiter caps concurrent inbound live migrations per
node with an in-memory slot registry. A slot is only released when the
owning VMI reaches a terminal MigrationState, and the KVVMI watcher dropped
delete events, so a VMI deleted mid-migration (e.g. cleanup, or a launcher
that went unresponsive before the migration failed) leaked its slot forever.
With the default limit of 1 per node this starves every subsequent inbound
migration to that node: targets wait for a slot past the qemu-timeout and
panic with 'timed out waiting for domain to be defined'.

Reconcile on VMI delete and release any slot held by that VMI. A deleted
VMI cannot have an active migration, so dropping its slots is safe.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx danilrwx added this to the v1.10.0 milestone Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant