fix(core): release inbound migration slot when VMI is deleted#2595
Draft
danilrwx wants to merge 1 commit into
Draft
fix(core): release inbound migration slot when VMI is deleted#2595danilrwx wants to merge 1 commit into
danilrwx wants to merge 1 commit into
Conversation
The inbound migration limiter caps concurrent inbound live migrations per node with an in-memory slot registry. A slot is only released when the owning VMI reaches a terminal MigrationState, and the KVVMI watcher dropped delete events, so a VMI deleted mid-migration (e.g. cleanup, or a launcher that went unresponsive before the migration failed) leaked its slot forever. With the default limit of 1 per node this starves every subsequent inbound migration to that node: targets wait for a slot past the qemu-timeout and panic with 'timed out waiting for domain to be defined'. Reconcile on VMI delete and release any slot held by that VMI. A deleted VMI cannot have an active migration, so dropping its slots is safe. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The inbound migration limiter caps concurrent inbound live migrations per node with an in-memory slot registry (default limit
1). A slot was only released when the owning VMI reached a terminalMigrationState(HandleobservingCompleted/Failed), and the KVVMI watcher dropped delete events (DeleteFuncreturnedfalse). So a VMI deleted mid-migration leaked its slot forever.This change:
DeleteFunc→true).InboundMigrationLimiter.ReleaseByVMI(namespace, name).Why do we need it, and what problem does it solve?
A VMI can disappear before its migration reaches a terminal state — test cleanup deletes the VM, or the source launcher goes unresponsive (
No command socket found for vmi) and the VMI is removed. In those casesReleasewas never called, so the in-memory slot stayed occupied by a ghost owner.With the default limit of 1 inbound migration per node, one leaked slot starves every subsequent inbound migration to that node. Since the migration target node is effectively fixed (source node excluded, control-plane tainted), all later migrations queue on the same node, wait for a slot past the target launcher's
--qemu-timeout(~5 min), and the target panics withtimed out waiting for domain to be defined→migration is not completed.This was observed in the Replicated E2E nightly: dozens of migrations succeeded, then from one point on every migration failed with the VMI stuck at
inbound-migration-slot: waitingon the same target node and no live VMI holding the slot — the ghost-slot signature.Releasing on delete is safe: a deleted VMI cannot have an active migration — the migration controller fails the migration when
vmi == nil(FailedReasonVMIDoesNotExist), and the limiter's own invariant (isInActiveMigration,Restore) only ever tracks a live VMI with a non-terminalMigrationState.What is the expected result?
A VMI deleted mid-migration frees its inbound slot immediately, so subsequent inbound migrations to that node are no longer starved. Covered by unit tests for
ReleaseByVMI(release by VMI when the migration UID is unknown; must not release a different VMI that shares a name prefix).Checklist
Changelog entries