Fix snapshot automount race causing AVL tree panic #17943
+87
−37
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amotin
previously approved these changes
Nov 18, 2025
Member
amotin
left a comment
amotin
left a comment
There was a problem hiding this comment.
It makes me wonder why parallel mounts may somehow succeed, considering identical sequential ones are failing. But if this is a state of things, then I have no objections.
amotin
dismissed
their stale review
It seems multiple successful parallel mount calls result in multiple mounts, which is not right. It needs deeper investigation and possibly different solution.
ixhamza
force-pushed
the
SEE-365-avl-panic
branch
from
f2010f8 to
e8584c7
Compare
ixhamza
force-pushed
the
SEE-365-avl-panic
branch
from
e8584c7 to
2b319c4
Compare
Multiple threads racing to automount the same snapshot can both spawn mount helper processes that successfully complete, causing both parent threads to attempt AVL tree registration and triggering a VERIFY() panic in avl_add(). This occurs because the fsconfig/fsmount API lacks the serialization provided by traditional mount() via lock_mount(). The fix adds a per-entry mutex (se_mtx) to zfs_snapentry_t that serializes mount and unmount operations on the same snapshot. The first mount thread creates a pending entry with se_spa=NULL and holds se_mtx during the helper execution. Concurrent mounts find the pending entry and return success without spawning duplicate helpers. Unmount waits on se_mtx if a mount is pending, ensuring proper serialization. This allows different snapshots to mount in parallel while preventing the AVL panic. Signed-off-by: Ameer Hamza <ahamza@ixsystems.com>
ixhamza
force-pushed
the
SEE-365-avl-panic
branch
from
2b319c4 to
1a568de
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
This fixes a race condition that causes kernel panics when multiple processes simultaneously access a fresh snapshot. The bug triggers a
VERIFY()assertion failure in the AVL tree code when concurrent threads attempt to add identical entries during snapshot automount.Description
The race condition occurs in
zfsctl_snapshot_mount()due to a time-of-check-time-of-use (TOCTOU) bug between checking if a snapshot is mounted and adding it to the AVL tree. The sequence is:zfsctl_snapshot_ismounted()- returns FALSEcall_usermodehelper())VERIFY()assertion failsThe fix adds a pending entry mechanism with per-entry mutex synchronization. The first mount thread creates a pending AVL entry and holds se_mtx during helper execution. Concurrent mounts find the pending entry and return success without spawning duplicate helpers, preventing the AVL panic.
Kernel Stack Trace:
How Has This Been Tested?
Reproduction script:
Results:
Types of changes
Checklist:
Signed-off-by.