Summary. On a domain-joined Windows host, the 15-second spinlock in shared.cc that is
meant to serialise one-time shared-memory initialisation can time out while the leading
process is still inside the critical section — typically blocked in internal_getpwsid()
on an account lookup. On expiry the spinlock admits a waiting process, which calls
mountinfo.init() a second time; mount_info::create_root_entry (mount.cc:555) then
attempts to insert / into a table that already has an immutable /, returns EPERM, and
api_fatal() kills the process before it executes a line:
sh.exe: *** fatal error - add_item ("\??\C:\Program Files\Git", "/", ...) failed, errno 1
The underlying mechanism was first described on the msys2-users list in 2016
(https://sourceforge.net/p/msys2/mailman/message/35070454/), but the reporter's own closing
question — whether removing the 15s timeout would be safe — went unanswered, and the same
timeout is present in the build this report was measured against.
Impact. Every subsequently-started MSYS process on the host fails identically for as long
as at least one holder keeps the shared region alive. Observed as a total, hours-long outage
of bash, sh, and every POSIX script on a shared Windows host, clearing only when the
holder count reached exactly zero. On a host running multiple tools that each spawn
short-lived shells (editor integrations, CI-style automation, IDE plugins), the count does
not reach zero on its own, so the outage can persist for hours rather than seconds.
Reproduction conditions. Domain-joined host; nsswitch.conf resolving passwd via db
(the Windows/AD-backed source); several MSYS processes starting concurrently; a cold account
lookup. Measured cold lookup on the reporting host: 11.1s against the 15s spinlock timeout —
74% of the budget consumed by a single account lookup before any concurrency is even added.
Environment. Git for Windows 2.55.0.windows.3; msys2-runtime commit b4195d69
(this fork); Windows 11 Pro build 26200.
Evidence gathered on the reporting host, beyond the original 2016 report:
-
A measured latency budget. A cold SID→name translation of one user's 16 groups (the
work internal_getpwsid() performs) took 11,093 ms against the 15,000 ms timeout — 74%
of the budget. Warm, the same work is 2–8 ms (LSA caching), which explains the
intermittency: the fault is load- and cache-state-dependent, not constant.
-
Concurrency data showing recovery is load-dependent. The shared region is released
only at exactly zero holders. While multiple MSYS-spawning processes were actively
running, 40 samples over 36s found 3–12 concurrent holders and zero samples at zero
holders; with the same processes idle, holds of at most 26.1s were observed with the
zero-holder window opening freely. A busy host stays faulted for hours; a quiet one
self-heals in seconds.
-
A clean self-heal observed directly. The fault was total (probe timing out at
15,068 ms, process exiting 0xC0000005), then cleared with nothing killed and no reboot,
in the same moment the holder count reached zero — then re-fired later the same hour.
Direct confirmation this is transient shared state, not installation damage.
-
A commonly-suspected explanation ruled out. Windows fast-startup/hybrid-shutdown was
not implicated on the reporting host (HiberbootEnabled = 0, no boot of type 0x1 in the
event record spanning the observation window) — worth noting since fast-startup is a
natural first guess for anything reboot-adjacent, and it wasn't the cause here.
Suggested directions.
(a) Treat spinlock expiry as an error rather than as permission to proceed, so a late waiter
re-checks initialisation state instead of assuming ownership of an uninitialised table.
(b) Make create_root_entry idempotent — an existing, identical / entry is not itself a
reason to abort.
(c) Move the account lookup out of the critical section entirely, so a slow LDAP round trip
cannot block a competing process's initialisation regardless of the timeout value.
Workaround, applied and verified on the reporting host. Take the account lookup off the
startup path: mkpasswd -l -c > /etc/passwd, mkgroup -l -c > /etc/group, then
passwd: files in nsswitch.conf. The -c (current user) flag matters on a domain-joined
host — -l alone (local accounts only) omits the actual domain account MSYS needs to
resolve, which would silently defeat the fix. Applied and measured on the reporting host:
cold account-lookup latency fell from 11,093 ms to 792 ms (93% reduction). This addresses the
trigger rather than the underlying race — it does not fix the abandonment-recovery defect in
the spinlock itself, only removes the input most likely to trip it on a domain-joined host.
Filed from an independent investigation of this fault on a shared multi-tenant Windows host;
happy to share further measurement data or reproduce specific scenarios on request.
Summary. On a domain-joined Windows host, the 15-second spinlock in
shared.ccthat ismeant to serialise one-time shared-memory initialisation can time out while the leading
process is still inside the critical section — typically blocked in
internal_getpwsid()on an account lookup. On expiry the spinlock admits a waiting process, which calls
mountinfo.init()a second time;mount_info::create_root_entry(mount.cc:555) thenattempts to insert
/into a table that already has an immutable/, returnsEPERM, andapi_fatal()kills the process before it executes a line:The underlying mechanism was first described on the msys2-users list in 2016
(
https://sourceforge.net/p/msys2/mailman/message/35070454/), but the reporter's own closingquestion — whether removing the 15s timeout would be safe — went unanswered, and the same
timeout is present in the build this report was measured against.
Impact. Every subsequently-started MSYS process on the host fails identically for as long
as at least one holder keeps the shared region alive. Observed as a total, hours-long outage
of
bash,sh, and every POSIX script on a shared Windows host, clearing only when theholder count reached exactly zero. On a host running multiple tools that each spawn
short-lived shells (editor integrations, CI-style automation, IDE plugins), the count does
not reach zero on its own, so the outage can persist for hours rather than seconds.
Reproduction conditions. Domain-joined host;
nsswitch.confresolvingpasswdviadb(the Windows/AD-backed source); several MSYS processes starting concurrently; a cold account
lookup. Measured cold lookup on the reporting host: 11.1s against the 15s spinlock timeout —
74% of the budget consumed by a single account lookup before any concurrency is even added.
Environment. Git for Windows
2.55.0.windows.3;msys2-runtimecommitb4195d69(this fork); Windows 11 Pro build 26200.
Evidence gathered on the reporting host, beyond the original 2016 report:
A measured latency budget. A cold SID→name translation of one user's 16 groups (the
work
internal_getpwsid()performs) took 11,093 ms against the 15,000 ms timeout — 74%of the budget. Warm, the same work is 2–8 ms (LSA caching), which explains the
intermittency: the fault is load- and cache-state-dependent, not constant.
Concurrency data showing recovery is load-dependent. The shared region is released
only at exactly zero holders. While multiple MSYS-spawning processes were actively
running, 40 samples over 36s found 3–12 concurrent holders and zero samples at zero
holders; with the same processes idle, holds of at most 26.1s were observed with the
zero-holder window opening freely. A busy host stays faulted for hours; a quiet one
self-heals in seconds.
A clean self-heal observed directly. The fault was total (probe timing out at
15,068 ms, process exiting
0xC0000005), then cleared with nothing killed and no reboot,in the same moment the holder count reached zero — then re-fired later the same hour.
Direct confirmation this is transient shared state, not installation damage.
A commonly-suspected explanation ruled out. Windows fast-startup/hybrid-shutdown was
not implicated on the reporting host (
HiberbootEnabled = 0, no boot of type0x1in theevent record spanning the observation window) — worth noting since fast-startup is a
natural first guess for anything reboot-adjacent, and it wasn't the cause here.
Suggested directions.
(a) Treat spinlock expiry as an error rather than as permission to proceed, so a late waiter
re-checks initialisation state instead of assuming ownership of an uninitialised table.
(b) Make
create_root_entryidempotent — an existing, identical/entry is not itself areason to abort.
(c) Move the account lookup out of the critical section entirely, so a slow LDAP round trip
cannot block a competing process's initialisation regardless of the timeout value.
Workaround, applied and verified on the reporting host. Take the account lookup off the
startup path:
mkpasswd -l -c > /etc/passwd,mkgroup -l -c > /etc/group, thenpasswd: filesinnsswitch.conf. The-c(current user) flag matters on a domain-joinedhost —
-lalone (local accounts only) omits the actual domain account MSYS needs toresolve, which would silently defeat the fix. Applied and measured on the reporting host:
cold account-lookup latency fell from 11,093 ms to 792 ms (93% reduction). This addresses the
trigger rather than the underlying race — it does not fix the abandonment-recovery defect in
the spinlock itself, only removes the input most likely to trip it on a domain-joined host.
Filed from an independent investigation of this fault on a shared multi-tenant Windows host;
happy to share further measurement data or reproduce specific scenarios on request.