fix: guard shared instance maps with a RWMutex - #127
Conversation
Three maps are created once in cmd/evolution-go/main.go — killChannel (map[string]chan bool), clientPointer (map[string]*whatsmeow.Client) and myClientPointer (map[string]*MyClient) — and shared by reference across every service package (whatsmeow, instance, sendMessage, chat, call, community, group, label, message, newsletter, user) and every *MyClient value. None of the reads, writes or deletes on them were synchronized. In production this fatals the process: ReconnectClient (whatsmeow.go) deletes from all three maps without a lock, StartClient writes to clientPointer/myClientPointer while its select loop and other goroutines (schedulePresenceUpdates, teardownQR, myEventHandler's LoggedOut/poll-vote paths) read killChannel/clientPointer, and every HTTP handler in the peripheral packages reads clientPointer via ensureClientConnected — all concurrently, since events.Disconnected fires ReconnectClient in its own goroutine while requests keep flowing in. Go's runtime treats concurrent map writes, or a read racing a write, as fatal — unrecoverable, not a panic — and that is exactly what showed up in production as internal/runtime/maps.fatal during bursts of reconnects. The fix adds one exported sync.RWMutex (ClientMapsMu, in the new pkg/whatsmeow/service/client_maps.go) shared by every package that touches these maps. Every indexed read takes RLock/RUnlock, every write (assign or delete) takes Lock/Unlock, and every guarded scope is kept as short as possible: the client/channel value is copied into a local variable while holding the lock, the lock is released, and only then is the value used for anything that can block or take a while (IsConnected/Disconnect, channel sends inside select, StartInstance/ReconnectClient/StartClient, network calls). This also closes a couple of adjacent bugs for free: StartClient's kill-signal select and schedulePresenceUpdates now re-resolve the channel from the map on each loop iteration instead of evaluating a stale expression, and ClearInstanceCache/Delete's check-then-delete on killChannel is now atomic, so two concurrent cleanups can no longer double-close the same channel. No lock is ever held across a call into another guarded function (ReconnectClient -> StartInstance -> StartClient all close their lock scopes before calling the next), so there is no new deadlock risk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @FlavioPulli, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
|
We hit this in production — confirming the bug with a crash trace. Running
The goroutine dump at the time of the crash shows the concurrency context: The trigger appears to be concurrent connects: in our case, several instances reconnecting while another was in an active QR pairing flow and webhooks were being dispatched. Worth noting that the codebase already acknowledges the hazard — the comment on This PR addresses exactly that. Any chance of getting it reviewed? Operational impact for anyone else hitting this: restarting the container does not bring instances back — they stay down until an explicit Happy to put together a minimal reproduction (parallel |
|
@Matheusagostinho thank you — that is the most useful thing that has happened to this PR. Two independent deployments, two different scales, same goroutine dump. Your On our side the trigger is the same shape: a deploy recreates the container, the API boot fires the connects nearly together, and one instance ends up in a state where One thing that may help you while this waits: Housekeeping: on #128 @iagocotta asked that PRs target Your offer of a minimal reproduction with parallel |
2 participants
killChannel,clientPointerandmyClientPointerare created inmain.goand shared by reference across all service packages and everyMyClient.ReconnectClientdeletes from all three without a lock,StartClientwrites to them, theDisconnectedhandler triggersReconnectClientfrom a goroutine, and every HTTP handler reads them concurrently. Go fatals on concurrent map access — we hit this in production asfatal error: concurrent map writes/internal/runtime/maps.fatalduring reconnection bursts (stack throughwhatsmeowService.StartClient/ReconnectClient), which kills the entire process and every connected instance with it.This adds a single exported
sync.RWMutexin the whatsmeow service package guarding the three maps everywhere they are touched. Lock scopes are kept minimal: values are copied out underRLockand the lock is released before any long call (Disconnect,Connect,StartInstance, channel sends), so no lock is ever held across network operations. As a side effect the killChannel check-and-delete paths became atomic, closing a latent double-close race.go build ./...andgo vet ./...clean.