Skip to content

Republish the remaining tracks and finish the reconnect when one track fails to republish - #1302

Open
j-mercker wants to merge 2 commits into
livekit:mainfrom
j-mercker:j-mercker/republish-failure
Open

j-mercker wants to merge 2 commits into
livekit:mainfrom
j-mercker:j-mercker/republish-failure

Conversation

@j-mercker

@j-mercker j-mercker commented Nov 22, 2026 •

Copy link
Copy Markdown

Fixes #1274. Alternative to #1279: this also republishes the tracks after the failing one.

Problem

After a full reconnect, one track that fails to republish leaves that track and every track after it unpublished, and the room never emits RoomReconnectedEvent. Remote track settings are not re-sent and listeners are not notified, so an app waiting for the reconnect stays in its reconnecting state.

#1274 shows it on Android and iOS. On web, with 2.13.2 on LiveKit Cloud, the trigger is a third one: after a full reconnect the microphone's AddTrackRequest gets no answer within the publish timeout, so Engine.addTrack throws TrackPublishException (engine.dart:342 in 2.13.2, line 407 on main). Two production sessions, client time relative to the full reconnect:

Session 1
+0.0 s    full reconnect starts (after a resume timed out)
+0.3 s    connected
+11.0 s   uncaught: LiveKit Exception: Failed to publish track
          the server sees no microphone publication after this; the participant rejoined by hand at +126 s

Session 2
+0.0 s    full reconnect starts
+0.4 s    connected
+10.0 s   uncaught: LiveKit Exception: Failed to publish track
          the participant, who had been speaking with audio published for about 2 h, stayed in the room for
          another 2 h 13 min with no microphone publication

The web stack ends at the Engine.addTrack timeout. The app had no publish of its own in flight, so we infer the caller was rePublishAllTracks(). In session 1 the unanswered publish offer then made every later resume fail, which #1301 fixes.

Cause

The EngineRestartedEvent handler in Room awaits localParticipant.rePublishAllTracks() without a try/catch. rePublishAllTracks() clears trackPublications, then republishes each track in turn, also without a catch. The first failure ends the loop, then escapes the handler before it re-sends track settings, emits RoomReconnectedEvent and notifies listeners.

Fix

  • rePublishAllTracks() catches a failure per track, logs it as a warning, and moves on to the next track.
  • A republish failure therefore no longer aborts the EngineRestartedEvent handler.
  • After republishing, the handler returns without announcing success if the engine closed or the connection is no longer connected, for example when the server deleted the room while a publish was pending. This matches the abort check after republishing in the JS SDK's handleSignalRestarted.

The JS SDK republishes tracks in parallel (Promise.all in LocalParticipant.republishAllTracks), so one failure doesn't stop the others, and handleSignalRestarted catches the error, logs it and carries on. Dart republishes one track at a time, so the catch goes per track to get the same result. #1279 catches in the handler only, which, as its description notes, still leaves the tracks after the failing one unpublished.

Testing

  • New test in test/core/room_e2e_test.dart, "full reconnect / republish failures do not stop other tracks or the reconnection": two local audio tracks whose republish throws. RoomReconnectedEvent is emitted and the second republish is still attempted. It fails on main and passes here.
  • New test in the same group, "no reconnected event when the server closes the room while a track is republished": it sends a ROOM_DELETED leave while a republish is pending. It fails without the check (RoomReconnectedEvent fires after RoomDisconnectedEvent) and passes here.
  • Full suite: 609 passed, 1 skipped. Analyze, format, import order and version checks pass.

Notes

  • Not a regression. 2.13.2, 2.14.0 and main behave the same.
  • A failed track's only signal to the app is now the warning log; before, it surfaced as an uncaught async error.

@CLAassistant

CLAassistant commented Nov 22, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment on lines +652 to +653
} catch (error, stackTrace) {
logger.warning('Failed to re-publish ${track.source} track ${track.sid}: $error', error, stackTrace);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Camera restarts after Unpublish all

While microphone republish waits, Unpublish all misses the camera in rePublishAllTracks’s private snapshot. After the microphone times out, the loop publishes the camera despite the user’s stop request.

Learn more

The example app’s Unpublish all control calls unpublishAllTracks() after confirmation and remains available on the room page during reconnect. unpublishAllTracks only sees entries in trackPublications. rePublishAllTracks clears that map before waiting for the first publish, so an unpublish request during that wait never reaches the queued camera. Before this change, a timeout on the first track stopped the loop, leaving the queued camera unpublished. Now the loop attempts the camera after the timeout, without honoring the completed Unpublish all request.

Example: A participant reconnects with microphone and camera enabled. While the microphone waits for its publish response, they confirm Unpublish all. The microphone times out, but the server answers the camera’s publish request, so video resumes.

Recommended fix: Make unpublishAllTracks() cancel pending reconnect publications as well as registered publications. Check that cancellation before attempting each queued track and reconcile any publish already in flight. Test Unpublish all during a pending microphone republish followed by a timeout, and assert that the camera never publishes.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The race is real, but it isn't new. On main the same thing happens whenever the microphone republish succeeds: rePublishAllTracks() has already cleared trackPublications, so unpublishAllTracks() during the wait can't see the queued camera. This PR only extends it to the case where the first republish fails. Cancelling queued republishes needs a cancellation hook in rePublishAllTracks(), which seems better as its own change.

Comment thread lib/src/participant/local.dart Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Full reconnect: a rePublishAllTracks() failure escapes the EngineRestartedEvent handler and RoomReconnectedEvent is never emitted

2 participants