Skip to content

Fall back to a full reconnect when a resume cannot renegotiate the publisher - #1301

Open
j-mercker wants to merge 1 commit into
livekit:mainfrom
j-mercker:j-mercker/resume-escalation
Open

j-mercker wants to merge 1 commit into
livekit:mainfrom
j-mercker:j-mercker/resume-escalation

Conversation

@j-mercker

Copy link
Copy Markdown

Problem

When a resume starts while the publisher is still waiting for the answer to an earlier offer, every resume attempt fails the same way and the room disconnects with reconnectAttemptsExceeded, although the signal connection comes back each time and a full reconnect or a rejoin works at once.

Seen on web with 2.13.2 on LiveKit Cloud: 5 users over 2 days, 1 to 13 failed resumes each. Two of the sessions, client time:

Windows, Chromium-based browser
+0.0 s    full reconnect (after a resume timed out), connected at +0.3 s
+11.0 s   microphone republish timed out (TrackPublishException, the rePublishAllTracks failure in #1274)
+15.8 s   attemptReconnect: resume failed: InvalidAccessError: Failed to execute 'setRemoteDescription' on
          'RTCPeerConnection': Failed to set remote answer sdp: The order of m-lines in answer doesn't match order
          in offer. Rejecting answer.
          8 more identical failures, no full reconnect attempted
+80.6 s   Disconnected, reconnectAttemptsExceeded
+126 s    user rejoined by hand, audio and video published within 1 s

iPhone (WebKit)
+0.0 s    microphone track ended; the app unpublished it and published the default microphone
+14.4 s   that publish timed out
+15.2 s   attemptReconnect: resume failed: InvalidAccessError: Failed to set remote answer sdp: Incompatible send direction
          2 more identical failures
+41.3 s   user reloaded the page

In each case a microphone publish had just timed out or its track had just ended.

Cause

A resume asks the publisher for an ICE-restart offer. If the publisher is in have-local-offer, Transport.createAndSendOffer first re-applies the previous remote answer. When the unanswered offer added or removed a track, that answer no longer fits it and the browser rejects it, with "The order of m-lines…" for an added or replaced track and "Incompatible send direction" for a removed one. The publisher stays in have-local-offer, so the next resume is rejected the same way.

attemptReconnect escalates a failed resume to a full reconnect only for WebSocketException and MediaConnectException, so the browser's InvalidAccessError keeps it on resume until the attempts run out.

The JS SDK re-applies the answer the same way (PCTransport.ts), but its attemptReconnect escalates every resume failure that is not a SignalReconnectError, so it recovers with a full reconnect.

The browser behavior reproduces with two local peer connections and no server: leave a change unanswered, then run the SDK's resume step three times. Chrome 158:

Pending change Each of 3 attempts State after
microphone publish … order of m-lines … have-local-offer
microphone replaced … order of m-lines … have-local-offer
microphone unpublish … Incompatible send direction have-local-offer
nothing pending (control) offer created have-local-offer
Repro page (open in a browser)
<!doctype html>
<meta charset="utf-8">
<title>Resume stale-answer repro</title>
<pre id="out">running…</pre>
<script>
// Reproduces, with two local RTCPeerConnections and no server, what livekit_client's
// PCTransport.createAndSendOffer(iceRestart: true) does when the publisher is still in
// have-local-offer: it re-applies the previous remote answer before creating the ICE-restart offer.
(async () => {
  const ac = new AudioContext();
  const audioTrack = () => ac.createMediaStreamDestination().stream.getAudioTracks()[0];
  const cv = Object.assign(document.createElement('canvas'), {width: 64, height: 64});
  cv.getContext('2d').fillRect(0, 0, 1, 1);
  const videoTrack = () => cv.captureStream(5).getVideoTracks()[0];

  async function negotiate(pub, srv, opts) {
    const offer = await pub.createOffer(opts);
    await pub.setLocalDescription(offer);
    await srv.setRemoteDescription(offer);
    const answer = await srv.createAnswer();
    await srv.setLocalDescription(answer);
    await pub.setRemoteDescription(answer);
  }

  // What the SDK does on a resume today.
  async function sdkResume(pub) {
    const r = {before: pub.signalingState};
    try {
      if (pub.signalingState === 'have-local-offer') {
        await pub.setRemoteDescription(pub.remoteDescription); // transport.dart: setRemoteDescription(currentSD)
      }
      await pub.setLocalDescription(await pub.createOffer({iceRestart: true}));
      r.result = 'ok';
    } catch (e) { r.error = `${e.name}: ${e.message}`; }
    r.after = pub.signalingState;
    return r;
  }

  async function scenario(name, setup, pendingChange) {
    const pub = new RTCPeerConnection(), srv = new RTCPeerConnection();
    pub.createDataChannel('_reliable');
    setup(pub);
    await negotiate(pub, srv);                               // published and answered
    pendingChange(pub);
    await pub.setLocalDescription(await pub.createOffer());  // offer sent, answer lost with the signal connection
    const attempts = [await sdkResume(pub), await sdkResume(pub), await sdkResume(pub)];
    pub.close(); srv.close();
    return {name, attempts};
  }

  const removeMic = pub => pub.removeTrack(pub.getSenders().find(s => s.track?.kind === 'audio'));
  const addMic = pub => pub.addTransceiver(audioTrack(), {direction: 'sendonly'});
  const results = [
    await scenario('microphone publish pending', () => {}, addMic),
    await scenario('microphone replaced (unpublish + publish) pending',
      pub => { addMic(pub); pub.addTransceiver(videoTrack(), {direction: 'sendonly'}); },
      pub => { removeMic(pub); addMic(pub); }),
    await scenario('microphone unpublish pending', addMic, removeMic),
  ];

  // Control: nothing pending, so the same resume step succeeds.
  {
    const pub = new RTCPeerConnection(), srv = new RTCPeerConnection();
    pub.createDataChannel('_reliable'); addMic(pub);
    await negotiate(pub, srv);
    results.push({name: 'control: nothing pending', attempts: [await sdkResume(pub)]});
    pub.close(); srv.close();
  }

  // Same pending offers, rolled back instead of answered with the stale answer.
  for (const [name, setup, change] of [['rollback: publish pending', () => {}, addMic], ['rollback: unpublish pending', addMic, removeMic]]) {
    const pub = new RTCPeerConnection(), srv = new RTCPeerConnection();
    pub.createDataChannel('_reliable'); setup(pub);
    await negotiate(pub, srv); change(pub);
    await pub.setLocalDescription(await pub.createOffer());
    const r = {name};
    try {
      await pub.setLocalDescription({type: 'rollback'});
      await negotiate(pub, srv, {iceRestart: true});
      r.result = `ok, state ${pub.signalingState}`;
      // The answered ICE-restart offer still carries the pending change.
      r.audioDirectionsInOffer = pub.localDescription.sdp.split('\r\n').filter((l, i, a) =>
        /^a=(sendonly|recvonly|inactive|sendrecv)$/.test(l) && a.slice(0, i).reverse().find(x => x.startsWith('m='))?.startsWith('m=audio'));
    } catch (e) { r.error = `${e.name}: ${e.message}`; }
    results.push(r); pub.close(); srv.close();
  }
  document.getElementById('out').textContent = navigator.userAgent + '\n\n' + JSON.stringify(results, null, 2);
})();
</script>

Fix

  • Transport.createAndSendOffer throws NegotiationError when it cannot restore the previous answer for an ICE restart.
  • attemptReconnect escalates a resume that failed with NegotiationError to a full reconnect, as _onPublisherNegotiationError already does outside a resume.

Only resumeConnection asks for an ICE restart, so only the resume path changes. On that path the existing NegotiationError from a rejected SDP munge (setMungedSDP) now escalates the resume too.

Testing

  • New unit test in test/core/transport_negotiate_test.dart, "an ICE restart that cannot restore the previous answer throws a NegotiationError" (and sends no offer). It fails on main and passes here.
  • New test/core/resume_negotiation_failure_test.dart, "a resume that fails with a NegotiationError is retried as a full reconnect". It fails on main (the next attempt is another resume) and passes here. E2EContainer takes an optional peerConnectionCreate for it, and the stale-answer fake is shared in test/mock/peerconnection_mock.dart.
  • Full suite: 609 passed, 1 skipped. Analyze, format, import order and version checks pass.

Notes

@CLAassistant

CLAassistant commented Nov 22, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants