Audio EOT

[chenghao-mou]

Requires livekit/protocol#1485

Adds streaming audio end-of-turn detection. Single user-facing AudioTurnDetector in livekit-plugins-turn-detector selects between two backends:

• eot-audio-cloud
• eot-audio-mini

On cloud transport error or predict_end_of_turn timeout, the session swaps to local for the rest of the stream (sticky per session, one warning per failure mode). Local failures emit the default 1.0 prediction and retry on the next turn. A user-set unlikely_threshold is scaled multiplicatively against the cloud default so the operating point survives a fallback.

Wired into AudioRecognition: VAD INFERENCE_DONE triggers warmup, END_OF_SPEECH activates the stream, predictions flow back through _run_eou_detection and arbitrate against the endpointing delay. A speaking guard cancels an in-flight bounce if VAD START_OF_SPEECH fires mid-window.

Structure

• livekit/agents/voice/turn.py — abstract _AudioTurnDetector / _AudioTurnDetectorStream (FSM) live alongside the existing _TurnDetector Protocol.
• livekit/plugins/turn_detector/audio.py — unified detector + concrete FSM stream that dispatches to the active transport.
• livekit/plugins/turn_detector/transports.py — AudioTurnDetectionTransport Protocol + _CloudTransport (WS + protobuf) + _LocalTransport (ctypes). Fallback swaps the transport instance, not the stream.
• livekit/plugins/turn_detector/languages.py — CLOUD_LANGUAGES (0.4) + LOCAL_LANGUAGES (0.3) per-language thresholds.

Test plan

• tests/test_turn_detection_fsm.py — 11 FSM cases incl. WARMING_UP / set_active(False) regression.
• tests/test_turn_detection_cloud_stream.py — 4 cloud-transport invariants (retry reset, FIFO send ordering).
• tests/test_audio_turn_detector_fallback.py — 15 cases: auto-select, explicit-mode errors, transport-error fallback, timeout fallback, persistence, missing-lib graceful, local-failure retry, warning dedupe, multiplicative threshold scaling.
• tests/test_audio_recognition_turn_detection.py — 10 cases: VAD/audio/sentinel forwarding into the stream, prediction-driven EOU + deactivation, speaking-guard race aborts commit.
• make format + make lint + make type-check clean (only pre-existing agent_activity.py InterruptionDetectionError errors remain).

Depending on livekit/python-sdks#676

[hsjun99]

@chenghao-mou Excited to see this! A couple of questions:

1. Will the multimodal EOT model be publicly accessible via model weights or agent-gateway.livekit.cloud, or in some other way?
2. Any rough timeline for when MultiModalTurnDetector gets fully wired up?

[chenghao-mou]

@chenghao-mou Excited to see this! A couple of questions:

1. Will the multimodal EOT model be publicly accessible via model weights or agent-gateway.livekit.cloud, or in some other way?
2. Any rough timeline for when MultiModalTurnDetector gets fully wired up?

Thanks for your patience! We don't have an official decision or timeline yet, but hopefully I can get it ready within a month or two.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 28 additional findings in Devin Review.

[devin-ai-integration]

🟡 WARNING-level log fires on every normal turn when audio EOT + STT are combined

When AudioTurnDetector is used with STT, predict_end_of_turn logs at WARNING level every time it's called after the audio EOT model already committed the turn. The normal flow is: (1) VAD END_OF_SPEECH triggers _run_eou_detection → bounce task → predict_end_of_turn returns cached prediction → turn committed → flush() sets _user_turn_started=False. (2) A late STT final transcript arrives → _run_eou_detection triggers again → bounce task → predict_end_of_turn sees _user_turn_started=False → logs WARNING and short-circuits. This WARNING fires once per committed turn in the most common configuration (audio EOT + Deepgram STT), flooding logs in production with a message that describes normal, expected behavior.

Suggested change

	# min_endpointing_delay path.
	logger.warning(
	"predict_end_of_turn called with no open user turn, short-circuiting",
	)
	return 1.0
	logger.debug(
	"predict_end_of_turn called with no open user turn, short-circuiting",
	)
	return 1.0

---

Was this helpful? React with 👍 or 👎 to provide feedback.