Implement beam search decoding for Whisper#1429
Open
muaz978 wants to merge 1 commit into
Open
Conversation
DecodingOptions already accepts beam_size and patience, and the scaffolding for group decoding (batch tiling for n_group > 1, MaximumLikelihoodRanker, Inference.rearrange_kv_cache) is in place, but requesting a beam size raised NotImplementedError. This adds the missing BeamSearchDecoder, ported from the reference implementation in openai/whisper and adapted to MLX: - Top-k candidate selection runs on the device with mx.argpartition and mx.take_along_axis, so only (beam_size + 1) token ids and log probabilities per beam cross to the host each step instead of a vocabulary sized matrix. - Candidate prefixes are kept on the host between steps instead of pulling the whole growing token matrix off the device every step. - finalize pads candidate sequences with EOT so the result stays rectangular, matching what DecodingTask.run expects before it truncates at the first EOT. rearrange_kv_cache reindexed every cached array, but the cache mixes self attention entries (batch = n_audio * n_group) with cross attention entries (batch = n_audio, broadcast over the group). Indexing the cross attention arrays with beam indices reads out of bounds, which is not checked on the GPU and silently corrupts the decode. It now only reindexes arrays whose batch dimension matches the selection; the cross attention rows are identical across beams anyway. Tested on Apple Silicon with Arabic and English speech: decoding with beam_size=5 is deterministic at temperature 0, works with word_timestamps and patience, and the temperature fallback chain (beam at t == 0, sampling above) behaves like openai/whisper. With whisper-large-v3-turbo the wall clock cost of beam_size=5 was within a few percent of greedy decoding on an M series GPU; smaller models pay relatively more because the per step bookkeeping is a larger share of their forward pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
DecodingOptionsalready acceptsbeam_sizeandpatience, and most of the machinery for group decoding exists (batch tiling forn_group > 1,MaximumLikelihoodRanker,Inference.rearrange_kv_cache), but requesting a beam size raisesNotImplementedError. This PR implements the missingBeamSearchDecoder, ported from the reference implementation in openai/whisper and adapted to MLX.Implementation notes
mx.argpartition+mx.take_along_axis, so only(beam_size + 1)token ids and log probabilities per beam cross to the host each step, instead of a(n_batch, n_vocab)matrix. The beam bookkeeping itself stays in python, like the reference implementation.finalize: finished sequences are padded with EOT so the result matches whatDecodingTask.runexpects before it truncates each row at the first EOT.rearrange_kv_cachereindexed every cached array, but the cache mixes self-attention entries (batch =n_audio * n_group) with cross-attention entries (batch =n_audio, broadcast over the group). Indexing the cross-attention arrays with beam indices reads out of bounds, which MLX does not bounds-check on the GPU, silently corrupting the decode into nondeterministic garbage. The rearrange now only touches arrays whose batch dimension matches the selection; the cross-attention rows are identical across beams anyway. (This also future-proofsbest_ofstyle uses that might want to rearrange.)Testing
On Apple Silicon (M series), with Arabic and English speech:
beam_size=5attemperature=0is deterministic across runs and returns correct transcriptions (previously:NotImplementedError).word_timestamps=True,patience, and the temperature fallback chain (beam_sizeapplies at t == 0,best_ofsampling above, matching openai/whisper).decodestill rejectsbeam_size+best_ofgiven together, andpatiencewithoutbeam_size.whisper-large-v3-turboon a 44 s clip:beam_size=5was within a few percent of greedy decoding. Smaller models pay relatively more (about 2x onwhisper-small) because the per-step bookkeeping is a larger share of their forward pass.Beam search noticeably improves difficult, accented, and dialectal speech compared to greedy decoding (it is what faster-whisper and openai/whisper use by default with
beam_size=5), so this closes one of the last accuracy gaps between mlx_whisper and those implementations.