Skip to content

Fix Chatterbox multilingual diacritic normalization - #468

Open
0xShug0 wants to merge 1 commit into
mainfrom
fix/chatterbox-diacritic-normalization-462
Open

Fix Chatterbox multilingual diacritic normalization#468
0xShug0 wants to merge 1 commit into
mainfrom
fix/chatterbox-diacritic-normalization-462

Conversation

@0xShug0

@0xShug0 0xShug0 commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes #462.

Chatterbox multilingual text preprocessing now uses the framework NFKD decomposition helper instead of the llama tokenizer NFD helper. The previous helper mapped each input codepoint to a single output codepoint, so precomposed diacritics such as Polish accented letters could lose their combining marks before tokenization.

This keeps the fix scoped to Chatterbox multilingual tokenization and does not change the English tokenizer path.

Validation

  • Built audiocpp_cli.
  • Reproduced the issue with Polish NFC input before the fix.
  • Verified that, after the fix, Polish NFC input and the same pre-decomposed input produce byte-identical audio with the same seed.
  • Verified the fixed NFC output with Qwen3 ASR; the transcription preserves Polish diacritics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chatterbox voice-cloning: combining diacritics silently dropped on standard (NFC) input

1 participant