Repository navigation
sentence_info text drifts away from its timestamps/speakers: Chinese VAD chunks are joined without a space before timestamp_sentence splits on spaces #3801
Description
Summary
With vad_model + punc_model (+ spk_model), the text of each sentence_info entry is shifted relative to its start/end/timestamp/spk. The shift grows by one unit at every VAD chunk boundary where two Chinese chunks meet, so on long audio the sentence text ends up far ahead of the time range and speaker it is attached to. The top-level text (punctuated) is correct; only sentence_info[*].text is wrong.
Version
- funasr 1.4.16 (same code in 1.4.17 and current main)
AutoModel(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", spk_model="cam++"), Python 3.12, macOS, CPU
Minimal reproduction (no audio needed)
from funasr.auto.auto_model import _join_vad_texts
from funasr.utils.timestamp_tools import timestamp_sentence
chunks = ["今 天 天 气 很 好", "我 们 去 公 园 吧"] # two VAD chunks
punc_input_text = _join_vad_texts(chunks)
print(repr(punc_input_text)) # '今 天 天 气 很 好我 们 去 公 园 吧' ("好我" fused)
timestamps = [[i * 200, i * 200 + 200] for i in range(12)]
punc = [1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 3] # "，" after 好, "。" after 吧
print(len(punc_input_text.split()), "text units vs", len(timestamps), "timestamps")
for s in timestamp_sentence(punc, timestamps, punc_input_text):
print(s["text"], s["start"], s["end"])Output:
11 text units vs 12 timestamps
今天天气很好我， 0 1200
们去公园吧。 1200 2400
Expected 今天天气很好， / 我们去公园吧。.
Real-world impact
On a 19.5-minute Mandarin recording: 5299 timestamps and 5299 punctuation ids, but punc_input_text.split() yields only 5082 units (217 fused boundaries). Sentence text drifts up to ~217 characters (~50 s of speech) ahead of its timestamps, so:
- sentences are cut 1–N characters off (e.g.
然，/后用于…) - since speakers are distributed by time, text is attributed to the wrong speaker, often switching speaker mid-sentence
Cause
In AutoModel.inference_with_vad, punc_input_text = _join_vad_texts(...) intentionally joins Chinese chunks without a space (fine for the punctuation model, whose tokenizer splits CJK into characters, so punc_array length still matches). But the same string is then used as timestamp_text, and timestamp_sentence pairs units with timestamps via text_postprocessed.split(). The fused boundary units shift every following unit. The existing check punc_length != len(sentence_timestamps) doesn't catch it because the number of text units is never compared.
Suggested fix
Use a space-joined version of the chunk texts for timestamp_text (or have timestamp_sentence consume one unit per timestamp), and/or assert that the number of text units equals the number of timestamps.
Workaround we use for now: call generate(..., return_raw_text=True) and rebuild each sentence's text by taking len(sentence["timestamp"]) units from result["raw_text"].split() in order, keeping the trailing punctuation from sentence["text"]. With that, sentence boundaries and speaker turns line up correctly.
Activity
carey-bk commentedon Oct 29, 2026
I would like to investigate and work on this. I plan to first reproduce the VAD-boundary text/timestamp mismatch using the real joining and sentence-timestamp functions offline, then keep timestamp alignment units separate from punctuation input if that is confirmed as the narrow fix. I will add CPU regressions for multiple Chinese chunk boundaries and mixed-language controls, and check that sentence timing and speaker association are preserved. I have not reproduced it locally yet; offline tests alone would not establish full audio/model correctness. Please let me know if someone is already working on this or if you prefer another scope.
LauraGPT commentedon Oct 31, 2026
The VAD chunk-boundary drift is now addressed on main in #3804 (8b7ab8a).
The timestamp-aligned token stream now retains the spaces between raw ASR chunks, separately from the compact Chinese text supplied to punctuation. The reported example produces
今天天气很好，and我们去公园吧。with the original per-token timestamps. Punctuation input/top-level text are unchanged.Validation includes 207 focused CPU tests plus 171 related KWS/VAD/speaker/MOSS tests, and the new exact-head hosted NumPy 1.26.4/2.4.0, KWS and MOSS checks. The two downloaded NumPy test artifacts contain the new regressions and no failures/errors/skips. Real L20Z FP32/BF16 CIF timestamps also passed 8 downstream alignment/speaker/raw-text cases; the model responses for those cases are synthetic fixtures, not pretrained ASR or a test of your original recording. The full repository suite was not run.
This change is not in the already-published PyPI 1.4.17 package. I am leaving this issue open for confirmation on the original long recording once using the updated main or the next release. Thanks for the precise reproduction and the investigation notes.
LauraGPT commentedon Oct 31, 2026
The #3804 fix is now included in the released FunASR 1.4.18 and PyPI package. There is no need to build main to get it:
python -m pip install -U "funasr==1.4.18"This supersedes the earlier note that the fix was only on main. The official PyPI wheel/sdist were downloaded and matched to the accepted GitHub payloads by SHA-256 and size. The previously reported CPU and synthetic L20Z alignment tests remain their stated scope; the original 19.5-minute recording has not been evaluated here. Keeping this issue open for original-recording confirmation rather than treating package availability as that confirmation.