[tanishqbaweja]

Summary

Fixes #3823

Since 4ad2a7b, CTTransformer.inference returns [{"key": ..., "text": "", "punc_array": None}] for empty or whitespace-only text. CTTransformerStreaming (funasr/models/ct_transformer_streaming/model.py, the class behind the realtime model iic/punc_ct-transformer_zh-cn-common-vad_realtime-vocab272727 in examples/industrial_data_pretraining/ct_transformer_streaming/demo.py) overrides inference without that check, so empty text fails in one of two ways:

• With a fresh cache, "" or " " gives no words. tokenizer.encode([]) returns [], and np.concatenate of the empty int32 cache_sent_id with that empty list produces float64 ids, so the embedding raises the error users reported for the offline model in runtime error for audio file in which there is actually no speech (blank voice) #1386, 无人声音频时指定punc_model报错 #1625 and auto_model 中punc模型入参为空触发的bug #1660:

  RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.DoubleTensor instead (while checking arguments for embedding)
  

• When earlier calls left words in cache["pre_text"], an empty call runs the model on those words again and skips them in the output. If no punctuation is predicted after the last of them, sentence_out is empty and if sentence_out[-1] in self.punc_list: raises IndexError: string index out of range.

AutoModel.generate(input="", cache=cache) passes the empty string straight to inference, and the demo calls generate once per VAD segment, so a segment with empty ASR text (silence or noise) takes one of these paths.

This adds the three-line check from CTTransformer.inference right after assert len(data_in) == 1, before the cache is read or written. Empty or whitespace-only text now returns text: "" and punc_array: None, and words waiting in cache["pre_text"] stay there for the next segment.

Type of change

• Bug fix
• Documentation
• Example or demo
• Runtime or deployment
• Benchmark or evaluation
• Model/training change

Validation

• python -m compileall funasr examples tests
• Docs or links checked
• Runtime/deployment command tested

Windows, Python 3.11.17, torch 2.10.0+cpu, numpy 2.4.6, pytest 9.2.0.

tests/test_ct_transformer_streaming_empty_input.py builds the real CTTransformerStreaming class (embedding, SANMVadEncoder with 2 blocks so the VAD mask runs, decoder) with tiny random weights and a decoder that always predicts _, plus the repo's CharTokenizer. Nothing is downloaded.

• test_empty_text_returns_empty_result: "" and " " with a fresh cache return [{"key": "utt", "text": "", "punc_array": None}].
• test_empty_segment_keeps_pending_words: calls AutoModel.generate the way the demo does, with the segments 跨境河流, "" and 是 and one cache. The texts are 跨境河流, "" and 是, and cache["pre_text"] is still ["跨", "境", "河", "流"] after the empty segment.

Commands:

• python -m pytest -q tests/test_ct_transformer_streaming_empty_input.py: on main (8710e1c), both subtests of the first test fail with the RuntimeError above, and the second test fails at the empty segment with IndexError: string index out of range (model.py:207). With this change: 2 passed, 2 subtests passed. I got the same results with numpy 1.26.4, the older of the two versions in the workflow matrix.
• python -m pytest -q tests/test_punc_model_none.py tests/test_paraformer_empty_token_batch.py tests/test_ct_transformer_docs.py: 29 passed, 14 subtests passed, both on main and with this change (with this change also under numpy 1.26.4).
• I added the test and the model file to both path filters of test-numpy-compatibility.yml, and the test to its pytest command next to the other CPU-only punctuation and Paraformer regressions. CI runs that list on both NumPy versions. Here on Windows the rest of the list does not run cleanly (no pytorch_wpe or ffmpeg, and some subprocess imports hit their 120 s timeout); none of those failures involve this change.
• black --line-length=100 (24.4.0, the pre-commit pin) leaves the new test unchanged and does not touch the added lines in model.py. git diff --check is clean.

User impact

Users of the realtime punctuation model through AutoModel, as in the streaming punctuation demo, get an empty result for a segment whose ASR text is empty or whitespace-only instead of a RuntimeError or IndexError, and the next segment is punctuated with the same pending context. Non-empty text takes the unchanged code path.

Notes for reviewers

• The check is the one CTTransformer.inference uses, so both punctuation models return the same structure for empty input. AutoModel.inference_with_vad already handles punc_array: None from the offline model.
• It returns before cache["pre_text"] is read or initialised, so an empty call leaves the cache as it was. On main, such a call either raised or recomputed pre_text from the same words.
• runtime/python/websocket/funasr_wss_server.py already skips empty text before calling this model. This change is for direct AutoModel callers such as the demo.

[LauraGPT]

Reviewed the three changed files at 8dfa8f566ba3bd8f1591607fcdfacbe73064363c. The early empty/whitespace return matches the non-streaming punctuation convention, avoids invalid embedding indices and empty-string indexing, and preserves pending pre_text without flushing or reprocessing cached tokens. Nonempty inference is unchanged. The workflow adds the regression to both NumPy jobs and retains read-only permissions.

Validation against current main and its exact potential merge tree:

• Baseline reproduced both submitted regression failures; all 31 selected CPU tests pass with the fix.
• Real L20Z FP32 execution: baseline fresh-empty RuntimeError and cached-empty IndexError reproduced. Candidate empty/whitespace and normal-empty-normal cases pass; cache content is unchanged across the empty chunk. Actual profiler records 381 CUDA kernel events for the candidate. This uses deterministic tiny randomly initialized CTTransformerStreaming weights (saved with hashes), not pretrained weights or an ASR-quality/full-suite/Windows/NPU claim.
• Exact-head NumPy 1.26.4 and 2.4.0 CI passed; downloaded artifact source commit, JUnit results, dependency records and pip-check were verified. The originally pending fork workflow was explicitly reviewed and approved, not bypassed or retried.

Source fix only: already-published 1.4.20 does not acquire this change by merging; package delivery needs a new release.