funasr_onnx: Fsmn_vad returns the segments of all input files in one list #3830

The funasr_onnx README documents str, np.ndarray, List[str] input for Fsmn_vad, but for a list of files the result holds one segment list for all of them. Fsmn_vad.__call__ (runtime/python/onnxruntime/funasr_onnx/vad_bin.py) starts from segments = [[]] * self.batch_size and adds the segments of each batch to segments[batch_num] for batch_num in range(0, self.batch_size). With the default batch_size=1, each file is its own batch, so the segments of every file end up in segments[0]:

• model([a, b]) returns [a_segments + b_segments] instead of [a_segments, b_segments], and b's times start again from 0 after a's last segment.
• model([silence, b]) returns [b_segments]: b's segments are reported for the silent file, and there is no entry for b.

Paraformer and SenseVoiceSmall in the same package return one result per input.

batch_size > 1 does not work either:

• One file: E2EVadModel returns one segment list per waveform, but the loop reads segments_part[batch_num] for every batch_num below batch_size. With batch_size=2, a file with speech raises IndexError: list index out of range, and a file without speech returns [[], []].
• Two or more files in a batch fail before that (numpy 1.26.4): np.array(waveform) raises ValueError for files of different lengths, and int(feats_len) raises TypeError for files of equal length. E2EVadModel also takes one waveform per call (assert len(self.scores) == 1, "only support batch_size = 1 now").

Repro on main (f122f6c) without downloading a model. Only the ONNX session is replaced, by a stub that scores loud frames as speech; WavFrontend (with the frontend_conf of funasr/models/fsmn_vad_streaming/template.yaml, no CMVN) and E2EVadModel are the real ones:

import numpy as np
import soundfile
from funasr_onnx.utils.frontend import WavFrontend
from funasr_onnx.vad_bin import Fsmn_vad


def fake_infer(inputs):  # loud frames are speech
    sil = np.where(inputs[0][0].mean(-1) > 5.0, 0.05, 0.95)
    return np.stack((sil, 1.0 - sil), axis=-1)[None].astype(np.float32), inputs[1:]


model = Fsmn_vad.__new__(Fsmn_vad)
model.frontend = WavFrontend(
    cmvn_file=None, fs=16000, window="hamming", n_mels=80, frame_length=25,
    frame_shift=10, dither=0.0, lfr_m=5, lfr_n=1,
)
model.batch_size, model.vad_scorer_config, model.max_end_sil = 1, {}, 800
model.encoder_conf = {"fsmn_layers": 1, "proj_dim": 2, "lorder": 2}
model.infer = fake_infer

# a.wav: 3 s, noise at 0.5-1.5 s. b.wav: 4.5 s, noise at 1-2 s and 3-3.5 s.
for name, length, noise in (
    ("a.wav", 48000, [(8000, 24000)]),
    ("b.wav", 72000, [(16000, 32000), (48000, 56000)]),
):
    rng = np.random.default_rng(0)
    audio = np.zeros(length, dtype=np.float32)
    for start, end in noise:
        audio[start:end] = rng.uniform(-0.3, 0.3, end - start)
    soundfile.write(name, audio, 16000)

print(model("a.wav"))             # [[[220, 1660]]]
print(model("b.wav"))             # [[[720, 2160], [2720, 3660]]]
print(model(["a.wav", "b.wav"]))  # [[[220, 1660], [720, 2160], [2720, 3660]]]

model.batch_size = 2
for audio_in in ("a.wav", ["a.wav", "b.wav"], ["a.wav", "a.wav"]):
    try:
        print(model(audio_in))
    except Exception as e:
        print(f"{type(e).__name__}: {e}")

With batch_size=2, the loop prints:

IndexError: list index out of range
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.
TypeError: only length-1 arrays can be converted to Python scalars

Expected: [[[220, 1660]], [[720, 2160], [2720, 3660]]] for the list, and with batch_size=2 the same results as with batch_size=1: [[[220, 1660]]], [[[220, 1660]], [[720, 2160], [2720, 3660]]] and [[[220, 1660]], [[220, 1660]]].

Environment: Windows 11, Python 3.11.17 and 3.12.15, numpy 1.26.4, onnxruntime 1.31.1, kaldi-native-fbank 1.22.5, librosa 0.11.0, soundfile 0.14.0, funasr-onnx 0.4.5 built from runtime/python/onnxruntime.