Repository navigation
Conversation
9ab4a6c to
3c1a82c
Compare
|
@IgorSwat Please rebase first. Edit: is the target branch correct? |
36b3a66 to
457beb2
Compare
d01eb76 to
34aa9aa
Compare
barhanc
left a comment
There was a problem hiding this comment.
I will test the example app first thing tomorrow.
| // one backend nearly every model exports to. Core ML sits above MLX only to | ||
| // make the order deterministic; every group publishing both pins its winner | ||
| // explicitly. |
There was a problem hiding this comment.
If it's not a big problem, we should keep an xnnpack delegate as a fallback (not every app dev might want to include any other backend).
| // less. All decode with the fp16 codec (the int8 Core ML one holds ~600 MB | ||
| // more on the GPU); INT4 encodes with int4 (40 MB smaller), the rest with | ||
| // int8. No XNNPACK export: the backbone needs MLX or Vulkan. | ||
| REVERB: variants({ |
There was a problem hiding this comment.
What's the rationale for keeping both int4 and int8?
| 'standard_male', | ||
| 'angry_male', | ||
| 'sensual_female', |
There was a problem hiding this comment.
Please make the number of male and female voices equal, add standard_female and maybe consider renaming these (especially the sensual_female).
| /** | ||
| * Sentence-ending punctuation. | ||
| * @category Speech / Constants | ||
| */ | ||
| export const EOS_PATTERN = /[.?!;…|।॥¿¡。.!?;]/; | ||
|
|
||
| /** | ||
| * Punctuation pausing within a sentence: commas, colons, dashes. | ||
| * @category Speech / Constants | ||
| */ | ||
| export const PAUSE_PATTERN = /[,:\u2014\u00AB\u00BB、,:〜~]/; |
There was a problem hiding this comment.
Maybe you can copy these constants over to Reverb files as not to expose them in public API.
| * @returns The prepared text. | ||
| */ | ||
| export function prepareText(text: string, language: ReverbLanguage): string { | ||
| const verbalizer = (verbalizers[language] ??= new Verbalizer(VERBALIS_LANGUAGES[language]())); |
There was a problem hiding this comment.
Does this caching actually bring any performance benefits, because a simpler code is preferable over minuscule perf improvements. Side question out of curiosity: why is Verbalizer a class, what state does it hold?
| /** Max seconds of speech per chunk. Defaults to 20. */ | ||
| readonly maxChunkDuration?: number; | ||
| /** Post-processing, or `false` for the raw decoder output. */ | ||
| readonly postprocess?: ReverbPostprocessOptions | false; |
There was a problem hiding this comment.
The | false union here looks weird, wouldn't checking if postprocess field is undefined be sufficient?
| const frameDim = (paramSide: 'input' | 'output', tensorIdx: number, dimIdx = 1): DimRef => ({ | ||
| paramSide, | ||
| tensorIdx, | ||
| dimIdx, | ||
| }); | ||
| const cache = () => f16(2, CACHE_LAYERS, CACHE_HEADS, Dyn('P'), CACHE_HEAD_DIM); |
There was a problem hiding this comment.
I think writing the spec directly without these helpers makes it clearer, even if it needs more lines---for methods with a lot of arguments they should be listed one under another with comments on the side explaining each argument, and for the constraints a simple comment describing the constraint. That might be personal preference though.
| runtime?: WorkletRuntime | ||
| ): Promise<ReverbTextToSpeech<K>> { | ||
| const scope = createResourceScope(); | ||
| let disposed = false; |
There was a problem hiding this comment.
This flag seems redundant, native code checks for disposal and throws accordingly, so no crash is possible in that case.
| const auxTensors: Tensor[] = []; | ||
| const alloc = (...args: Parameters<typeof tensor>): Tensor => { | ||
| const t = tensor(...args); | ||
| auxTensors.push(t); | ||
| return t; | ||
| }; |
There was a problem hiding this comment.
You could use local scope here instead of this adhoc implementation.
| samplePass(tokens, frames, pass, step, schedule, remaskRatio, random); | ||
| } | ||
|
|
||
| // 3. Polish codebooks 2-7 |
There was a problem hiding this comment.
Maybe it's a stupid question, but does "Polish" here mean the language or "polish" as in "improve"? If the latter, perhaps consider renaming this (and maybe the method name itself if it's not too much hassle) to "improve" / "enhance" / "refine".
Description
This PR adds a full pipeline (task extension, model registry, demo app) for Reverb-TTS model.
Requires v0.11.2 binaries from #1637 .
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Run the following tests for the Reverb pipeline:
or play with demo app in
apps/speechfor a manual testing.Screenshots
Related issues
Checklist
Additional notes