omnivoice-ipa-onnx

OmniVoice ONNX + IPA fine-tune diff

An ONNX conversion of k2-fsa/OmniVoice (a non-autoregressive diffusion-LM TTS), plus a small IPA fine-tune diff that teaches the model to accept IPA phoneme strings (from vernacula-phonemizer) as text input — so the phonemizer, not the model, owns the linguistic G2P: dictionary + neural G2P, stress, pitch accent, and text normalization (numbers, %, currency, units spoken in-language) all happen before the model sees a token.

Try it

vernacula.netlify.app runs this model entirely in your browser — 193 languages, no upload, no server inference. It downloads the quantized build below (472 MB, cached after the first load) and does the phonemization locally too.

The same pipeline also ships as a desktop app and a command-line renderer in vernacula. The desktop version is runs at 2.5x realtime on a 3090.

Files

The voice library

Every language gets a native voice where a usable clip exists, else a donor from a related language; default marks the one a consumer should start from, and source names the exact clip so a poor exemplar can be traced and replaced. The web demo and the desktop app read the same two files, so all three renderers share one voice per language. How the clips were chosen, and what still lacks one, is in docs/voice_sourcing.md. Native speakers that can help align the voices audio and reference IPA would be appreciated.

The transformer is the base (un-fine-tuned) graph; the encoder/decoder are the codec, unchanged by the fine-tune. The IPA fine-tune is distributed as a 31 MB diff rather than a second 2.45 GB merged graph.

⚠ Verify the base before applying the diff

The diff holds a LoRA plus absolute replacement embedding rows, so it is only meaningful against the exact base it was extracted from. Applied to any other weights it produces a plausible-looking model that is quietly wrong — the perturbation is the same order as the fine-tune itself — and nothing in the diff can detect this: the per-row delta distributions for a right and a wrong base overlap. Check the fingerprint:

sha256sum omnivoice_transformer.onnx.data
# 2ea0980e184bbf8457048fbb3ed2a01f8f8c3816a8ee9fbff3ce0886c1aeeb4a

Applying the diff

The diff carries, per Linear (q/k/v/o/gate/up/down x 28 layers = 196 modules), the LoRA factors A (r=16, in) and B (out, r) with lora_scale = alpha/r = 2.0 in the model metadata, plus the changed embed_tokens rows as embed_rows (fp16) and embed_idx (int32).

Recommended — rewrite the graph, keeping the LoRA factored. Leave the base weights alone and add nodes:

MatMul(x, W) -> Y      becomes
MatMul(x, W) -> Y_base ;  MatMul(x, A^T) -> t ;  MatMul(t, scale*B^T) -> d ;  Add(Y_base, d) -> Y

with a sparse additive correction for the embedding (a compact (changed+1, hidden) delta table with a zero row 0 and a (vocab,) int32 map, as two Gathers and an Add). Insert each chain at the original node's position, not at the end of the graph, or the Add lands after its own consumers. Because no weight is replaced, the runtime loads the base from the untouched .onnx.data and this works on every execution provider. Measured against a full PyTorch merge: argmax agreement 100.000%, max|dlogit| 1.1e-4 (CPU) / 6.1e-5 (CUDA).

Alternative — fold into the weights (W += ((B@A)*scale)^T, overwrite the changed embed rows). Also exact, but note that ⚠ if you feed the folded weights back through SessionOptions.AddInitializer, ONNX Runtime will silently ignore them on any non-CPU provider — a user-supplied initializer must be a CPU tensor, so when the session is planned on CUDA every one is rejected ("Cannot use user supplied initializer ... planned memory location device is different") and the base graph is served instead. The output is then bit-identical to using no diff at all, which for IPA input is noise. If you fold, write the merged model out and load it from disk.

Reference implementations: apply_diff.py (Python, fold) and OmniVoiceDiff (C#, graph rewrite) in vernacula.

Using it

Text -> IPA -> audio, with the phonemizer owning all G2P. The model is conditioned language-agnostically (<|lang_start|>None<|lang_end|>); the IPA stream carries everything.

⚠ Short input needs a reference voice. The fine-tune corpus is FLEURS read prose — median 12 s, only 0.21% under 3 s — so under ~5 s the model is out of distribution and can emit noise rather than degrade gracefully. Supplying a reference clip for voice cloning fixes it, as does lengthening the text. Generation is greedy (temperature 0) and therefore deterministic: re-running reproduces a failure byte-for-byte.

Attribution & license

⚠ This repo is licensed PER FILE, and the HuggingFace tag names only the strictest part. The files come from two different upstreams with different terms, and treating them alike would either over-restrict the codec or under-restrict the transformer.