OmniVoice-Luna-vi

OmniVoice-Luna-vi

Single-speaker Vietnamese text-to-speech. A LoRA finetune of k2-fsa/OmniVoice on one Vietnamese voice ("Luna"), with the adapter merged back into the base weights — load it and it speaks in that voice.

No reference clip needed. The base model is zero-shot and wants a voice prompt on every call; here the voice lives in the weights. Dropping the prompt removes a ~300-token prefix that every target token has to attend to, which is worth 1.3 GB of VRAM at serve time (7,134 MB → 5,794 MB measured) on top of being one less thing to ship.

Usage

import soundfile as sf
import torch
from omnivoice import OmniVoice

model = OmniVoice.from_pretrained(
    "thanhhai03/OmniVoice-Luna-vi", device_map="cuda:0", dtype=torch.float16)

wav = model.generate(
    text=["Xin chào, đây là giọng đọc tiếng Việt."],
    language="vi",
    voice_clone_prompt=None,   # giọng nằm trong trọng số — không cần clip tham chiếu
    num_step=16,
    guidance_scale=2.0,
    pad_duration=0.0,
    fade_duration=0.02,
)
sf.write("out.wav", wav[0], 24000)

Output is 24 kHz mono float32.

Choosing num_step

num_step is the number of denoising steps. It is the one knob that matters, and it trades accuracy against latency on the same weights:

Start at 16. Drop to 4 only if you are latency-bound; below 4 quality falls off sharply.

Evaluation

435 held-out Vietnamese sentences, of which 422 after removing 13 that overlap the training set. ASR for CER/WER is faster-whisper large-v3. Speaker similarity (SECS) is Resemblyzer cosine against a 500-clip reference set of the target speaker; UTMOS is SpeechMOS utmos22_strong; FAD is VGGish against the same reference set.

Against the zero-shot base on the same test set and the same metrics: 5.4× lower CER, 3.2× lower WER, speaker similarity up from 0.8476 to 0.9228 (the human reference itself scores 0.9507, so this closes most of the gap), and FAD down from 3.056 to 0.784.

Two caveats worth reading before you quote these numbers:

• UTMOS does not rank what you think it ranks. The human recordings score 2.422 — below every synthetic system measured here. Treat UTMOS as a detector for gross degradation, not as a naturalness ranking between two systems that both work.
• f0 std moves away from the reference, not toward it. The human speaker sits at 2.387; the base model is flatter (2.274) and this finetune is more animated (2.597 at 4 steps, 2.877 at 16). The metric cannot tell "expressive" from "unstable" — listen before drawing a conclusion.

Latency (single request, RTX-class GPU)

Measured through a WebSocket server with FlashInfer + torchao fp8 + torch.compile, at num_step=4:

Finetuning does not make the model faster — latency is identical within noise. What it buys is the 1.3 GB, and that comes specifically from being able to drop the voice prompt: serving this model with a prompt still costs 7,183 MB.

Training

LoRA on the base model, then merged.

audio_embeddings and audio_heads are trained in full rather than through LoRA: the model has 8 parallel codebooks with a head each, and a low-rank update on those two layers is too narrow to move the audio-token distribution to a new speaker.

Per-codebook loss weights are [8, 8, 6, 6, 4, 4, 2, 2] — the first codebooks carry the coarse envelope and timbre and are weighted 4× the last ones, which only carry residual detail.

⚠️ Do not pick checkpoints by eval loss. OmniVoice re-samples its mask positions at every evaluation, so eval loss is stochastic and does not track audio quality. Select on CER/SECS from generated audio instead.

Limitations

• One voice, one language. Vietnamese only. The zero-shot voice-cloning ability of the base model is not preserved — this model speaks in one voice.
• Trained on read speech; expressiveness outside that domain is untested.
• No text frontend. Numbers, dates and acronyms are read as written; normalise upstream if your text contains them.

License and intended use

CC-BY-NC-4.0 — non-commercial. This is inherited from the base model, whose weights are CC-BY-NC because of constraints in its training data (Emilia); it is not a choice made here and it cannot be relaxed downstream.

The bundled audio_tokenizer/ carries its own separate license (Boson Higgs Audio 2 Community License, derived from the Meta Llama 3 license) — see audio_tokenizer/LICENSE.

This model reproduces the voice of a specific speaker. Do not use it to impersonate anyone, to attribute speech to a person without their consent, or to produce audio intended to deceive. Synthetic speech should be disclosed as synthetic wherever a listener might reasonably assume otherwise.

Credits

Base model: k2-fsa/OmniVoice (Qwen3-0.6B backbone, masked-diffusion audio LM, 8 codebooks at 25 Hz). Audio codec: Boson Higgs Audio 2 tokenizer.