OmniVoice-Onnx

OmniVoice → ONNX

ONNX conversion of Prince-1/OmniVoice — a zero-shot text-to-speech model supporting 600+ languages — via Olive and onnxruntime-genai ModelBuilder.

OmniVoice is not a vision-language model. It is a TTS model: a Qwen3-0.6B backbone driving an 8-codebook audio codec (Higgs Audio V2 Tokenizer) through a 32-step non-autoregressive unmasking loop. All conversion configuration is defined in Python (optimize.py) — there are no external Olive JSON files.

Architecture

Text ─▶ [audio_embeddings_encoder]  (fuses text + audio-codec token embeds; +ref codes for cloning)
              │ inputs_embeds (B,S,1024)
              ▼
        [llm_decoder]               Qwen3-0.6B, 28 layers  (exclude_embeds + exclude_lm_head)
              │ hidden_states (B,S,1024)
              ▼
        [audio_heads_decoder]       Linear 1024 → 8×1025
              │ logits (B,8,S,1025)
              ▼
        32-step iterative unmasking → audio_codes (8,T)
              │
              ▼
        [higgs_decoder]             Higgs Audio V2 Tokenizer → waveform @ 24 kHz

Key config: hidden_size=1024, num_codebooks=8, audio_vocab_size=1025 (1024 codes + mask id 1024), codebook weights [8,8,6,6,4,4,2,2], 32 decoding steps, 24 kHz output.

Why the LLM bypasses Olive

onnxruntime-genai's valid precision × execution-provider combos are FP32/INT4 on CPU and FP16/INT4 on CUDA — there is no FP16 LLM on CPU. Olive's ModelBuilder pass therefore can't emit a CPU fp16 LLM, so optimize.py calls genai create_model directly and builds the LLM int4 on CPU (fp16 on GPU). The audio sub-models still go through Olive. The genai LLM declares only inputs_embeds + attention_mask (+ KV cache) and computes positions internally — no position_ids input.

Precision profiles

Higgs is exported fp16 (default) or fp32 via --higgs-precision — never int4 (too lossy for the DAC codec). It is precision-shared: an int4 backbone still uses the fp16/fp32 audio_tokenizer/.

Built sizes (verified)

Prerequisites

pip install -r requirements.txt          # olive-ai, onnxruntime, onnxruntime-genai, transformers 5.x
pip install omnivoice                     # registers the base OmniVoice architecture for loading

omnivoice is only needed at build time (to load the source checkpoint); inference needs only onnxruntime + numpy + soundfile. In this repo's env the build is run isolated as uv run --with omnivoice python optimize.py … to keep the base environment clean.

Build

# CPU int4 (all sub-models int4)
python optimize.py --device cpu      --output onnx/int4 --model ./model/

# CPU fp16 (audio fp16 + LLM int4)
python optimize.py --device cpu_fp16 --output onnx      --model ./model/

# CUDA fp16 (audio fp16 + LLM fp16)   — needs an NVIDIA GPU + onnxruntime-gpu
python optimize.py --device gpu      --output onnx/cuda --model ./model/

# Higgs tokenizer only (fp16 or fp32) → <output>/audio_tokenizer
python optimize.py --higgs-only --higgs-precision fp16 --output onnx --model ./model/

# Backbone + Higgs together
python optimize.py --device cpu_fp16 --include-higgs --output onnx --model ./model/

optimize.py (1) saves OmniVoice's internal Qwen3 as a standalone Qwen3ForCausalLM dir, (2) builds the two audio sub-models via inline Olive configs and the LLM via genai create_model, (3) copies the tokenizer + chat_template.jinja, and (4) writes omnivoice_manifest.json. All model_config.json paths are relativized to bare basenames so the folder is portable.

Inference (CPU)

# fp16 backbone
python inference.py --model_dir onnx      --higgs_dir onnx/audio_tokenizer \
    --text "Hello from OmniVoice." --output out.wav

# int4 backbone (shares the same Higgs tokenizer — int4 dir has no audio_tokenizer/ of its own)
python inference.py --model_dir onnx/int4 --higgs_dir onnx/audio_tokenizer \
    --text "Hello from OmniVoice." --output out_int4.wav

--num_audio_tokens sets output length in frames (≈ frames × 0.04 s); --num_steps the unmasking steps (the loop fills ⌈remaining/remaining_steps⌉ frames per step so every frame is decoded). Voice cloning: add --ref_audio ref.wav --ref_text "…" (uses the Higgs encoders).

Standalone codec round-trip:

python higgs_inference.py --models-dir onnx/audio_tokenizer --input in.wav --output rt.wav

Eval

# RTF benchmark (onnxruntime only)
python eval.py --mode rtf   --model_dir onnx      --higgs_dir onnx/audio_tokenizer
python eval.py --mode rtf   --model_dir onnx/int4 --higgs_dir onnx/audio_tokenizer

# Numerical equivalence vs PyTorch (needs `omnivoice` + the source model)
python eval.py --mode equiv --model_dir onnx      --higgs_dir onnx/audio_tokenizer

Verified (CPU, this repo)

int4 is slower than fp16 on CPU here (int4 weight dequant overhead without an accelerated kernel); both are comfortably real-time.

Notes

• No external JSON — every Olive config is built in Python in optimize.py; the old cpu_and_mobile/, cpu_fp16/, cuda/, higgs/ *.json files are obsolete.
• Tokenizer is the standard Qwen2 fast tokenizer (tokenizer.json) — loaded from the local dir, no trust_remote_code / omnivoice needed at inference.
• KV cache is passed empty each step ((B, kv_heads, 0, head_dim)) — full-sequence forward, no cache reuse.
• Higgs uses the transformers-native higgs_audio_v2_tokenizer (transformers ≥ 5.4) — no boson-multimodal dependency.