VibeVoice-AcousticTokenizer-Onnx

VibeVoice Acoustic Tokenizer

VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text. It addresses significant challenges in traditional Text-to-Speech (TTS) systems, particularly in scalability, speaker consistency, and natural turn-taking.

A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details.

The speech tokenizer is a key component for both VibeVoice TTS and ASR.

➡️ Technical Report: VibeVoice Technical Report

➡️ Project Page: microsoft/VibeVoice

<p align="left"> <img src="figs/tokenizer_comparison.png" alt="Tokenizer Comparison" height="250px"> </p>

Models

VibeVoice Acoustic Tokenizer — ONNX

ONNX conversion of microsoft/VibeVoice-AcousticTokenizer — the continuous acoustic speech tokenizer (VAE codec) shared by VibeVoice TTS and ASR. It operates at an ultra-low 7.5 Hz frame rate (hop = 3200 samples @ 24 kHz), encoding waveform to a 64-dim latent per frame and decoding it back.

Exported with Olive (dynamo → ONNX). No PyTorch / transformers needed at runtime — only onnxruntime + numpy (+ soundfile/librosa for audio I/O).

Files

This tokenizer is shared across the VibeVoice family — the same acoustic codec feeds the TTS and ASR models — so inference_tokenizer.py is the general encode/decode driver, usable standalone or as the codec front/back-end of any VibeVoice ONNX pipeline.

Two precisions: fp32/ (1.3 GB, reference) and fp16/ (≈0.7 GB). int4 is not provided — the tokenizer is a conv VAE with no MatMul weights to quantize (int4 would be a no-op = fp32).

Encoder length is baked at 24000 samples. The PyTorch model accepts arbitrary length; this ONNX encoder processes exactly 1 s per call, so longer audio is encoded in 1 s chunks (zero-padded to a whole number of chunks) and the latents concatenated. The decoder is dynamic — it decodes any number of frames in one pass. acoustic_inference.py does the chunking for you.

Setup

pip install onnxruntime numpy soundfile librosa
# GPU: pip install onnxruntime-gpu   (and pass --cuda)

Encoding and decoding (ONNX)

The ONNX equivalent of the model card's model.encode(...) / model.decode(...):

import numpy as np, onnxruntime as ort, soundfile as sf, librosa

SR, CHUNK = 24_000, 24_000                       # 24 kHz; encoder input is fixed at 1 s

def sess(p):
    so = ort.SessionOptions(); so.log_severity_level = 3
    return ort.InferenceSession(p, so, providers=["CPUExecutionProvider"])

enc, dec = sess("fp32/acoustic_encoder.onnx"), sess("fp32/acoustic_decoder.onnx")
npdt = np.float16 if "float16" in enc.get_inputs()[0].type else np.float32

# load + resample to 24 kHz, mono
wav, sr = sf.read("en-Alice_woman.wav", dtype="float32", always_2d=False)
if wav.ndim > 1: wav = wav.mean(1)
if sr != SR: wav = librosa.resample(wav, orig_sr=sr, target_sr=SR)

# ENCODE — 1 s chunks (encoder is fixed at 24000 samples) -> concat latents
n = int(np.ceil(len(wav) / CHUNK))
wav = np.pad(wav, (0, n * CHUNK - len(wav)))
latents = np.concatenate(
    [enc.run(["latents"], {"audio": wav[i*CHUNK:(i+1)*CHUNK][None, None, :].astype(npdt)})[0]
     for i in range(n)], axis=1)                 # [1, 7*n, 64]
print("Latent shape:", latents.shape)

# DECODE — dynamic, one pass -> 3200 samples per frame
audio = dec.run(["audio"], {"latents": latents.astype(npdt)})[0].ravel()
sf.write("reconstructed.wav", audio, SR, subtype="PCM_16")
print("Reconstructed:", audio.shape)

Or just use the driver (prints the encoder → latents → decoder flow):

python inference_tokenizer.py --models-dir fp32 --input en-Alice_woman.wav --output reconstructed.wav
python inference_tokenizer.py --models-dir fp32 --input in.wav  --latents-out codes.npy --encode-only
python inference_tokenizer.py --models-dir fp32 --latents-in codes.npy --output out.wav --decode-only
python inference_tokenizer.py --models-dir fp16 --input in.wav --output out.wav        # fp16

Example flow output:

[encode] 24000 samples (1.00s) @ 24000 Hz  ->  1 x 24000 chunk(s), fp32
  encoder: audio[1,1,24000]  ->  latents[1, 7, 64]   (chunk 1/1)
[decode] decoder: latents[1,7,64]  ->  audio[1,1,22400]  (0.93s, 3200/frame)
saved -> reconstructed.wav   round-trip corr=+0.999

Parity vs PyTorch (verified)

Component parity (ONNX vs VibeVoiceAcousticTokenizerModel) and codec round-trip on a synthetic tone:

¹ the decoder's cosine is measured on near-silent output (random latents ≈ silence), where max|Δ| ≈ 1e-10 — numerically identical, cosine is just noise-dominated (standard neural-codec eval caveat).

Notes

• fp16 was exported with an op_block_list (ConstantOfShape/ConvTranspose/Resize/Range) kept in fp32 — those shape/resample ops produce invalid graphs under fp16 conversion and gain nothing.
• Streaming (use_cache / padding_cache in the PyTorch API) is not exported here — these are fixed-context encode/decode graphs. Streaming would need the cached-state variants exported separately.
• Multi-chunk round-trip: a 1 s window reconstructs at corr ~0.999, but the fixed encoder emits 22400 samples per 24000-sample chunk (7 frames × 3200), so naively encoding >1 s in chunks and decoding the concatenated latents accumulates a small per-chunk time offset that lowers waveform corr. For downstream use the latents are the product (fed to / produced by the LLM); exact full-length reconstruction of long audio wants the streaming (padding_cache) path.
• The upstream AutoFeatureExtractor just formats a mono 24 kHz waveform (pad_to_multiple_of=3200); the ONNX encoder takes the raw [1,1,24000] tensor directly, so no feature-extractor is required.