VibeVoice-Embed

VibeVoice-Embed

The acoustic encoder of microsoft/VibeVoice-1.5B, fully separated into a standalone voice-embedding model: 24 kHz audio in, one 64-dimensional speaker vector out.

• 344M parameters (the encoder alone — no language model, no diffusion head, no decoder)
• 64-d embeddings, mean-pooled over causal-VAE latent frames at 7.5 Hz
• Deterministic: the encoder's distribution mean is used, never a sample — the same clip always produces the same vector
• Loads in seconds, runs at >100× realtime on a modest GPU slice (~2.7 GB at fp32)

How it was made

Upstream VibeVoice pairs this encoder with a decoder as a reconstruction VAE feeding a 1.5B LM. The weights here are the 276 model.acoustic_tokenizer.encoder.* tensors, copied bit-for-bit (bfloat16, exactly as stored) out of the source checkpoint and renamed to encoder.*.

One encoder covers the whole family

The 1.5B and 7B (Large) VibeVoice checkpoints ship the same acoustic encoder. Measured against a 7B export (fp16 storage): 111/276 tensors bit-identical, the rest differ only by bf16-vs-fp16 storage rounding (≤5×10⁻⁶ of tensor max, concentrated in elements small enough to hit fp16's subnormal range); embeddings of identical clips agree to ~2×10⁻⁵ relative and speaker-separation metrics are identical (d′ +3.61, AUC 0.997 for both). So this export is the voice embedder for either model size. The modeling code is likewise mechanically extracted from upstream modular_vibevoice_tokenizer.py with the decoder classes removed — only VibeVoiceEmbedModel (construction, pooling, padding handling) is new. Provenance, including the exact source revision, is recorded machine-readably under vibevoice_embed_metadata in config.json.

Does a reconstruction VAE make a speaker embedder?

Measured on a controlled set — 4 speakers × 4 utterances, all different text, each clip cloned from one fixed reference so identity is pinned (24 same-speaker pairs, 96 different-speaker pairs):

So mean-pooled VibeVoice latents separate speakers on par with a purpose-trained speaker-verification embedder on this set, despite coming from an encoder trained for reconstruction.

Honest caveats:

• Every test clip is synthetic output of one TTS engine, so recording channel is matched in a way that flatters the encoder. Not yet validated on real, diverse-channel diarized audio.
• No metric achieved a positive worst-case margin — same/different distributions overlap slightly, so this supports ranking and clustering, not a single global accept/reject threshold.
• Pooling is mean-only for a measured reason: std-pooling carries no speaker information here (std vectors of different clips have cosine 0.97–0.98), and concatenating it degrades the embedding (its norm is 3–4× the mean's).

Distance metric

Vectors are returned unnormalised (raw preserves information; normalising is one line). But the per-clip magnitude is noise — scored alone it is at chance (AUC 0.562) — so ranking by L2 on raw vectors is measurably worse than cosine. L2-normalise before indexing, or configure your vector store for cosine. On unit vectors, cosine, Euclidean and dot product produce identical rankings.

Usage

import torch, torchaudio
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "lemuriandezapada/VibeVoice-Embed",
    torch_dtype=torch.float32,   # recommended: pooling is precision-sensitive
    trust_remote_code=True,
).eval().to("cuda")

wav, sr = torchaudio.load("clip.wav")
wav = wav.mean(0, keepdim=True)  # mono
if sr != model.config.sampling_rate:
    wav = torchaudio.functional.resample(wav, sr, model.config.sampling_rate)

with torch.no_grad():
    out = model(wav.to("cuda"))

embedding = out.pooler_output[0]            # (64,), float32, unnormalised
unit = torch.nn.functional.normalize(embedding, dim=0)  # for cosine indexing
frames = out.last_hidden_state              # (1, frames, 64) latents at 7.5 Hz

Batching variable-length clips

The encoder is causal, so right-padding cannot corrupt a clip's real frames — but the frames emitted for the padding must not be averaged in. Pass padding_mask and the model pools mask-aware:

batch = torch.nn.utils.rnn.pad_sequence([wav_a[0], wav_b[0]], batch_first=True)
mask = torch.zeros_like(batch, dtype=torch.bool)
mask[0, : wav_a.shape[-1]] = True
mask[1, : wav_b.shape[-1]] = True
out = model(batch.to("cuda"), padding_mask=mask.to("cuda"))

A clip embedded in a batch equals the same clip embedded alone.

Serving

serving/openai_embeddings_server.py is a self-contained OpenAI-compatible /v1/embeddings server (FastAPI): audio goes in the input field as base64 or a data: URI, optionally "normalize": true for unit vectors.

python serving/openai_embeddings_server.py --model lemuriandezapada/VibeVoice-Embed --port 8080

There is no vLLM plugin, deliberately: this is a pure convolutional encoder — no attention, no KV cache, no tokens — so vLLM's engine has nothing to schedule for it, and at >100× realtime the serving bottleneck is transport, not compute.

Repository layout

• model.safetensors — encoder weights, bfloat16, keys encoder.*
• config.json — encoder hyperparameters (verbatim from the source checkpoint), auto_map for trust_remote_code, and vibevoice_embed_metadata provenance
• configuration_vibevoice_embed.py / modeling_vibevoice_embed.py — standalone modeling code, no dependency on the VibeVoice package
• serving/openai_embeddings_server.py — reference server

Upstream references

• Code: https://github.com/microsoft/VibeVoice (community fork: https://github.com/vibevoice-community/VibeVoice)
• Base model: https://huggingface.co/microsoft/VibeVoice-1.5B
• Related export from this workspace: VibeVoice-ASR-awq-int4

Notes

• This is a derivative export of the upstream checkpoint's acoustic encoder, not a fine-tune; no weights were modified.
• Base model licensing and usage terms follow the upstream VibeVoice release (MIT).