VibeVoice-Realtime-0.5B-mlx

VibeVoice-Realtime-0.5B for mlx-audio-swift

A repackaging of microsoft/VibeVoice-Realtime-0.5B so it loads in one step from mlx-audio-swift on Apple silicon.

The weights are unmodified. model.safetensors is byte-for-byte the upstream file. What this repo adds is the two things upstream does not ship in a loadable form:

1. voices/ — the speaker prompts, converted from the .pt pickles in the upstream GitHub repo (demo/voices/streaming_model/) to safetensors. Tensor values are unchanged; only the container and key names differ.
2. The Qwen2.5-0.5B tokenizer files, which the model needs but the upstream model repo does not include (its preprocessor_config.json points at Qwen/Qwen2.5-0.5B).

Usage

let model = try await TTS.loadModel(modelRepo: "mnmly/VibeVoice-Realtime-0.5B-mlx")
for try await event in model.generateStream(text: "Hello there.", voice: "en-Carter_man") {
    if case .audio(let chunk) = event { player.enqueue(chunk) }
}

Voices: en-Carter_man, en-Davis_man, en-Emma_woman, en-Frank_man, en-Grace_woman, en-Mike_man, in-Samuel_man, plus two each for de, fr, it, jp, kr, nl, pl, pt, sp.

Voice prompt format

VibeVoice-Realtime does not support voice cloning, by design: the checkpoint contains no acoustic encoder, so there is no path from a recording to a new speaker. A voice is instead prefilled conversation state — a last_hidden_state and a KV cache per branch:

<branch>.last_hidden_state        # [1, prompt_len, 896]
<branch>.key.<layer>              # [1, 2, prompt_len, 64]
<branch>.value.<layer>

for lm (4 layers), tts_lm (20), neg_lm (4) and neg_tts_lm (20). neg_lm is carried for fidelity with upstream but never read during generation.

License and provenance

MIT, inherited from upstream — see LICENSE (© 2025 Microsoft). The bundled Qwen2.5-0.5B tokenizer files are Apache-2.0, from Qwen/Qwen2.5-0.5B.

From Microsoft's model card, and worth repeating: this model is intended for research and development purposes only, and they do not recommend using it in commercial or real-world applications without further testing. If you publish generated speech, disclose that it is AI-generated.