VibeVoice-Realtime-0.5B-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) -->

This model has no row on DeviceMark, the on-device LLM leaderboard.

<!-- gen-cards:devicemark end -->

VibeVoice-Realtime-0.5B — Core AI

microsoft/VibeVoice-Realtime-0.5B (MIT) converted to Apple Core AI — the zoo's first multi-speaker / dialogue (podcast-style) TTS. iPhone + Mac, all-fp16.

Not a "first on-device VibeVoice" claim — other CoreML/GGUF ports exist. What this is: the zoo's first multi-speaker TTS, app-integrated, and the other half of a generate -> diarize loop with the zoo's Streaming Sortformer diarizer.

Architecture

Dual Qwen2.5 LM (4-layer text context LM, norm = Identity + 20-layer speech trunk) -> per-frame next-token diffusion (4-layer adaLN head, DDPM cosine, v-prediction, DPMSolver++ 5-step, CFG 1.5) -> causal-conv acoustic VAE decoder (7.5 Hz latent -> 24 kHz, 3200 samples/frame). The LM predicts one latent per frame; the diffusion head denoises it; the VAE decoder renders audio. Multi-speaker output is host turn-switching: each Speaker N: turn is generated from its own voice preset and the turns are concatenated — no multi-speaker prefill, no acoustic encoder.

Contents

fp16 is required. int8 LMs diverge inside the speech feedback loop (min cos 0.187, early EOS); the diffusion head is fp16-sensitive too (pure-torch fp16 collapses to 0.79 — Core AI keeps the RMSNorm/adaLN reductions in fp32, so the host DDPM reference must run fp32).

Fixed shapes only. Every graph is static (q=1 decode, fixed-T decoder), so the runtime must not be given the expectFrequentReshapes hint on iOS. With the h18p AOT bundles on the iPhone 17 Pro, the hint made the runtime skip the AOT specialization and compile on device. That compile segfaulted inside the MPSGraph AICode compiler.

Gates

On the iPhone 17 Pro with the h18p AOT bundles: 6 graph loads in 2.6 s (warm), 24 latents / 3.20 s of audio in 2.3 s = 10.6 tok/s ~ 1.4x real-time.

Measured 2026-09-26 on an iPhone 18 Pro (iPhone19,2, iOS 27.0 build 24A437, h19p) with the zoo's DecideGate app in its load-only mode, without the increased-memory entitlement. Each first load was the graph's first in an app container that held no specialization of it; the app had run other graphs before. The call is one run on all-zero inputs; a graph with a KV state was loaded only. Each first load wrote a specialization of about the bundle's size into the app container, and the load after a relaunch reused it. One measurement per graph (knowledge/jit-distribution.md). The same phone refuses an h18p bundle with incompatibleCompiledAssetArchitecture.

Use it

import CoreAIKit

let dialogue = try await KitDialogue(catalog: "vibevoice-realtime-0.5b")
let (audio, turns) = try await dialogue.perform("""
    Speaker 1: Did you know this runs entirely on the phone?
    Speaker 2: No cloud at all? That is wild.
    """)

Swift host reference: ondevice/VibeVoiceRunner (Mac) and VibeVoiceSelfTest.swift in the zoo's coreai-audio app — raw Core AI stateful-KV loop + a Swift DPMSolver++ sampler. Python host + conversion recipe: conversion/vibevoice (host_e2e.py = the full generate loop, host_multispeaker.py = the dialogue demo).

Base model: microsoft/VibeVoice-Realtime-0.5B (MIT). EN/ZH. Community port — not an Apple model.

<!-- funnel:v1 -->

---

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.

<!-- /funnel:v1 -->