chatterbox-multilingual-coreml

Chatterbox Multilingual — all-CoreML (ANE-optimized)

On-device Apple Silicon build of ResembleAI's multilingual Chatterbox (t3_mtl23ls_v3, 23 languages) for iOS 18+ / macOS 15+, arm64. The whole pipeline is CoreML; T3 prefill + decode run on the Apple Neural Engine and the S3Gen synth on ANE + GPU. v1 retains full multilingual capability and is validated/tuned on Russian (on-device, ear-checked, iPhone 17 Pro Max / iOS 26.5.1).

All 23 languages generate speech with upstream's own per-language demo prompts (checked for plausible output length, not ear-checked outside Russian).

Turbo (English, GPT-2) sibling: iliasaz/chatterbox-turbo-coreml.

Links

• Upstream model — ResembleAI/chatterbox by Resemble AI.
• Runtime — iliasaz/chatterbox-coreml, the Swift package + demo app that loads this repo.
• Watermarking — iliasaz/perth-coreml (Swift) / weights. Every utterance the runtime generates is watermarked with Perth, as upstream chatterbox does.
• Russian stress — iliasaz/ruaccent-coreml (Swift) / weights, a CoreML port of RUAccent.

How it differs from turbo

The multilingual T3 is a different transformer, not a re-export:

• LLaMA_520M, 30 layers, RoPE + RMSNorm + SwiGLU (SiLU), split q/k/v/o, bias-free (turbo = GPT-2-medium, 24 layers, learned wpe, LayerNorm, GELU).
• Classifier-free guidance at batch=2 — the uncond lane is the same prefix with the text embedding zeroed; per step logits = cond + cfg·(cond − uncond).
• Cond prefix = speaker(1) + Perceiver(32) + emotion(1) = 34 rows; the Perceiver resampler and emotion vector are computed host-side.
• Grapheme BPE tokenizer (text vocab 2454); speech side (start 6561 / stop 6562) unchanged. Russian stress comes from iliasaz/ruaccent-coreml (neural), with a manual U+0301 override / dictionary fallback.

Artifacts

The presence of perceiver_query.npy is how the consumer app auto-detects the multilingual variant.

Pipeline

text (+ optional Russian stress) → grapheme BPE → host-assemble inputs_embeds
     (speaker + Perceiver(32) + emotion + text) + position_ids + masks
     → T3LM "prefill" (CoreML, ANE)            — batch-2 CFG, writes shared KV state
     → T3LM "decode"  (CoreML, ANE)            — autoregressive, host CFG combine per step
     → S3Encoder (ANE) → S3CFM 10-step CFG (GPU) → S3Vocoder (GPU)  — 24 kHz waveform

On-device numbers (iPhone 17 Pro Max, iOS 26.5.1, warm)

The fp16 T3LM fails the iPhone ANE plan build (error -14); the shipped T3LM is 8-bit palettized, which loads both functions on the ANE (parity vs fp16: cos 0.9972, 9/9). The runtime persists the compiled model across launches and overlaps chunk N+1 decode (ANE) with chunk N synth (GPU).

Russian defaults

Mirrors the upstream multilingual server: temperature 0.8, exaggeration 1.3, cfg_weight 0.5, top_p 0.95, repetition_penalty 1.2 (min-p sampling, no top-k).

License

MIT. These weights are a CoreML format conversion of ResembleAI/chatterbox, which Resemble AI publishes under the MIT licence; the conversion inherits those terms, and copyright in the weights stays with Resemble AI — no claim of ownership is made over them here. Redistribute with attribution to Resemble AI.

The Swift runtime that loads them is separately MIT-licensed; see the LICENSE and NOTICE files in iliasaz/chatterbox-coreml.