chatterbox-multilingual-hi-mlx-fp16

chatterbox-multilingual-hi-mlx-fp16

Chatterbox Multilingual (Hindi) by Resemble AI, converted to MLX format (fp16) for fast local inference on Apple Silicon via mlx-audio.

This is a single-language Hindi text-to-speech and voice-cloning model, fine-tuned by Resemble AI from their Chatterbox Multilingual v3 base model. All credit for the model itself goes to Resemble AI — this repo only repackages the same weights for the MLX runtime.

Usage

pip install -U mlx-audio

mlx_audio.tts.generate --model gagan1985/chatterbox-multilingual-hi-mlx-fp16 \
    --text "नमस्ते, यह हिंदी में एक परीक्षण है।" \
    --ref_audio reference.wav \
    --lang_code hi

from mlx_audio.tts.generate import generate_audio

generate_audio(
    text="नमस्ते, यह हिंदी में एक परीक्षण है।",
    model="gagan1985/chatterbox-multilingual-hi-mlx-fp16",
    ref_audio="reference.wav",
    lang_code="hi",
    file_prefix="output",
)

ref_audio is a short reference clip (a clean 10-15s sample works best) used for zero-shot voice cloning — Chatterbox always requires one, there's no built-in default voice baked into this checkpoint. On first run, mlx-audio also auto-downloads the shared S3TokenizerV2 weights it depends on.

Useful generation flags: --exaggeration (0-1, emotional intensity, default 0.5) and Chatterbox's cfg_weight (default 0.5, guidance strength) — note that on current mlx-audio CLI releases, cfg_weight isn't reachable via a --cfg_scale-style flag for Chatterbox specifically; use the Python API and pass cfg_weight= directly to model.generate(...) if you need to tune it.

Conversion notes

This checkpoint combines:

• t3_hi.safetensors — the Hindi-fine-tuned T3 (text-to-speech-token) model, from ResembleAI/Chatterbox-Multilingual-hi
• ve.safetensors and s3gen.safetensors — the shared voice encoder and speech decoder, from the base ResembleAI/chatterbox repo (only T3 was fine-tuned for Hindi; these components are unchanged)

Note on the speech decoder: the Hindi repo also ships an s3gen_v3.safetensors checkpoint, structurally identical (same tensor names/shapes) to the base repo's s3gen.safetensors. An earlier version of this conversion used s3gen_v3.safetensors, since it's what Resemble AI bundles alongside the Hindi checkpoint — but that produced a persistent breathy/airy artifact in the MLX output that wasn't present when the same weights were run through the original PyTorch implementation, and wasn't present in mlx-community/chatterbox-multilingual-v3 either (which uses the base s3gen.safetensors). Swapping to the base repo's s3gen.safetensors — same file mlx-community's own v3 conversion and the official PyTorch loader both use — resolved it. This repo uses that known-good decoder. The likely explanation is an incompatibility between mlx-audio's S3Gen port and the specific s3gen_v3 checkpoint's values; worth reporting upstream to Blaizzy/mlx-audio if you hit similar artifacts converting other v3-era Chatterbox checkpoints.

Converted using mlx-audio's own model classes and sanitize() methods (the same code path used for the official mlx-community Chatterbox conversions), with strict weight-validation on load to catch any mapping mismatch at conversion time.

License

MIT, matching the source model.