pocket-tts-mlx-language-converter

pocket-tts-mlx-language-converter

Converts the non-English kyutai/pocket-tts language checkpoints into the layout that mlx-audio can load.

No weights are redistributed here. kyutai/pocket-tts is a gated repository — accept their conditions and download it yourself.

Source and issues: https://github.com/OpenMinded-Labs/pocket-tts-mlx-language-converter

The problem

Kyutai ships language checkpoints (German, French, Italian, Spanish, Portuguese) under languages/ in their original layout. mlx-audio implements the converted layout published as mlx-community/pocket-tts, which covers English only. Pointing mlx-audio at a language checkpoint fails on tensor names, then on shapes.

Usage

pip install mlx mlx-audio huggingface_hub numpy
huggingface-cli login          # the source repo is gated

python convert_language.py german
python load_speaker.py german lola "Ein Satz zum Testen." out.wav

Available: german, french, italian, spanish, portuguese, plus the deeper *_24l variants (pass --layers 24 to load_speaker.py).

What it does

Tensor names. The two layouts name the Mimi modules differently (mimi.decoder.model.0.conv.weight vs mimi.decoder.init_conv1d.conv.conv.weight). The mapping is not hardcoded from module structure — it is derived. The English weights exist in both layouts, so the same numbers appear under both names; matching by value fingerprint gives the mapping and the per-tensor operation (identity, reshape, or axis permutation). One trap worth naming: a byte fingerprint ignores shape, so (64,1,7) and (64,7,1) hash identically. The required operation is therefore determined by comparing each pair separately.

Latent width. Exactly two tensors differ in shape from English:

mimi.downsample.conv.conv.weight   lang (32,512,32)   en (512,512,32)
flow_lm.speaker_proj_weight        lang (1024,32)     en (1024,512)

encode_to_latent() does not use the quantizer; the latent comes from downsample, and the language checkpoints project 512 → 32 in that step, while mlx-audio's ConvDownsample1d uses a single dim for both sides. The converter zero-pads the downsample output channels to 512 and pads the speaker projection to match, so that

latent512 = [latent32 | 0]
W512      = [W32      | 0]
latent512 @ W512.T == latent32 @ W32.T

This is exact, not an approximation, and the script verifies it (maximum deviation must be 0) before writing anything. No model code is patched.

Layer count. The *_24l variants have 24 transformer layers where English has 6; the derived mapping is generalised over the layer index.

What it does not do

Voice cloning from your own audio does not work. The language checkpoints are a newer generation than the one mlx-audio implements: their bundled speaker files are pre-filled transformer KV caches, not the audio_prompt latents the English release uses. mlx-audio's ref_audio path builds state through _encode_audio(), which these checkpoints do not use — generation terminates after a fraction of a second.

Use the bundled speakers instead (languages/<lang>/embeddings/, 26 of them for German). load_speaker.py loads one directly into the model state; the only adjustment needed is the axis order, since the file stores (batch, position, head, dim) and mlx-audio wants (batch, head, position, dim).

Supporting cloning would mean implementing that newer conditioning path in mlx-audio. That is a port, not a weight remap, and is out of scope here.

Unconditioned generation is unreliable. With an empty state the 6-layer German model stops almost immediately; the 24-layer one produces audio but of poor quality. Pass a speaker.

Verifying output

Generation stopping early is the failure mode to watch for, and it is silent — you get a valid WAV file containing a fraction of the text. load_speaker.py prints the duration against a rough expectation (~15 characters of text per second of speech) and warns when it falls far short. Worth keeping if you build on this.

Tested with

macOS 15 on Apple M4, mlx-audio 0.2.x, Python 3.13. German 6-layer and 24-layer checkpoints. The other languages use the same layout but have not been listened to.

Licence

The scripts are MIT. The weights they convert are kyutai's, CC-BY-4.0, and subject to the access conditions of their repository.