chatterbox-turbo-lora

chatterbox-turbo-lora

LoRA-aware ONNX export of the complete Chatterbox-Turbo TTS pipeline, quantized to q4f16 for on-device inference (Android / iOS via ORT).

The language_model subgraph is a custom fork that exposes 192 extra input ports so a PEFT-trained LoRA adapter can be bound at runtime. The other three subgraphs (speech_encoder, embed_tokens, conditional_decoder) are byte-identical mirrors of the corresponding files from ResembleAI/chatterbox-turbo-ONNX/onnx/, hosted here so an on-device app has a single-repo source for the full pipeline.

What's new in v2.0.0

v1.0.0 of the language-model subgraph had no LoRA input ports — adapter weights could not be bound at runtime. v2.0.0 wraps every Conv1D projection inside the transformer (attn.c_attn, attn.c_proj, mlp.c_fc, mlp.c_proj) in a LoraConv1D that exposes two extra input tensors per layer:

lora.layers.{N}.{attn_c_attn|attn_c_proj|mlp_c_fc|mlp_c_proj}.{A|B}

→ 24 layers × 4 projections × 2 tensors = 192 extra graph inputs.

At inference time the host runtime binds a PEFT-trained LoRA adapter (rank, scaling baked in) to these inputs via the ONNX Runtime OrtLoraAdapter API. Unbound inputs default to zero-sized initializers so the graph still runs without any adapter loaded — the base voice synthesizes as-is.

Files

Four subgraphs, each a .onnx protobuf + a .onnx_data external weight blob. Every .onnx file references its .onnx_data companion by relative filename, so both files of a pair must be downloaded and kept next to each other.

Total: ~581 MB.

Quantization details

• Format: q4f16 (4-bit asymmetric weights + fp16 scales)
• Block size: 32 (matches Chatterbox's original 1.0.0 layout)
• Quantizer: onnxruntime.quantization.MatMulNBitsQuantizer
• Excluded from quantization (language_model only): all 192 LoRA-path MatMul/Add nodes (kept fp32 so adapter weights can be hot-swapped without retraining scales)
• The 96 GPT-2 projection layers in language_model are first rewritten from Gemm → MatMul + Add so they fall inside the quantizer's coverage; without this step the file would stay at ~1.2 GB.

Embedding tables (wte, wpe) remain fp32 because they are read via Gather, not MatMul, and MatMulNBitsQuantizer does not touch them.

The three mirrored subgraphs (speech_encoder, embed_tokens, conditional_decoder) use whatever quantization the upstream ResembleAI build applied. They are bit-for-bit identical to ResembleAI/chatterbox-turbo-ONNX/onnx/*_q4f16.onnx{,_data} — you can verify with sha256 against the upstream repo.

License

MIT, inherited from the upstream Chatterbox project.