Hojo-TTS-Light-40M-INT8

Hojo-TTS-Light-40M-INT8

int8 quantization of the decoder LM graph (Hojo-TTS-Light-40M-llm.onnx) from HojoAI/Hojo-TTS-Light-40M at revision c3cf21c77dc04b220aa4dabe20fc59e66a2a5485.

This repository intentionally holds only that one graph. The install pipeline resolves each snapshot file independently, so the remaining components (decoder, fine_local, voice bank, tokenizer, config) continue to come from the upstream snapshot; only the LM is served from here.

File

The upstream file at the revision above is BF16 (63138277 bytes) and is promoted to FP32 (125813561 bytes) at install time. This file replaces that intermediate step: it is already int8, so the promotion pass finds no BF16 tensors and leaves the numerics untouched (verified bitwise-identical step outputs).

How it was produced

pip install onnxruntime
python tool/quantize_hojo_tts_llm.py <installed-model-dir>

which runs:

onnxruntime.quantization.quantize_dynamic(
    source, target,
    weight_type=QuantType.QInt8,
    op_types_to_quantize=["MatMul", "Gemm"],
    per_channel=True,
    reduce_range=False,
)

Only the autoregressive decoder LM is quantized. The vocoder graphs (decoder, fine_local) are deliberately left at their original precision: dynamic weight-only quantization buys little on their convolutions, and they are where audible artifacts would come from.

Why this graph

The LM runs once per generated audio token — roughly 150-190 session runs for a 35 character sentence — so it dominates synthesis time. One decode step, measured on an i5-12600KF (16 logical cores, 6P+4E):

End to end through the synthesis pipeline the real-time factor for the same sentences falls from 1.20-1.32 to 0.57-0.67 — from slower than real time to faster than real time, which is what lets a read-aloud queue keep up with speech.

Note that quantizing shifts the optimal thread count: 8 threads beat 4 for int8, while the reverse holds for fp32.

License

Apache-2.0, inherited from the base model (see LICENSE). This repository is a precision-only derivative of the upstream weights; the architecture is unchanged.