Hy3-4bit-mtp-mlx

Hy3-4bit-mtp-mlx

4-bit MLX quant of tencent/Hy3 with the MTP head actually included.

The circulating MLX quants of Hy3 drop the multi-token-prediction layer (layer 80) because mlx-lm's converter doesn't map it. Their configs still say num_nextn_predict_layers: 1, but the weights aren't there, so speculative decoding can't run.

Contents

• Backbone: the 34 shards from XavierLocalAI/Hy3-4bit, unchanged. Checked before reuse: pass-through tensors are bit-identical to the official bf16 weights, and dequantization error sits at the expected bound for affine 4-bit.
• Layer 80 (MTP head, in model-mtp-00001.safetensors): quantized from the official tencent/Hy3 bf16 weights with mlx.core.quantize, same recipe as the backbone (affine, group size 64, 4-bit, router gate 8-bit). 99% of elements reconstruct within half a quantization step of the originals.
• Config: adds the layer 80 router gate override so the quant block matches the weights.

Serving

Runs on vLLM TPU (tpu-inference) with speculative decoding:

vllm serve enyoukai/Hy3-4bit-mtp-mlx \
  --trust-remote-code --enable-expert-parallel \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Measured on a v6e-8: draft acceptance length about 1.5 at temperature 0.7 with batch 8, about 1.7 single stream. Same ballpark as the preview-era quants got from their own MTP head.

Credits

• Base model: Tencent (tencent/Hy3, Apache-2.0)
• Backbone quant: XavierLocalAI / bicVanYonk
• MTP layer quant: this repo