Hy3-MLX-MXFP4-imatrix

Hy3-MLX-MXFP4-imatrix

MX-float (mxfp4/mxfp8, group size 32) quantization of tencent/Hy3 (HunYuan V3, 80-layer MoE: 192 routed experts + 1 shared, sigmoid router with expert bias), placed by the imatrix + sensitivity analysis of its affine sibling unigilby/Hy3-oQ4e: mxfp4 where the oQ4e chose 4-bit (237 modules), mxfp8 where it boosted precision (562 modules — concentrated in the most sensitive layers 76–79 and 2–4), bf16 elsewhere.

• 150 GB (vs 158 GB affine oQ4e, from the 598 GB BF16 source).
• Same 128×512-token imatrix calibration (876 entries incl. per-expert statistics for all 192 experts) as the oQ4e.
• MTP (num_nextn_predict_layers) weights stripped; pure decoder checkpoint.
• Coherency at temperature 0: 5/6 vs the oQ4e's 6/6 (one 3-digit multiplication slip, 30504 for 847×36=30492; reasoning, code, facts, and format-following all exact).

⚠️ Benchmarked: prefer the affine oQ4e for this model

On a 12-category graded benchmark (thinking mode on), this MX-float quant lost clearly to its affine sibling Hy3-oQ4e at identical speed (~25 tok/s on M3 Ultra). The failure mode is sporadic stray-token garbling in long outputs — corrupted identifiers appearing inside otherwise-correct code, configs, and contract clauses (coding graded C vs the oQ4e's A-; prose and medical categories held A-range). The cause: mxfp4's parameter-free E2M1 rounding (power-of-two block scales, no zero-point, nothing for the imatrix to optimize within a block) damages precision-critical circuits that the imatrix-weighted affine fit preserves. The longer the generation, the more often a garble lands.

Interesting datapoint: the same recipe on MiniMax-M3 produced an MX-float sibling that matched its affine twin — MX-float-at-affine-placements is model-dependent, and for Hy3 it does not hold. This repo stays up as that datapoint; for serving, use Hy3-oQ4e.

Requirements

model_type: hy_v3 support is not yet merged into mlx-lm — it requires mlx-lm PR #1211 (adds mlx_lm/models/hy_v3.py, the hy_v3 tool parsers, and thinking-tag inference). Apply the PR (or install from its branch) before loading:

from mlx_lm import load, generate

model, tokenizer = load("unigilby/Hy3-MLX-MXFP4-imatrix")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello!"}], add_generation_prompt=True
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=200))

Quantized with oMLX quantize_oq_streaming (MX-float predicate at oQ4e placements) on a Mac Studio M3 Ultra.