Hy3-oQ4e

Hy3-oQ4e

oMLX oQ4e (imatrix-enhanced affine 4-bit, group size 64, sensitivity-driven mixed placements) quantization of tencent/Hy3 (HunYuan V3, 80-layer MoE: 192 routed experts + 1 shared, sigmoid router with expert bias).

• 158 GB (from the 598 GB BF16 source) across 32 safetensors shards.
• imatrix: 128×512-token calibration (876 entries incl. per-expert statistics for all 192 experts), collected by running an 8-bit affine proxy of the model — the BF16 source exceeds 512 GB unified memory, so calibration ran on the half-size proxy and the resulting activation statistics were applied to the BF16→4-bit quantization.
• Sensitivity: data-driven per-layer measurement (uniform 4-bit disk proxy); the most sensitive projections (final layers 76–79, early layers 2–4) receive 5-bit boosts.
• MTP (num_nextn_predict_layers) weights are stripped; this is a pure decoder checkpoint.
• Coherency-verified at temperature 0 (arithmetic, factual recall, code generation, trick-question reasoning, strict format following): 6/6.

Benchmark results (12-category graded suite, thinking mode on)

Graded near-expert across the board on a role-mapped benchmark (coding, QA, sales, marketing, legal, operations, sysops, devops + clinical/pharma/psych): A in eight categories, A- in coding/devops/pharma. Single-stream ~25 tok/s, TPOT ~41 ms on an M3 Ultra (512 GB). Run with reasoning_effort: "high" and the official sampling (temperature=0.9, top_p=1.0 — greedy decoding degrades the chain-of-thought).

This affine oQ4e clearly beat its MX-float sibling (Hy3-MLX-MXFP4-imatrix) at identical speed: mxfp4's parameter-free E2M1 rounding produces sporadic stray-token garbling in long outputs (corrupted identifiers inside otherwise-correct code and contract text; coding graded C vs this model's A-), while the imatrix-weighted affine fit preserves those precision-critical circuits. If you're choosing between the two, use this one.

Known model-level limitation (both quants, matches Tencent's Hy3-preview notes): weak error recovery in multi-turn tool calling — mid-conversation the model can emit tool calls as plain text without its sentinel tokens. Single-shot tool calls parse fine.

Requirements

model_type: hy_v3 support is not yet merged into mlx-lm — it requires mlx-lm PR #1211 (adds mlx_lm/models/hy_v3.py, the hy_v3 tool parsers, and thinking-tag inference). Apply the PR (or install from its branch) before loading:

from mlx_lm import load, generate

model, tokenizer = load("unigilby/Hy3-oQ4e")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello!"}], add_generation_prompt=True
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=200))

Quantized with oMLX quantize_oq_streaming (enhanced/imatrix mode) on a Mac Studio M3 Ultra.