Hy3-Alis-MLX-Dynamic

Hy3-Alis-MLX-Dynamic

Part of the Alis MLX — big models on Apple Silicon collection.

Mixed-precision MLX quantizations of tencent/Hy3 (295B total / 21B active MoE) for Apple Silicon, sized for 128 / 256 / 512 GiB unified-memory Macs. One repo, one card — each quantization lives on its own branch.

* single-stream decode on an M3 Ultra, short prompt; longer contexts decode slower.

Final state, all tiers

Everything below is measured, not projected — KL vs the 8-bit reference on a fixed EN/code/ZH slice, PPL on fixed corpora, and needle retrieval actually run at the stated context. All builds keep the router in bf16 and expert_bias in fp32.

• T128 — 87.6 GB, 128 GiB Macs, experimental. KL 0.550, PPL 5.80 / 2.44 / 5.40. The only build that fits a 128 GiB Mac's default wired limit; needle verified to 16K fp16 (32K with int8 KV, needs a wired-limit bump). Received both low-bit levers — sensitivity-placed 3-bit down_proj + Chinese-mixed DWQ (v2) — halving its KL vs v1.
• main (T256) — 168.5 GB, 256 GiB Macs, daily driver. KL 0.094, PPL 3.97 / 1.98 / 3.73. Needle verified to 64K fp16 (peak 190 GB, inside the default wired limit); 128K via int8 KV. Uniform 4-bit — already near-optimal for its size (see above).
• T512 — 242 GB, 512 GiB Macs, flagship. KL 0.020 (near-lossless), PPL 3.85 / 1.93 / 3.54, and ~25% faster to decode than the 8-bit. Needle verified to 128K fp16; full 256K fp16 KV fits the default wired limit.
• T512REF — 313.5 GB, 512 GiB Macs, reference. Uniform 8-bit; the KL ground truth and DWQ teacher. Ship only if you want the maximum-quality artifact.
• mtp-bf16 — 7.5 GB sidecar + harness. Hy3's MTP head plus a self-speculative decode loop; measured 1.13× lossless (k=1) on top of any tier.

Which build?

• 512 GiB Mac → T512. Faster than the 8-bit and statistically near-lossless (KL 0.020 nats). 128K-token needle retrieval verified at 285 GB peak; 256K fp16 KV fits in the default wired limit (242 + 80 = 322 GB < ~384 GB).
• 256 GiB Mac → main. 64K-token needle verified at 190 GB peak — just inside the default ~192 GB wired limit. For 128K, use int8 KV (--kv-bits 8, generate path) or raise iogpu.wired_limit_mb.
• 128 GiB Mac → T128, with eyes open: see the experimental warning below. 16K fp16 verified at 93.8 GB peak; 32K with int8 KV peaked at 109 GB, which requires raising the wired limit on a 128 GB machine.
• KL ground truth / archival → T512REF (uniform 8-bit).

Usage

Hy3 (hy_v3) support is not yet in an mlx-lm release (ml-explore/mlx-lm#1211 is open). Until it merges, install from the pinned branch (PR #1211 plus an fp32-router correctness patch, commit 14f7837):

pip install git+https://github.com/avlp12/mlx-lm@hy3-support

from mlx_lm import load, generate

# main = the 256 GiB build; pass revision= for other tiers
model, tokenizer = load("avlp12/Hy3-Alis-MLX-Dynamic")
model, tokenizer = load("avlp12/Hy3-Alis-MLX-Dynamic", revision="T512")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain MoE routing in two sentences."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

CLI / server:

mlx_lm.generate --model avlp12/Hy3-Alis-MLX-Dynamic --prompt "..." -m 512
hf download avlp12/Hy3-Alis-MLX-Dynamic --revision T512 --local-dir Hy3-T512
mlx_lm.server --model ./Hy3-T512 --port 8080

Long context on the smaller tiers: quantize the KV cache in the generate path (--kv-bits 8 --kv-group-size 64). Note mlx_lm.server currently serves fp16 KV only.

Recipes (verified from each build's config.json, not from intent)

97.0% of Hy3's parameters are the 192 routed experts, so expert bit-width sets the size; everything else is cheap to keep at high precision. Common to all tiers:

• mlp.router.gate — never quantized (bf16). The sigmoid+bias top-8 routing is a discrete control path; 62M params total, so full precision is free.
• expert_bias (e-score correction) kept fp32, verified in the output shards.
• Attention bits = min(expert bits + 2, 8).

T128 (v2) — two byte-neutral quality levers. (1) The 12 down_proj layers upgraded to 3-bit are chosen by measured sensitivity (each MoE layer's KL when dropped to 2-bit, probed on the 8-bit reference), not an early/late heuristic — the most sensitive layers turned out to be the middle band (46-65). (2) A DWQ pass (scales/ biases distilled against the 8-bit build's logits) on a 45% Chinese data mix, so the language-specialized experts actually receive gradient. Same 87.6 GB, same recipe bits. Net vs the raw quantization: overall KL −42%, ZH −46% (1.55→0.83). The prior v1 (early/late layers + English-only DWQ) reached only ZH 1.43; it is tagged T128-v1.

Quality

KL divergence & top-1 flip rate vs T512REF — measured on a fixed 3,072-token slice (English / code / Chinese thirds). Disclosure: the reference is the 8-bit build, not bf16 — the bf16 model (598 GB) does not fit in 512 GB of RAM.

Perplexity (strided, ctx 2048 / stride 1024, fixed corpora):

lm-eval (hellaswag / piqa / winogrande, 0-shot, limit 500, same harness across tiers — absolute numbers are base-style loglikelihood, useful only for tier-vs-tier comparison): all three tiers are statistically indistinguishable (±2σ), e.g. piqa acc 0.75–0.77 across every tier. Loglikelihood tasks are insensitive at this scale; the KL table above is the discriminating metric.

Long-context needle retrieval (needle at 25% depth, greedy): every listed claim was tested, not extrapolated —

⚠️ T128 is experimental

2.375 bpw on a 295B MoE with 18.9M-parameter experts is aggressive. The v2 build generates fluent English/Korean/Chinese in under 88 GB and passes 16K needle retrieval (peak 93.8 GB). Its remaining distribution gap vs the 8-bit reference is modest overall (KL 0.55) and no longer collapses on Chinese (ZH KL 0.83 vs v1's 1.43, ZH PPL 5.40 vs v1's 9.01). Still the most aggressive tier — verify on your workload; for headroom, the 256 GiB main build is far closer to lossless.

Why only T128 got the extra tuning

The two levers that transformed T128 — placing the higher-bit down_proj layers by measured per-layer sensitivity, and a Chinese-mixed DWQ pass — are 2-bit-specific, and we verified this rather than assumed it:

• Sensitivity shifts with bit-width. Probing each MoE layer's KL when dropped to a given bit-width, the most fragile layers at 2-bit are the middle band (46–65) but at 4-bit they are the early layers (1–14) — so T128's layer set doesn't transfer. And a single layer's 4-bit damage is only ~39% of its 2-bit damage: far less to reclaim.
• The byte-neutral trick backfires at 4-bit. A mixed down_proj recipe (5-bit on sensitive layers, 3-bit on robust ones, same total size) improved T128 but measured worse on main (paired ΔKL −0.013): the 4-bit error surface is flat enough that demoting robust layers costs more than promoting sensitive ones saves.
• DWQ has little to correct at 4-bit. Starting a DWQ pass on main, the initial validation loss was 0.05 vs T128's ~0.5 — main already sits ~10× closer to the 8-bit reference, and its Chinese KL (0.169) never collapsed the way T128's did (1.55).

So main, T512, and T512REF ship as uniform quantizations: at 4-bit and above the naive recipe is already near-optimal for its size, and every intervention we measured was neutral-to-negative. The aggressive tuning is reserved for the tier that needs it.

How it compares to other Hy3 MLX ports

Every public Hy3(-preview) MLX quantization we could find, measured on the same harness, same machine (M3 Ultra 512GB, wired-limit set, quiet disk): identical 3,072-token KL slice vs our 8-bit reference, identical PPL corpora, same prompt for decode speed. Sorted by size:

† KL(8-bit ref ‖ model) on the fixed EN/code/ZH slice — lower is better. ‡ mlx-community's build quantizes Hy3-preview, a different base checkpoint than the final Hy3; its KL vs our final-Hy3 reference includes base-model drift, so compare it on PPL. Its uniform 4-bit g64 recipe (8-bit routers) decodes faster than mixed recipes because its attention path carries fewer bytes per token.

Reading the table:

• 4-bit class (~161–169 GB): at essentially the same size, main has the lowest KL of any final-Hy3 build (0.094 vs 0.172 for the imatrix MXFP4 build — ~45% lower) and the best code/ZH perplexity, at equal speed.
• low-bit class: T128 (v2) at 87.6 GB now has the lowest KL of any sub-160 GB Hy3 build measured (0.550 vs ox-ox's 0.686 at 112.6 GB and unigilby's 0.172 at 161 GB), and it is the only build that fits a 128 GB Mac's default wired limit. ox-ox spends 25 GB more (8-bit attention) yet scores worse and exceeds the 128 GB ceiling before the first token, decoding at 8.9 tok/s vs T128's 27. If your Mac has 256 GB, use main instead; if it has 128 GB, T128 is the build that actually fits.
• Routers: ours keep the sigmoid+bias router in bf16 (ox-ox and mlx-community quantize it to 8-bit; unigilby's config applies its 4-bit default to the router). With 192 experts and near-tie top-8 margins, full-precision routing is the cheapest correctness insurance in the recipe (~124 MB total).

mtp-bf16 branch — MTP self-speculative decoding (measured: 1.13×, lossless)

Hy3 ships a Multi-Token-Prediction layer (model.layers.80.*, 3.75B params, DeepSeek-V3-style eh_proj/enorm/hnorm + MoE block) that mlx-lm strips at conversion. The mtp-bf16 branch preserves it verbatim in bf16 (7.5 GB) plus a runnable self-speculative-decode harness (mtp_run.py + module + step generators).

Measured on the 8-bit build (M3 Ultra, greedy, 200 tokens, output verified token-identical to plain decode):

Notes: k=1 with the bf16 sidecar is the operating point. A 4-bit draft was measured too (isolated draft call 2.26 → 1.74 ms, −23%): the saving is real but amounts to ~1% of loop time (the 47 ms verify forward dominates), while acceptance drops slightly and consistently across EN/KO/code prompts (accept-len −0.02…−0.06) — net ≈ −1…−4% vs bf16, tie on code. Use the 4-bit draft only if you want its 5.4 GB memory saving. The harness uses the MTP's own KV cache and final-layernorm ("normed hidden") chaining for k>1.

hf download avlp12/Hy3-Alis-MLX-Dynamic --revision mtp-bf16 --local-dir mtp
python mtp/mtp_run.py --model <any-tier> --mtp mtp/Hy3-MTP-bf16.safetensors --k 1

Correctness

• Cross-framework logit parity vs transformers 5.12.1 reference implementation on tiny-random weights: top-1 agreement 100%, max |Δlogit| ≤ 5e-3 (fp32).
• One correctness patch on top of PR #1211: the router matmul runs in fp32, matching the reference F.linear(hidden.float(), weight.float()) — near-tie top-8 selection under a sigmoid+bias router is sensitive to bf16 matmul noise.
• Recipe verification: quantization blocks parsed from each output config.json; router dtype and expert_bias fp32 checked in the shards themselves.

Credits

• Tencent Hunyuan for Hy3 (Apache 2.0) and the AngelSlim compression toolkit — the official quantization path for CUDA deployments (FP8/INT4 for vLLM/SGLang). These MLX builds are an independent, complementary Apple-Silicon path.
• kernelpool for the hy_v3 MLX port (ml-explore/mlx-lm#1211).
• Built with MLX and mlx-lm.

Citation — if you use these builds, please cite Tencent's Hy3 release (tencent/Hy3, 2026) and Apple's MLX framework.