GLM-5.2-Alis-MLX-Dynamic-3.5bpw

GLM-5.2-Alis-MLX-Dynamic-3.5bpw

Part of the GLM-5.2 · Alis MLX Dynamic collection.

Apple Silicon (MLX) mixed-precision quantization of zai-org/GLM-5.2 — a 744B-parameter (~40B active) Mixture-of-Experts model with DeepSeek-V3.2-style MLA + DeepSeek Sparse Attention (DSA, glm_moe_dsa).

This build targets the "golden spot" for a 512 GB M3 Ultra: the best quality that still runs a full 1M-token context comfortably — ~3.5 bits/weight, served with int8 quantization of the MLA latent KV cache.

New: the checkpoint now ships GLM-5.2's native MTP (nextn) layer for lossless self-speculative decoding (--mtp — see Native MTP). Fully backward-compatible: runtimes without MTP support simply strip the extra layer and behave exactly as before.

New (2026-07): quality-flagship sibling — GLM-5.2-Alis-MLX-Dynamic-4.5bpw promotes the routed experts to NVFP4 (4-bit float, gs16): strided PPL ≈−2.3% (wikitext) / ≈−0.6% (code) vs this build (successive DWQ retunes have narrowed the gap: −7.9%/−3.8% pre-DWQ → −4.8%/−1.1% with the 4.5 bpw teacher → this, with the 8-bit teacher — see below), and MTP pays more on its cleaner target (accept-len 2.61–2.87, +15–16%), at ~424 GB with a ~500–600K-token context ceiling (requires mlx ≥ 0.30.3). This build remains the 1M-context choice — pick the 4.5 bpw sibling when quality matters more than context budget.

Update (2026-07, current main): anchor-guarded clip-search + the same 8-bit-teacher DWQ, stacked. Before re-running DWQ, the routed experts' packed codes were re-decided with a per-group clip search (alis-dwq clip_quantize): a clipped grid is accepted only when it lowers reconstruction MSE without raising the group's max abs error beyond 1.1× the min-max grid's — protecting the super-weights that min-max grids anchor exactly (the MSE-only variant destroys the model; full saga in the E1 floor-spike case study). Wikitext 2.946 (pre-DWQ) → 2.879 (clip only) → 2.7774 [2.745–2.810] — a further −1.3% beyond the previous retune (preserved on dwq-8bit-noclip); ZH −1.4%; code/tulu within noise. Greedy degeneration probe clean on all slices. A router-KD arm (gates trained alongside scales/biases) was A/B-tested and came out neutral — not shipped. Provenance: the expert codes now derive from the public Q8 checkpoint (pipenetwork/GLM-5.2-MLX-8bit), not bf16 — Q8's noise floor sits ~30 dB below 3-bit quantization noise; disclosed for rigor.

Update (2026-07): DWQ-retuned against an 8-bit teacher. main now carries quantization scales/biases re-tuned with layerwise DWQ (alis-dwq; upstream flag: mlx-lm#1499) against an 8-bit teacher — higher-precision than the 4.5 bpw sibling used for the first retune — with a 45%-ZH calibration mix. The 8-bit teacher is 790 GB (too large for one box), so its top-k logit targets were dumped via a distributed pipeline forward across two 512 GB boxes, then the student was tuned single-node against the dump. Same recipe, size class and speed; strided PPL improves over both the pre-DWQ build and the earlier 4.5 bpw-teacher retune:

The 8-bit teacher adds a further −1.3% wikitext beyond the 4.5-teacher retune — statistically significant, the 4.5-teacher figure sits above this build's 95% CI [2.781, 2.847]; code is nominally lower but within noise. The gain is modest because at 3.5 bpw the student's capacity, not the teacher's precision, is the binding constraint. We tested the more-quantized 2.56 bpw build (its pre-clip retune; that build's main has since moved to a clip + 4.5-teacher rework) with this same 8-bit teacher and it did not improve — strided PPL edged up slightly (within noise) even though its DWQ-vs-teacher loss dropped 35%: at 2 bpw the student is too capacity-starved to exploit a sharper teacher, and chasing its finer targets overfits the calibration set without helping held-out PPL. There is a teacher-precision / student-capacity sweet spot, and 3.5 bpw sits on the right side of it while 2.56 bpw sits on the wrong side. The pre-DWQ weights remain on the pre-dwq branch; the earlier 4.5 bpw-teacher retune on dwq-4.5teacher. The Quality figures below are measured on the current main (clip + 8-bit-teacher DWQ) weights.

⚠️ Requires a patched mlx-lm with the glm_moe_dsa correctness fixes and int8 MLA-KV support.

This build uses IndexShare — the DSA indexer is stored only on the 21 "full" layers ([0,1,2,6,10,…,74]); the other 57 layers reuse it. Stock mlx-lm / omlx build an indexer on every layer, so loading fails with ValueError: Missing 285 parameters: …self_attn.indexer… (57 shared layers × 5 params). Use avlp12/mlx-lm, branch glm-5.2-dsa-inference (upstream PR ml-explore/mlx-lm#1419, open as of writing), which adds IndexShare + non-interleaved indexer RoPE + int8 MLA-KV. Serve with mlx_lm.server from that fork. On stock mlx-lm, --kv-bits is also a silent no-op for this architecture (see Correctness and Long context).

---

Quality

A 2-bit (≤256 GB) build of this model is bit-starved; lifting the budget to 512 GB and spending 3.5 bpw is the single biggest quality lever.

Strided perplexity (ctx 2048 / stride 1024) from a fixed local harness — relative numbers for comparing builds, not directly comparable to perplexities other quantizers report on different corpora.

---

Recipe

Bits are allocated by sensitivity — cheap bits on the robust expert bulk, full precision on the discrete routing/sparse-attention control paths.

---

Long context (1M) — int8 KV

GLM-5.2's MLA stores the compressed latent (kv_lora 512 + rope 64 per layer), so the 1M-token KV cache is small: ~95 GB at fp16, ~48 GB at int8. Quantizing the latent cache to int8 is what makes 1M fit comfortably at 3.5 bpw.

Serve with int8 KV:

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --kv-bits 8 --kv-group-size 64 --quantized-kv-start 4096 \
  --prompt "…"

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --kv-bits 8 --quantized-kv-start 4096

The patched runtime quantizes only the MLA latent cache to int8 (the DSA indexer cache stays fp16) and dequantizes the latent on read inside MLA attention. At ≤128K context, fp16 KV is fine (a few GB); int8 is only needed to keep 1M comfortable.

Why int8 and not fp8? A common suggestion is to switch the KV latent to fp8 for long-context quality. Measured on the actual latent, that would hurt: the MLA latent is post-kv_a_layernorm (RMSNorm), so its distribution is tight and well-conditioned (measured std ≈ 0.006, max ≈ 0.03). Reconstruction fidelity on the real captured latent — int8-affine (g64) cos 0.99998 vs MXFP8 (g32) 0.9997 vs plain fp8 e4m3 0.994 — int8 is 100–200× lower MSE on both the latent and the decoupled RoPE key. fp8's wider dynamic range doesn't help a narrow, well-conditioned distribution where precision (mantissa bits) is the binding constraint, and MLX dequantizes the latent to fp16 for the MLA matmul anyway (no native fp8 GEMM on this path, so no throughput win either). At the same ~1-byte footprint, int8 is the better choice for long-context fidelity — hence this build keeps int8.

---

Native MTP — self-speculative decoding

GLM-5.2 ships a built-in MTP (multi-token-prediction / "nextn") layer that predicts token t+2 from the backbone's hidden state at t — DeepSeek-V3-style. Every public MLX build (including earlier revisions of this one) stripped it at conversion; this build restores it as model.layers.78.* (one extra shard, +4.51 GB), quantized to match the rest of the recipe (experts 3-bit g64, attention 4-bit, indexer/norms/eh_proj bf16 — measured: drafter precision does not affect acceptance, bf16 ≡ 3-bit).

Backward compatible: loaders without MTP support drop layer 78 in sanitize() and behave byte-identically to the previous revision. With the fork, --mtp turns it on:

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --mtp --prompt "…"                        # k=2 chained drafts (default)

mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --mtp --mtp-num-draft-tokens 2            # OpenAI-compatible serving
  # optional: --mtp-hybrid  (adds conservative prompt-lookup drafting
  #            for repetition-heavy workloads: long quotes, JSON, boilerplate)

Measured on this build (M3 Ultra 512 GB, single request, greedy, short context; these MTP figures predate the 2026-07 DWQ retune — plain decode is format-identical under DWQ, and single-request MTP stays roughly neutral on the cleaner DWQ'd target, where draft acceptance holds or improves slightly):

Set expectations accordingly: on this 3-bit-expert build, single-request MTP is roughly performance-neutral — the aggressively-quantized target accepts fewer drafts (its own outputs are noisier and harder to predict) and each verify pass streams extra MoE experts, which offsets the drafting gain. A 4-bit sibling of this quant measured +11% (22.0 → 24.6 tok/s, accept-len 2.46) with the identical code path, and the gain grows further with target precision and with multi-node serving (speculation amortizes fixed per-forward costs). It stays exactly lossless — at temperature 0 the output is the model's own greedy output; at temperature > 0 drafts are accepted only when they equal the target's sampled token (distribution-exact) — so enabling it costs nothing but the ~4.5 GB layer.

Long context: MTP works past index_topk (2048 tokens — the DSA-sparse regime). Draft acceptance holds there, and the fork's small-L gather path (commit 946c412) keeps the multi-token verify from paying the sparse-mask setup per iteration (before that fix, long-context MTP decoded at 0.26× plain). Measured on a 4-bit sibling at ~2.1K context, prefill-separated: --mtp k=2 ≈ 0.85× plain, k=1 ≈ 0.98× — so at very long contexts prefer --mtp-num-draft-tokens 1 (≈ neutral) or plain; the k=2 gain (+15%) lives in the ≤2048 dense regime. The remaining long-context gap is acceptance economics (accepted length ~1.8 vs ~2.1 break-even), not path cost.

Notes for other integrators (the three footguns, all verified against vLLM's deepseek_mtp semantics):

1. eh_proj input concat order is [enorm(embed(t+1)), hnorm(h_t)] — embedding first.
2. The hidden fed to the MTP is the backbone's pre-final-norm residual stream.
3. For chained multi-token drafting, feed back the MTP's shared_head.norm-normalized hidden — chaining the raw layer output halves chained acceptance (measured 0.27 → 0.75 conditional at position 2).

---

How it compares

Against other public GLM-5.2 MLX builds, this one is the smallest on disk and the only one that runs a full 1M-token context within 512 GB — it spends its budget on context headroom + an int8 MLA-KV cache instead of purely on weight bits. (For the author's own higher-quality tier, see the 4.5 bpw nvfp4 sibling, compared in detail on its card.)

Per-component bit allocation was parsed from each build's config.json. The higher-bit builds (4.8–5.0 bpw) likely carry higher raw weight fidelity, but their footprint cannot hold a 1M KV cache on 512 GB. This build also keeps the DSA indexer at fp16 (the others quantize it) and bakes the indexer RoPE/eps long-context fixes.

¹ 328 GB base + 4.5 GB optional MTP layer (stripped automatically by runtimes without MTP support; the memory charts above show the base build — add +4.5 GB when serving with --mtp).

---

Benchmarks

Reproduced with mlx_lm.evaluate (0-shot) and mlx_lm.perplexity (seq 2048, 50 samples, seed 123), against the author's earlier GLM-5.1 quant under the same harness and settings:

Note (2026-07-10): the GLM-5.1 · 2.7 bpw build referenced below (avlp12/GLM-5.1-Alis-MLX-Dynamic-2.7bpw) has been removed from the Hub to reclaim storage; its figures are kept here for reference only.

Perplexity here is on allenai/tulu-3-sft-mixture (the mlx_lm.perplexity default) — a different corpus and method from the wikitext strided figure in Quality above, so values are not comparable across the two sections. Task accuracies use a 500-sample limit (CI ±0.02–0.04). All figures in this column are measured on the current main (clip + 8-bit-teacher DWQ): tulu 3.644 ±0.029 and HellaSwag 0.626 / PIQA 0.838 / WinoGrande 0.780 — each within 1 CI of the previous retune (dwq-8bit-noclip: 3.630, 0.604 / 0.822 / 0.780), i.e. no task-level regression. The campaign's real, significant gain is on the more sensitive strided PPL in Quality above. The 2.56 bpw column is that build's current main (clip + 4.5-teacher DWQ, re-measured 2026-07). GLM-5.1 is a different (older) base model, so cross-generation gaps reflect both the newer model and quantization.

---

Correctness (verified vs the HF reference)

glm_moe_dsa needs fixes beyond the stock mlx-lm port; this build was produced with a patched fork and validated:

• IndexShare — the DSA indexer runs only on "full" layers; "shared" layers reuse its top-k (index_topk_freq=4). The stock port built an indexer on every layer → missing-weights / wrong >2048-token output.
• Indexer RoPE / eps — non-interleaved (half-split) RoPE + LayerNorm eps 1e-6, distinct from the interleaved main attention. Baked into config.json (indexer_rope_traditional=false, indexer_norm_eps=1e-6); post-RoPE q matches the reference to ~1e-7.
• int8 MLA-KV — CacheList.to_quantized/offset + MLA dequant-on-read, so --kv-bits 8 actually engages for the latent cache (silently ignored on stock mlx-lm).
• Native MTP — layer 78 restored + --mtp self-speculative decoding in the fork; greedy output verified identical to non-speculative decoding, chained drafting matches vLLM's deepseek_mtp semantics (normed-hidden chaining).

Validation: full-attention logits match the HF reference to float precision at ≤index_topk context; long-context needle retrieval in the sparse-DSA regime; coherent prose + code generation; int8-KV verified coherent past the quantization threshold; peak memory measured.

---

Usage

# requires mlx-lm with the glm_moe_dsa fixes + int8 MLA-KV patch
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw \
  --prompt "Write a quicksort in Python."

Hardware

Built for 512 GB Apple Silicon (M3 Ultra). Weights ~328 GB; with int8 KV a 1M-token context runs in ~376 GB, leaving comfortable headroom for the OS and other apps. For ≤256 GB machines, use the 2.56 bpw build instead.

Credits

• Base model: Zhipu / Z.ai — GLM-5.2 (MIT).
• MLX & mlx-lm: Apple ml-explore.
• Mixed-precision quantization, glm_moe_dsa correctness fixes, int8 MLA-KV cache support, and native-MTP restoration + speculative serving: Alis (avlp12).

Citation

Alis (avlp12) (2026). GLM-5.2-Alis-MLX-Dynamic-3.5bpw — 3.5 bpw MLX quantization of GLM-5.2. https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-3.5bpw