GLM-5.2-Alis-MLX-Dynamic-2.56bpw

GLM-5.2-Alis-MLX-Dynamic-2.56bpw

Part of the GLM-5.2 · Alis MLX Dynamic collection.

Apple Silicon (MLX) mixed-precision quantization of zai-org/GLM-5.2 — a 744B-parameter (~40B active) Mixture-of-Experts model with DeepSeek-V3.2-style MLA + DeepSeek Sparse Attention (DSA, glm_moe_dsa). Quantized to ~2.56 bits/weight so the full model runs in ≤256 GB of unified memory.

⚠️ Requires a patched mlx-lm with the glm_moe_dsa indexer fixes (see Correctness below). The stock port is incomplete for GLM-5.2; loading there fails or degrades long-context output.

New: the checkpoint now ships GLM-5.2's native MTP (nextn) layer for self-speculative decoding (--mtp — see Native MTP). Fully backward-compatible: runtimes without MTP support strip the extra layer and behave exactly as before.

Update 2 (2026-07): anchor-guarded clip + DWQ re-run. main is now rebuilt from a Q8 source with an anchor-guarded clip search (per-group outlier clipping at max-err slack 1.1 that preserves the min-max anchor weights; method in alis-dwq), then DWQ-retuned with the same recipe as before (4.5 bpw-teacher targets, 45%-ZH calibration, upstream flag mlx-lm#1499). Held-out perplexity improves or holds everywhere vs the previous DWQ main: strided wikitext 3.774 → 3.698 (−2.0%), code 2.069 → 2.054, tulu-3 flat (3.564 → 3.571 ±0.027); the clip alone moves the raw pre-DWQ build 4.34 → 4.17. The checkpoint also sheds 4.5 GB: the MTP layer used to ship twice (a module-named copy inside the base shards plus the standalone MTP shard), and now ships once.

How to read this table: the reference is the 4.5 bpw quantized sibling, not ground truth. The first DWQ was derived from the same 4-bit-lattice family as that reference, so it sits unusually close to it; the clip step re-derives every tensor from a Q8 source (removing the lattice-resonance failure mode), which drifts slightly away from this particular reference even as held-out perplexity — the metric that tracks real quality — improves. Judge builds by the PPL numbers. Greedy loop-probe (256 tok × EN/code/ZH): clean, no cycles.

Weights history (retired 2026-08-28): original quant (pre-dwq) · first DWQ (dwq-noclip) were removed from the Hub. Only main (clip + DWQ) still hosts weights.

Metrics

Why this model

GLM-5.2 is a frontier agentic-coding MoE, but at 744B it is ~1.5 TB in bf16 — out of reach for consumer memory, and existing MLX builds start at ~360 GB (≥4-bit, 512 GB-class machines). This build uses Unsloth-style per-tensor mixed precision: the routed experts (~97% of params) go to 2-bit while the sensitive paths keep higher precision, landing under 256 GB while preserving long-context retrieval and coding quality.

Quality

This is the ≤256 GB option — the routed experts are 2-bit, so it is deliberately bit-starved. If you have a 512 GB machine, the 3.5 bpw build is still better on prose/code strided PPL (−25% wikitext, −11% code; both builds are clip+DWQ retunes, the 3.5 bpw sibling against an 8-bit teacher) and runs a full 1M context.

Strided perplexity from a fixed local harness — relative numbers for comparing these two builds, not directly comparable to perplexities other quantizers report on different corpora.

Benchmarks

Reproduced with mlx_lm.evaluate (0-shot) and mlx_lm.perplexity (seq 2048, 50 samples, seed 123), against the author's earlier GLM-5.1 quant under the same harness and settings:

Note (2026-07-10): the GLM-5.1 · 2.7 bpw build referenced below (avlp12/GLM-5.1-Alis-MLX-Dynamic-2.7bpw) has been removed from the Hub to reclaim storage; its figures are kept here for reference only.

Perplexity here is on allenai/tulu-3-sft-mixture (the mlx_lm.perplexity default) — a different corpus and method from the wikitext strided figure above, so values are not comparable across the two. Task accuracies use a 500-sample limit (CI ±0.02–0.04). All figures in this column are measured on the current main (clip+DWQ); vs the previous retune (dwq-noclip: 0.652 / 0.808 / 0.780) the task deltas are within combined sampling noise (≤1.3σ at this sample size), consistent with the flat tulu. The 3.5 bpw column is that build's current main. GLM-5.1 is a different (older) base model, so cross-generation gaps reflect both the newer model and quantization.

Quantization recipe

Native MTP — self-speculative decoding

GLM-5.2 ships a built-in MTP (multi-token-prediction / "nextn") layer that predicts token t+2 from the backbone's hidden state at t — DeepSeek-V3-style. Public MLX builds (including earlier revisions of this one) stripped it at conversion; this build restores it as model.layers.78.* (one extra shard, +4.51 GB; experts 3-bit g64, attention 4-bit, indexer/norms/eh_proj bf16 — measured on the 3.5 bpw sibling: drafter precision does not affect acceptance, bf16 ≡ 3-bit).

Backward compatible: loaders without MTP support drop layer 78 in sanitize() and behave byte-identically to the previous revision. With the fork, --mtp turns it on:

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --mtp --prompt "…"                        # k=2 chained drafts (default)

mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --mtp --mtp-num-draft-tokens 2            # OpenAI-compatible serving
  # optional: --mtp-hybrid  (adds conservative prompt-lookup drafting
  #            for repetition-heavy workloads: long quotes, JSON, boilerplate)

Reference numbers (M3 Ultra 512 GB, single request, greedy): the 3.5 bpw sibling measures ~performance-neutral (20.4 → 20.7 tok/s, accept-len ~2.3) and a 4-bit sibling measured +11% (22.0 → 24.6, accept-len 2.46) — draft acceptance drops as the target is quantized harder, because a noisier target's own outputs are harder for any drafter to predict. On this 2-bit-expert build expect neutral-at-best single-request speed (not yet benchmarked end-to-end here); the layer stays worth shipping because it is exactly lossless, and gains grow with target precision and multi-node serving.

Lossless: at temperature 0 the output is the model's own greedy output; at temperature > 0 drafts are accepted only when they equal the target's sampled token (distribution-exact).

Long context: MTP works past index_topk (2048 tokens — the DSA-sparse regime). Draft acceptance holds there, and the fork's small-L gather path (commit 946c412) keeps the multi-token verify from paying the sparse-mask setup per iteration (before that fix, long-context MTP decoded at 0.26× plain). Measured on a 4-bit sibling at ~2.1K context: --mtp k=2 ≈ 0.85× plain, k=1 ≈ 0.98× — at very long contexts prefer --mtp-num-draft-tokens 1 or plain; the k=2 gain lives in the ≤2048 dense regime.

Memory note: the MTP-capable fork attaches the module at load (+4.5 GB) even without --mtp, trimming the KV+activation budget by that much. To reclaim it, set "num_nextn_predict_layers": 0 in a local config.json copy (or use a runtime without MTP support, which strips the layer). See Long context & memory for the machine-level ceiling.

Notes for other integrators (verified against vLLM's deepseek_mtp semantics): eh_proj concat order is [enorm(embed(t+1)), hnorm(h_t)]; the hidden fed to the MTP is the backbone's pre-final-norm residual stream; for chained drafting feed back the shared_head.norm-normalized hidden (chaining the raw layer output halves chained acceptance — measured 0.27 → 0.75 conditional at position 2).

Long context & memory

GLM-5.2's MLA stores a compressed latent KV (kv_lora 512 + rope 64 per layer), so the KV cache itself is small — ~88 KB/token at fp16, ~44 KB/token with int8. Quantize it with --kv-bits (the patched fork engages int8 on the MLA latent; stock mlx-lm silently ignores it):

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --kv-bits 8 --kv-group-size 64 --quantized-kv-start 4096 --prompt "…"

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --kv-bits 8 --quantized-kv-start 4096

On a 256 GiB machine the KV is not the binding constraint — the DSA prefill activation is. Measured peak on this build (M3 Ultra, int8 KV, MTP-attached weights ≈ 242 GB):

The DSA lightning indexer scores every past token per prefill chunk, so the activation peak climbs ~+20 GB per ~30K tokens — independent of KV bits (int8 vs fp16 differed by only ~1 GB at 26K, because the KV is tiny there). So on a 256 GiB box, keep prompts ≤ ~26–32K. --kv-bits 8 still helps where the KV is what grows — long decode/accumulated context, and 512 GiB machines with headroom for longer prefills. For genuinely long (100K–1M) context, use the 3.5 bpw build on a 512 GiB machine.

Correctness (verified vs the HF reference)

GLM-5.2's glm_moe_dsa needed fixes beyond the stock mlx-lm port; this build was produced with a patched fork and validated:

• IndexShare — the DSA indexer runs only on "full" layers; "shared" layers reuse its top-k (index_topk_freq=4). The stock port built an indexer on every layer → missing-weights / wrong >2048-token output.
• Indexer RoPE/eps — the indexer uses non-interleaved (half-split) RoPE + LayerNorm eps 1e-6, distinct from the interleaved main attention. Post-RoPE q matches the HF reference to ~1e-7. Recorded in config.json (indexer_rope_traditional=false, indexer_norm_eps=1e-6).
• Native MTP — layer 78 restored + --mtp self-speculative decoding in the fork; greedy output verified identical to non-speculative decoding (on the 3.5 bpw sibling), chained drafting matches vLLM's deepseek_mtp semantics (normed-hidden chaining).

Validation: full-attention logits match the HF reference to float precision at ≤index_topk context; needle retrieval succeeds through a 7,586-token prompt (sparse-DSA regime); coherent code generation; peak ≤256 GB.

Usage

# requires mlx-lm with the glm_moe_dsa indexer fixes
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw \
  --prompt "Write a quicksort in Python."

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw

# with native MTP speculative decoding (see the Native MTP section)
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw --mtp

Hardware

Runs in ≤256 GB unified memory (Apple Silicon). A "256 GB" Mac is 256 GiB = 274.9 GB (not 256 decimal); the ~242 GB of (MTP-attached) weights leave ~33 GB. But the DSA sparse-attention prefill activation grows with context (measured +~20 GB at 26K, +~50 GB at 64K), so the practical prefill ceiling on a 256 GiB machine is ~26–32K tokens (peak ~263–268 GB) — not the architecture's 1M. For 100K+ context use a 512 GB (512 GiB) machine. See Long context & memory below.

Credits

• Base model: Zhipu / Z.ai — GLM-5.2 (MIT).
• MLX & mlx-lm: Apple ml-explore.
• Mixed-precision quantization, glm_moe_dsa correctness fixes, and native-MTP restoration + speculative serving: Alis (avlp12).

Citation

Alis (avlp12) (2026). GLM-5.2-Alis-MLX-Dynamic-2.56bpw — 2.56 bpw MLX quantization of GLM-5.2 for ≤256 GB Apple Silicon. https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-2.56bpw