GLM-5.2-Alis-MLX-Dynamic-2.3bpw

GLM-5.2-Alis-MLX-Dynamic-2.3bpw · ⚠️ experimental floor build

Part of the GLM-5.2 · Alis MLX Dynamic collection.

Apple Silicon (MLX) mixed-precision quantization of zai-org/GLM-5.2 — a 744B-parameter (~40B active) Mixture-of-Experts model with DeepSeek-V3.2-style MLA + DeepSeek Sparse Attention (DSA, glm_moe_dsa) — pushed to the lowest effective bit-rate the MLX affine container supports (routed experts at 2-bit/g128): 2.3225 bpw measured, 219.75 GB on disk. It is the smallest published GLM-5.2 build in any format we know of, and the first in this family produced with anchor-guarded clip-search + layerwise DWQ stacked.

⚠️ Experimental. This is a floor experiment that ended up genuinely usable. On the same ≤256 GB machine class the 2.56 bpw sibling is still somewhat better (−5.2% wikitext PPL on its own clip+DWQ rework) at +23 GB; pick this build when the extra ~23 GB of headroom matters (longer prefills / KV on a 256 GiB box). Full methodology, the teacher A/B, and the clip-search saga: alis-dwq E1 floor-spike case study (parts 1 & 2).

⚠️ Requires a patched mlx-lm with the glm_moe_dsa indexer fixes (see Correctness). The stock port is incomplete for GLM-5.2; loading there fails or degrades long-context output.

The checkpoint ships GLM-5.2's native MTP (nextn) layer for self-speculative decoding (--mtp — see Native MTP). Backward-compatible: runtimes without MTP support strip the extra layer.

How the weights were made (two orthogonal passes, both chosen by measured A/B):

1. Anchor-guarded clip-search requantization (alis-dwq clip_quantize): per 128-weight group, candidate clipped grids are accepted only when they lower reconstruction MSE without raising the group's max abs error beyond 1.1× the min-max grid's — preserving the "super weights" that min-max anchors exactly. Source: the public Q8 checkpoint (quasi-continuous; a dequantized nvfp4 sibling as source kills the model via correlated rounding — measured, see the case study). Raw wikitext 4.7109 → 4.4244.
2. Layerwise DWQ against the 4.5 bpw sibling as teacher (an 8-bit teacher measurably loses at this student capacity — teacher-capacity sweet spot), 45%-ZH calibration, K=6 × 13 rounds with per-round held-out rollback. Wikitext 4.4244 → 3.8985.

Metrics

Why this build exists

1. Find the MLX floor. 2-bit is the container's minimum and group size tops out at 128 → 2.25 bpw effective is the hard floor for the expert bulk. This build tests whether that floor is usable — it is, with the right passes on top.
2. Method A/Bs on a real 745B student. This repo's family now carries three paired experiments: DWQ teacher precision (4.5 beats 8-bit here), clip acceptance rules (anchor-guarded beats MSE-only, which is fatal), and requant source class (quasi-continuous beats lattice, which is fatal). All documented with numbers in the case study.

Market context: the most aggressive GLM-5.2 quant elsewhere (llama.cpp dynamic "1-bit" UD-IQ1_S) ships at 223 GB ≈ 2.4 bpw effective — every stack converges to this size class on this model. This build undercuts it with measurably recovered quality.

Quality

Strided perplexity from a fixed local harness — relative numbers for comparing these builds, not directly comparable to other quantizers' corpora. Sibling columns (and the chart) show each build's current main — both are clip+DWQ retunes as of 2026-07.

CI ±0.02. At 2.32 bpw the task suite sits within ~1 CI of both larger siblings — the bit-rate cost shows up in perplexity, not in these accuracies. (Sibling columns are their current clip+DWQ mains.)

Degeneration probe (greedy 256 tokens/slice, alis-dwq --loop-probe): distinct-4gram EN 0.929 · code 0.901 · ZH 0.984, no cycles detected on any slice — aggregate-score parity can hide loop degeneration (it doubled in a published REAP prune at eval parity), so this build gates on it explicitly.

vs the llama.cpp dynamic quants (same metric class)

Unsloth's GLM-5.2 dynamic GGUFs publish quality as top-1 token agreement against a BF16-or-Q8_0 baseline on a sampled token set. Measuring this build with the same metric (baseline = the Q8_0 checkpoint's stored top-1024 logits; 73,173 tokens across 177 seed-7 calibration samples):

Smaller than their 1-bit and above its agreement score — while our token set is plausibly harder (45% Chinese, and ZH is the weakest slice at this bit-rate for every build we've measured). Caveat for rigor: same metric class and a baseline within their stated definition ("BF16 or Q8_0"), but different token sets — theirs is a training-corpus subset, ours the calibration mix above — so treat this as indicative, not a shared benchmark. Their 2-bit at +19 GB still leads on this metric, consistent with our own PPL ladder.

Teacher A/B — why the 4.5 bpw teacher ships and the 8-bit one doesn't

Both DWQ arms trained from the same raw student, identical hyperparameters; only the teacher-logit dump differs. Pre-registered prediction: the 4.5 bpw teacher wins at this capacity. It did — on every held-out metric (wikitext +1.25%, code +0.82%, tulu +0.74% for the 8-bit arm), while the 8-bit arm's training valid loss dropped more — the overfit-toward-teacher signature. Rule: judge DWQ teachers on held-out PPL, never on training/valid loss. (Measured on the pre-clip student; the shipped build re-ran the winning arm on the clipped student.)

Quantization recipe

Native MTP — self-speculative decoding

GLM-5.2 ships a built-in MTP ("nextn") layer predicting token t+2; this build restores it as model.layers.78.* (one extra shard, +4.51 GB). Provenance note: the Q8 source ships no MTP tensors, so the attached layer is the 3-bit-expert sidecar inherited from the 2.56 build's lineage — fine for plain decode (loaders without MTP support drop it in sanitize()), but --mtp speculative gains are unvalidated on this build; expect neutral-at-best single-request speed (acceptance falls as the target quantizes harder). It ships because MTP is exactly lossless.

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.3bpw \
  --mtp --prompt "…"                        # k=2 chained drafts (default)

mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.3bpw \
  --mtp --mtp-num-draft-tokens 1            # long-context-heavy serving: prefer k=1

See the 2.56 card for full MTP long-context notes and integrator details.

Long context & memory

MLA stores a compressed latent KV (~88 KB/token fp16, ~44 KB/token int8; --kv-bits 8 — the patched fork engages int8 on the MLA latent, stock mlx-lm silently ignores it). As on the siblings, the DSA prefill activation — not the KV — binds on a 256 GiB box (~+20 GB per ~30K prompt tokens).

Context-dependent peaks have not been re-measured on this build. Weights-resident peak measured ~215.4 GB — ~23 GB more headroom than the 2.56 build, whose measured curve (249 @8K → 268 @32K → 293 @64K) puts this build's practical prefill ceiling at ~32–40K tokens on a 256 GiB (274.9 GB) machine — an estimate by transposition, not a measurement. For genuinely long context use the 3.5 bpw build on a 512 GiB machine.

Correctness

Produced with the same patched fork as the siblings (glm_moe_dsa fixes: IndexShare top-k reuse; indexer non-interleaved RoPE + eps 1e-6, post-RoPE q matching the HF reference to ~1e-7). Build-specific verification: bit-plan audit (exactly 225 routed-expert modules at 2-bit/g128, router unquantized bf16, tokenizer byte-identical to source, sha 19e77364…), pre-DWQ quality gate, KL-harness consistency check, per-round DWQ rollback gates (12 accepted, 1 late revert — the natural stopping signal), and the degeneration probe above.

Honest caveats:

1. ZH remains the weakest slice (KL 0.686 vs EN 0.446 after both passes) — the usual low-bit pattern, substantially recovered (raw was 1.154) but not erased.
2. Instruction-following at this bit-rate: pre-clip raw sanity checks showed occasional English analysis scaffolding instead of the requested output language/format. The shipped build passes the greedy degeneration probe cleanly (above), but a full generation-quality pass is still pending.
3. From-Q8 provenance (see Metrics) is a disclosed variable vs the from-bf16 siblings — Q8's noise floor is ~38 dB below the 2-bit noise it feeds.

Usage

# requires mlx-lm with the glm_moe_dsa indexer fixes
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.3bpw \
  --prompt "Write a quicksort in Python."

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.3bpw

# long context: int8 KV
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-2.3bpw \
  --kv-bits 8 --kv-group-size 64 --quantized-kv-start 4096 --prompt "…"

Generation speed: ~23.3 tok/s single-request greedy (M3 Ultra 512 GB, short prompt, measured on the same-format raw build; clip+DWQ change codes/scales only, not the decode format).

Hardware

Runs in ≤256 GB unified memory (Apple Silicon) with more headroom than any other GLM-5.2 build: a "256 GB" Mac is 256 GiB = 274.9 GB; ~220 GB of MTP-attached weights leave ~55 GB for KV + DSA prefill activation (vs ~33 GB on the 2.56 build).

Credits

• Base model: Zhipu / Z.ai — GLM-5.2 (MIT).
• MLX & mlx-lm: Apple ml-explore.
• Q8 source checkpoint: pipenetwork. Clip-search inspiration: four-over-six (Cook et al., via the humans& NVFP4 RL recipe).
• Mixed-precision recipe, glm_moe_dsa fixes, anchor-guarded clip-search + layerwise DWQ (alis-dwq), teacher/source/acceptance A/Bs, native-MTP restoration: Alis (avlp12).

Citation

Alis (avlp12) (2026). GLM-5.2-Alis-MLX-Dynamic-2.3bpw — experimental 2.32 bpw MLX floor build of GLM-5.2 with anchor-guarded clip-search + layerwise DWQ. https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-2.3bpw