GLM-5.2-VQ-1.9bit

GLM-5.2 — mixed-bit VQ (AQLM) ~1.90 bpw

A 1.90 bits/weight vector-quantized build of GLM-5.2 (744B-parameter reasoning MoE, 78 layers, 256 experts, MIT), tuned for Japanese / English / Chinese general quality and built to serve on 2× RTX PRO 6000 (sm_120) with 16k sparse long context.

Serving code: github.com/mmzz164/vqmoe · Quantizer: OneCompression

What it is

• Experts: mixed 1 / 2 / 3-bit vector quantization (AQLM-style — per-group codes + shared codebook + GPTQ error compensation). Module bit distribution {1: 29456, 2: 12441, 3: 15703}.
• Non-expert spine (attention / router / norm / embed / lm_head): scalar 4/8-bit.
• Size: ~172 GiB. Allocation: loss-aware (output-Fisher × input-energy × quantization error, an Optimal-Brain-Quantization 2nd-order cost) at a 1.90 bpw budget.

Quality (KL to BF16, lower = better; fake-quant on held-out corpora)

Systematic improvement (all 8/8 evaluated sequences) on both a Japanese-reasoning yardstick and a neutral multilingual one — a distribution-wide gain, not a single-domain artifact. Greedy arithmetic eval terminates 22/22 (JA/EN/ZH); needle retrieval verified live.

Serving (sm_120 / consumer Blackwell)

• Backend: vLLM + OneCompression VQ kernels + an sm_120 sparse-attention gather fallback (sm_120 ships no FlashMLA sparse kernel). Tensor-parallel = 2. See vqmoe.
• Context: 16384 (sparse). KV at gpu_util 0.97 ≈ 25,856 tokens.
• Weights: ~87.3 GiB/GPU. Throughput: prefill ~15× vs dense; decode ~13–16 tok/s.
• Defaults: no-think (answers in content; thinking is opt-in per request via chat_template_kwargs {"enable_thinking": true}), with LZ-penalty + 3000-token think-budget forcing as low-bit reasoning safety nets.

Limitations

• Thinking mode does not always self-terminate on casual prompts at this bit-width (an inherent sub-2-bit "no-exit" attractor); the no-think default + budget-forcing contain it.
• Decode ~16 tok/s is an sm_120 MLA single-query ceiling, not tunable here.
• Context > 16k and multi-query speculative decode are unavailable on sm_120 with current kernels.
• Reproducing the exact serving stack additionally needs the sm_120 vLLM patch stack — see the vqmoe repo's "Reproducibility status".

License

MIT. Base model GLM-5.2 (MIT, zai-org); quantizer OneCompression (MIT, Fujitsu Ltd. + VQ extensions © mmzz164). This checkpoint is a quantized derivative.