GLM-5.2-EXL3-TR3-3.40bpw-KVarN-K4V2

GLM-5.2-EXL3-TR3-3.40bpw-KVarN-K4V2

GLM-5.2 756B MoE serving checkpoint for four 96 GB RTX PRO 6000 Blackwell GPUs: EXL3 ~3.40 bpw mixed-precision routed experts ("TR3") with the KVarN K4V2 MLA KV cache, served on a local-inference-lab/vllm fork with the b12x sparse MLA backend and MTP3 speculative decoding.

Inspired by the community EXL3 lineage for this hardware — brandonmusic/GLM-5.2-EXL3-TR3v4-3.5bpw-MTP78 and willfalco/GLM-5.2-EXL3-TR3-3.42bpw — this checkpoint pushes the other side of the trade: the lowest directly-measured KLD published for GLM-5.2 EXL3 at any width, with more usable context than either.

What is different here

• KVarN K4V2 KV cache — 290 B/token/layer latent-tile records with a self-describing dtype (kvarn_mla_k2_g64). Smaller than compact NVFP4 MLA KV (368 B) and roughly 2.3x smaller than FP8 (656 B), at higher measured fidelity than both on this model.
• b12x (B12X_MLA_SPARSE) backend — sparse MLA attention with the paged top-k indexer, CKV gather prefill, and direct-packed DCP4 decode; deterministic top-k merge at zero measured cost.
• local-inference-lab/vllm fork — the runtime base this stack is developed against; the serving image bakes the fork plus b12x, KVarN, and sparkinfer kernels into one bootable unit.

Checkpoint design: the MCG promotion ladder

The TR3 allocation was built in one evidence-driven pass, not uniform bits:

1. Full MCG3 base quantization — routed experts quantized to MCG3 using captures from the BF16 model on our own corpus.
2. Activation-aware promotion — candidates for promotion were selected by measured activation alignment on coding, reasoning, and agentic workloads (the domains this checkpoint is built for).
3. MCG4 as the promotion ceiling — aligned candidates were promoted MCG3 -> MCG4; nothing went higher because no measurement justified it.
4. Width milestones 3.00 -> 3.25 -> 3.38 -> 3.40 bpw — each step was validated, and 3.40 was the last balance point: pushing further would have forced cutting context capacity below the serving contract.
5. KVarN was a from-the-start design focus, not a retrofit — the KV format's size/quality potential shaped the whole budget, and the measured payoff (290 B/token at KLD 0.0558) is what makes 512K context coexist with the lowest published KLD.

Positioning vs the community checkpoints

brandonmusic's 3.5 bpw R7 optimizes for general-quality evidence with a broader (including legal-domain) emphasis. This checkpoint deliberately skews the other way: the promotion ladder weighted coding, reasoning, and agentic activation evidence, and the width stopped at 3.40 bpw to protect context capacity. willfalco's 3.42 bpw is the closest sibling in spirit (coding/agentic first); the difference here is KVarN KV at 290 B/token — smaller and more faithful than the KV formats in either recipe — which is what buys the 512K context alongside the quality.

Correction & retraction: KLD claim (2026-08-19)

We retract the claim that this checkpoint has "the lowest directly-measured KLD published for GLM-5.2 EXL3 at any width." That claim was measured on our calibration-domain panel (item13, the same corpus family the quantization allocation was tuned on). On held-out data with the community methodology, the claim does not hold. Credit to brandonmusic and willfalco, whose public held-out measurements and reference logits (festr2/GLM-5.2-BF16-KLD-Reference-Logits-20260708) prompted and enabled this correction.

Held-out measurement (wikitext-2 test window, 2,048 tokens, festr2 BF16 reference logits, fp8-KV reference per their methodology, same 3.40 bpw checkpoint varying only KV format):

For comparison on the same held-out reference (matched fp8-KV configs): brandonmusic's 3.5 bpw TR3v4 measures 0.0654 ± 0.0021, and willfalco's 3.36 bpw EXL3 measures 0.0685 — both closer to BF16 than our 3.40 bpw checkpoint. Since a narrower quant (3.36) outperforms ours, the gap is attributable to allocation, not width: our promotion scheme (activation-weighted on a coding/reasoning/agentic calibration corpus) appears overfit to that corpus — 0.0555 on our calibration panel vs 0.0871 held-out at identical config. The calibration-domain numbers remain correct measurements of in-distribution fidelity; they are not comparable to held-out numbers and should not be quoted as general quality.

What still stands:

• KV economics: K4V2 costs +0.0016 over FP8; the community's Dynamic NVFP4 costs +0.0344 on the same weights — 20x the format charge. At 290 B/token vs 368, this is a smaller cache that is also more faithful.
• Context: 819K proven (vs ~347K for nvfp4-KV deployments at this hardware class).
• The item13 panel, plan SHA 072e37f..., and per-row token_ids_sha256 values are published for full reproduction of our calibration-domain numbers.

We intend to re-measure and, if a re-allocation campaign succeeds, re-enter the held-out comparison on these terms.

---

Quality evidence (calibration-domain — in-distribution; see the retraction above before quoting these numbers)

Direct BF16-reference KLD, full corpus item13 captures, 2,047 positions:

For context only (different evaluators and procedures — not a leaderboard): brandonmusic 3.5 bpw R7 reported 0.0613 +/- 0.0014; the standardized 3.42 bpw matrix reported 0.0741 with FP8 KV and 0.1080 with NVFP4 MLA KV. The KVarN numbers above were measured with the procedure in -assets/kld/ (capture plan, reference, and reports are archived for reproduction).

Long-context: estonia-long 29/30 answers at 133K-token prompts.

Measured serving profile

TP4/DCP4/MTP3, one PCIe switch, 290 W locks, greedy:

KV capacity field observations: 512K at 2000 blocks/GPU with the c1-focused profile; the c4-capable 256K pool trades context for the c1/c2/c4 = 86/104/168 tok/s grid. DCP2 at 360K context is enabled and validated at champion speeds (decode 89, prefill 2,433); DCP1 is unsupported for KVarN and fails closed at config time.

Quick start

Runtime image (public): jmpsequeira/glm52-kvarn-k4v2:tr3 (all code baked — b12x, KVarN reader, sparkinfer, EXL3 loader).

IMPORTANT — re-pull the image if you downloaded it before 2026-08-19. Earlier pulls contain two prefix-caching bugs (KVarN KV + --enable-prefix-caching): garbage output on cache hits and persistent server-wide corruption after long requests. Fixed in tr3.2 (digest df889e27461a), which is also the current tr3. docker run does NOT auto-update a cached tag — run docker pull jmpsequeira/glm52-kvarn-k4v2:tr3 (delta download, seconds) or pin tr3.2. With the fix, prefix caching is safe and recommended; quality is unchanged (KLD 0.0555 vs 0.0558 of-record).

---

RESOLVED (2026-08-22) — Root cause found: not corruption. The 2-bit KV format (kvarn_mla_k2_g64) is below the generation-coherence floor. Every store/read kernel was verified mathematically sound (pack/rehydrate round-trip matches an independent reference exactly; the native SM120 decode reader agrees with record bytes to 0.3%; pool latents confirmed Hadamard-gaussianized; ownership/mapping verified valid at 55K probed decode steps). The 2-bit latent grid carries an inherent ~45% relative quantization error (measured; matches pure-RTN theory) — sufficient for prompt-side KL panels (short contexts are served from the FP8 exact pool and never touch packed rows — the 0.0558 KLD was an evaluation blind spot), but beyond what generation tolerates whenever quantized rows are attended: hot-window packing (PRECISION_TAIL_TOKENS=0) garbles immediately; old-row selection at larger tails garbles sporadically under history pressure. 4-bit (kvarn_mla_k4_g64, ~9% error) is clean in identical batteries when the hot window stays exact.

Layer-wise mixed precision — RETRACTED for serving (2026-08-23): the runtime supports demoting layers to the 2-bit record (KVARN_MLA_K2_LAYERS), and late layers (40-77) passed every standard gate (generation battery, tool calls, 30-turn soak, estonia-long 10/10 at 133K, cache-eviction churn). It is nonetheless too unstable for production: prefix-cache rehydration corrupts through the early layers' degenerate subnormal latent tiles, and extreme contexts (~700K) poison the serving boot outright. Until the format gains a story for those regimes, serve pure kvarn_mla_k4_g64 with KVARN_MLA_PRECISION_TAIL_TOKENS=3072.

k2-mix UPDATE (tr3.4, 2026-08-26): the retraction's churn-corruption leg is disproven — it was the general records-ptr provenance wipe fixed in tr3.4. Layers 40-77 demoted to kvarn_mla_k2_g64 on a k4 base pass the full churn soak post-fix (pressures, re-entries at full APC hit, exact needle retrieval). The k2-mix (~356 B/token, ~15% KV savings) is viable again for contexts ≤512K. The retraction now rests on a single remaining defect: ~700K contexts poison the serving boot (CKV lease issue, unfixed). The 2-bit record remains useful only for short-context, read-mostly workloads.

RESOLVED in tr3.4 (2026-08-26, digest f65645545938) — general cache-eviction corruption: the probabilistic, state-accumulating KV corruption under multi-100K eviction churn (all KV dtypes, worst under 2-bit mixes) was traced to a worker-side bookkeeping defect: a single shared records-pointer slot was alternated by per-layer rebinds and misread as record-storage reallocation, wiping flush provenance every rebind cycle and silently disabling record rehydrate on prefix-cache hits after eviction. Fixed by per-impl pointer keying; verified by a deterministic double-eviction reproducer and a 4-round churn soak (coherent re-entries at full 118K-token APC hit, exact needle retrieval). docker pull jmpsequeira/glm52-kvarn-k4v2:tr3.4 (also :latest). Earlier tags (tr3.3 and below) still carry the defect — the churn caveat applies to them only. This also re-attributes the 2-bit retraction's "rehydration corrupts through subnormal latent tiles" mechanism: that corruption was this bug; the 2-bit retraction stands on its remaining evidence (~700K boot poisoning, degraded gates).

tr3.5 (2026-08-26, digest 61ab86886a9c): fp8-RoPE record cell — opt-in. The KVarN packed record's RoPE rows move from BF16 to E4M3 + per-token fp16 amax (the #95 cell design): 418 → 356 B/token (−14.8%), ~15% more context per KV pool. Enable with KVARN_FP8_ROPE_RECORD=1. Validated end-to-end on the churn battery at 327680 ctx (pressures, 1s full-hit re-entries through fp8 records, exact needle retrieval, canary). Default OFF — tr3.5 without the flag is byte-identical serving behavior to tr3.4.

Serve with --kv-cache-dtype kvarn_mla_k4_g64 and KVARN_MLA_PRECISION_TAIL_TOKENS=3072 (the launcher defaults). The KVarN KLD table's 2-bit rows are valid only for short-context workloads. Separately filed: an MTP accepted-token-resolution race (fail-closed crash, not corruption), exposed under heavy per-step instrumentation — tracked in the runtime repo. root cause and regression suite once landed).

docker run --rm --name glm52-kvarn \
  --gpus '"device=0,1,2,3"' --network host --ipc host \
  --ulimit memlock=-1:-1 --ulimit stack=67108864:67108864 \
  -v ~/.cache/huggingface:/hf-cache:ro \
  -v "$PWD/vllm-cache:/cache" \
  jmpsequeira/glm52-kvarn-k4v2:tr3 \
  --model jpsequeira/GLM-5.2-EXL3-TR3-3.40bpw-KVarN-K4V2 \
  ... (vllm serve flags per profile below)

The model is sourced through the Hugging Face cache: mount your ~/.cache/huggingface read-only and pass the repo id as --model (HF_HUB_CACHE=/hf-cache/hub, offline resolution — no download if the snapshot is cached, automatic fetch if not). The MTP head is layer 78, merged natively into this checkpoint — no draft dir, no spec "model" key; the loader sources model.layers.78.* from the main index like every other layer (EXL3 MCG3 3.0 bpw experts in projection-mixed-layer-078, shared-H rotations in shared-h-layer-078, BF16-protected modules in bf16-protected-layer-078). VLLM_EXL3_TRELLIS_MIN_M=1 is mandatory. Measured zero acceptance cost (AL 2.85/2.84 in-band vs the NVFP4 draft).

Core profile env/flags (the full validated contract ships in the -src repo's boot-from-image.sh):

• TP=4, DCP=4, MTP=3, MAX_NUM_SEQS=1, MAX_BATCHED_TOKENS=2048
• MAX_MODEL_LEN=512000, NUM_GPU_BLOCKS=2000, GPU_MEMORY_UTILIZATION=0.984
• attention backend B12X_MLA_SPARSE, kv_cache_dtype kvarn_mla_k2_g64
• cudagraph sizes 4,1; chunked prefill; no prefix caching

Keep /cache persistent — first boot JIT-compiles and later boots reuse it.

KV format ladder (KVarN)

K4V2 is the right-sized default: K4 buys zero measured KLD at +44% bytes. The exact-precision tail option was swept and measured neutral (speed and quality); the default is zero retained tail.

Provenance

• Checkpoint: EXL3 ~3.40 bpw mixed-precision allocation assembled from per-projection cached-local-BF16-output recovery (expert_precision_map.json, projection-assembly.json in this folder). Base snapshot revision: b4734de4facf877f85769a911abafc5283eab3d9 (zai-org/GLM-5.2).
• Runtime: local-inference-lab/vllm fork + b12x backend + KVarN + sparkinfer. Full source: github.com/JMPSequeira/glm52-kvarn-k4v2-runtime (the Docker image builds directly from that repo via its Containerfile; MANIFEST.md there maps every file to its upstream for PRs to local-inference-lab/vllm, b12x, and sparkinfer).
• History: HISTORY.md in this folder (design decisions, KLDs, throughputs, rejected alternatives).
• Measurement protocol notes: 5-minute post-boot settle; report prefill run index (first run of a pair reads ~3% fast from self-heating); same-session controls for prefill A/Bs.

Credits and references

Built on the open GLM-5.2 EXL3 serving community — the 3.5 bpw R7 and 3.42 bpw TR3 checkpoints and their deployment guides (brandonmusic, willfalco) defined the evidence standard this work follows.

• KVarN — the variance-normalized KV-cache quantization this checkpoint's cache format builds on: paper arXiv:2606.03458, code huawei-csl/KVarN (native vLLM KV-cache backend).
• b12x / sparkinfer — the sparse MLA backend and SM120 CuTe DSL kernel library serving this model (sparkinfer is b12x's current name).
• local-inference-lab/vllm — the vLLM fork this runtime develops against.
• Runtime source — JMPSequeira/glm52-kvarn-k4v2-runtime; image jmpsequeira/glm52-kvarn-k4v2:tr3 builds directly from it.