GLM-5.2-EXL3-TR3-MTP78

GLM-5.2 MTP Layer-78 — EXL3 Trellis overlays (2/3/4 bpw × keep64/keep0)

Quantized MTP draft layer (layer 78) overlays for GLM-5.2-EXL3-TR3-3.0bpw, produced with Brandon Music's encode_tr3_v31.py (LDLQ-calibrated trellis, MCG codebook 0xCBAC1FED, exllamav3 v0.0.43 kernels, TP4 rank-sliced tensors) from a full-corpus calibration capture: 7,288,310 tokens of layer-78 MoE inputs + routed top-8 expert ids, captured during live serving over the complete 12,228-row / 4-axis owner corpus (reap_recall_calib.jsonl, sha256 cf247acc…). Capture dataset published separately: malaiwah/GLM-5.2-MTP78-calibration-capture.

Variants

* keep64 variants protect the 64 highest trellis-roundtrip-error experts in BF16 — but the current rank-sliced EXL3 runtime has no mixed BF16+trellis expert path (fails with KeyError …w2_weight); the base checkpoint's own convention is keep0 (nvfp4_keep_per_layer: 0). Published for future loaders.

Validation (3bpw-keep0)

Identical 20-prompt greedy bench (300 tok/req), vLLM 0.17.0rc1 RC2 image, 4× RTX PRO 6000 Blackwell, TP4, MTP-3:

Draft VRAM at TP4: BF16 ≈ 4.8 GB/GPU → 3bpw-keep0 ≈ 0.95 GB/GPU — the reclaimed ~3.8 GB/GPU goes straight to KV cache.

Deploying (graft onto GLM-5.2-EXL3-TR3-3.0bpw)

1. Replace model-layer-078.safetensors with a merged shard: the 23 non-expert layer-78 tensors (BF16, from the original shard) + all tensors from 3bpw-keep0/tr3-layer-078.safetensors; rebuild model.safetensors.index.json accordingly (drop the 768 old expert .weight entries).
2. config.json: set hybrid_tr3_tail.moe_layers to [3, 78] and remove the model.layers.78* entry from quantization_config.ignore.
3. vLLM RC2 needs a 1-hunk patch: deepseek_mtp.py load_weights must apply quant_config.normalize_rank_sliced_weight_name() like deepseek_v2.load_weights already does (without it: KeyError …routed_experts.w2_rank0.mcg). Patch in tools/.
4. Config sensitivity on the RC2 image: the trellis draft validated on the cudagraph/DCP4 serving config; the BF16 draft conversely crashes there and needs DCP1/eager. Pick per draft type.

tools/ contains the graft builder, the vLLM patch, the capture hook, the bench driver, and per-variant layer-078.done.json (full per-expert routed counts + rel-RT-mse from the encode).

Credits

zai-org (GLM-5.2 base, MIT) · Brandon Music (EXL3 TR3 encoder + owner corpus + base checkpoint) · Josh Cartu / jcartu (MTP78 recipe, rank-sliced MTP runtime) · Luke Alonso (b12x) · turboderp-org (exllamav3, MIT) · Martin Vit + @yatesdr (RC2 image). Capture/encode/validation: malaiwah, 2026-07-24/25, 4× RTX PRO 6000 rental.