GLM-5.3-Vision-NVFP4-AQLM-hybrid-1m

GLM-5.3-Vision-NVFP4-AQLM-hybrid-1m

The 1M-context variant of jarrelscy/GLM-5.3-Vision-NVFP4-AQLM-hybrid: same two-tier per-expert recipe, re-tiered with a smaller hot set so the checkpoint (272.7 GiB, 14.8 GiB lighter) leaves enough VRAM headroom on 4× B200 (TP4, MTP, DCP4) to serve the full 1M context.

This revision (v3) re-tiers and re-calibrates with a multimodal-blended calibration set: expert selection scores are 0.75 × text REAP + 0.25 × vision-token salience (per-layer normalized), and the cold-expert Hessians mix text and vision activations 75/25. Rationale: vision tokens route through substantially different experts than text (per-layer salience correlation ≈ 0.10 with the text calibration set); the previous text-only hot set covered only 38% of vision-token salience. The blend lifts that to 70% while giving back ~3% of text salience coverage.

• Hot experts (5750 of 19,200 = 30%): donor NVFP4 (4.5 bpw effective), chosen per layer by the blended score above (floor 8 / cap 176 per layer; hot/layer 45–99). Text REAP = routing-weighted expert salience (Σ gate·‖f_e(x)‖) × per-expert 2-bit quantization error over a 15M-token calibration set; vision salience from 1,200 image+caption samples (medical imaging, natural images, OCR/documents, screenshots) spliced through the 5.2V encoder.
• Cold experts: 2 bpw AQLM (1×16 codebook, group 8), k-means init + activation-aware CD re-encode against the donor teacher (diag-Hessian over the 75/25 text/vision activation mix), then PV-tuned: codebooks + scales trained by gradient descent on cold MoE output MSE vs the donor (codes frozen, held-out val split, best-checkpoint revert).
• MTP layer 78: all 256 experts requantized BF16→NVFP4 (rel-frob ≈ 0.095); input_scale synthesized from the 5.2 L77→L78 ratio (validate MTP acceptance rate when serving).
• Non-expert weights: donor NVFP4/BF16 passthrough. Total 272.7 GiB.

Vision

The GLM-5.2V vision tower + projector (from baseten/GLM-5.2-Vision-NVFP4, byte-identical) are grafted on: vision_tower.safetensors + mm_projector.safetensors, glm5v config, Kimi-K2.5-style processor. The encoder was trained against the frozen 5.2 text tower; 5.2→5.3 drift is small (mean embedding row-cos 0.992). An A100 probe (projected image tokens spliced at <|image|>, teacher-forced caption CE through this hybrid) shows correct caption preference both ways with wide margins. Fine-grained VQA/OCR quality vs 5.2V has not yet been benchmarked.

Quality

Teacher-forced ppl, non-overlapping 2048-token windows, A100 bf16-dequant:

Multimodal caption ppl (teacher-forced caption CE with projected image tokens spliced at <|image|>, 1,200 held-in calibration samples):

The mm blend buys 6.5% on OCR and 2.3% on screenshots (screenshots end up better than the donor) for ~0.5–1.7% text give-back in-domain; medical/natural shifts are noise-level. The wikitext gap (a documented bias of the code/technical-heavy text calibration set) also improves slightly. In-domain (code/agentic/technical) quality is within ~2–4% of the full NVFP4 donor.

Terminal-Bench 2.1

78/86 = 90.7% pass rate excluding timeouts (89 tasks: 78 pass, 8 fail, 3 timeouts).

• Harbor 0.21.0, terminus-2 agent, default sampling, served by vLLM at TP4 on 4× RTX PRO 6000.
• Fails: build-pov-ray, extract-elf, gcode-to-text, mteb-leaderboard, protein-assembly, qemu-startup, raman-fitting, video-processing.
• Timeouts: extract-moves-from-video, filter-js-from-html, make-doom-for-mips.
• Timed-out and interrupted tasks were rerun; each task counts its best completed attempt.

Agent trajectories, verifier output and result.json for every trial are in benchmarks/tb2.1/.

Serving

Same stack as the 5.2 hybrid: an sglang/vLLM build with the nvfp4_aqlm_hybrid quant method (per-layer hyb_kind routes each expert to the NVFP4 or AQLM dequant path) and the glm5v wrapper (MoonViT tower + PatchMerger projector + GlmMoeDsa text backbone). On 4× B200 the freed ~3.7 GiB/GPU of weights covers the 12.11 GiB/GPU KV requirement of the 1M-token config (util 0.97, MTP on, DCP4). Dequant checks: hot experts bit-exact vs donor NVFP4; cold codes bit-exact vs the PV output (SC-2 max|d| = 0).

4× RTX PRO 6000 (SM120)

The Terminal-Bench results above were served from jarrelscy/vllm-glm52-sm120 (main, image built from Dockerfile.glm52-sm120) at TP4 + DCP4 + MTP with a 950K context window, ~84 tok/s single-stream code decode. Build and run steps, and the performance flags, are in the fork's GLM-5.3 serving guide.

Build code

code/ has the recovered 5.3 build scripts, the run plan and log, and this checkpoint's hot/cold expert assignment. The vision-blended tiering and PV tuning scripts were not recovered; code/README.md lists what is missing.