GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision

GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision

⚠️ Usable — read the hazards. Loads ONLY through the custom serving image below (in-house NF3 3-bit kernel + a small vision plugin — no official vLLM/inference stack runs it). Serves on 4× 96GB sm120 GPUs with a 509,467-token FP8 KV pool @ 250k max context at DCP4 (measured live 2026-07-23) — 2.04× concurrency at full length. NOT the official model's 1M context, and not officially supported.

The full GLM-5.2 — 753B, all 256 experts, no pruning — with eyes, on 4× 96GB GPUs.

This is madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid (v3.6 text checkpoint, carried over byte-exact) with the GLM-5.2-Vision tower and projector grafted on from baseten/GLM-5.2-Vision-NVFP4 (revision f6eab611, ≈0.93 GB of vision payload: MoonViT-3d tower + multimodal projector, served through vLLM's Kimi-K2.5 vision stack via the bundled plugin).

The text tier is exactly the hybrid recipe from the base card:

• NVFP4 for the 64 damage-ranked experts per layer (byte-exact from lukealonso/GLM-5.2-NVFP4 — credit to Luke for the best-in-class low-KLD 4-bit tier);
• NF3 — madeby561's in-house 3-bit expert format with a matching custom sm120 kernel — for the long tail (192 experts/layer);
• non-expert layers stored true BF16, served as MXFP8 (single BF16→mxfp8 hop at load), which is what buys the large KV pool.

≈342 GB total (184 × ≤2 GB shards + vision payload). The only GLM-5.2 with vision that fits 4 cards with a 500k-class KV pool.

How this was made: the full graft-and-serve reproduction guide is in REPRODUCING.md, with the checkpoint-assembly scripts in graft/.

Accuracy

Text tier: identical weights to the base model, so its measured numbers carry over unchanged — GPQA-Diamond 88.89 (vs 89.39 full NVFP4 on 8 GPUs, vs 86.87 for the REAP-594B prune), top-1 agreement ≈99.98% vs FP8, KLD 0.126 in the community vs-BF16 frame (Luke NVFP4 anchor: 0.060). Full tables, protocol, and the vs-FP8 / vs-BF16 frame warnings are on the base model card — they were measured on the text checkpoint this repo carries, not re-run here.

Vision tier: the tower/projector ride NVFP4 exactly as shipped by baseten; no additional quantization was applied here. Validation so far is smoke-level (text generation TEXT_OK, image captioning verified on a COCO bus photo, clean post-start error scan). No vision benchmark suite has been run on this artifact yet — treat vision quality as inherited-but-unverified.

Serving

Everything is baked into one image — NF3 kernel, hybrid loader, native mxfp8 tier, and the glm52-vision-vllm plugin that registers the Glm5vForConditionalGeneration architecture:

ghcr.io/0xsero/glm52-nf3-vision:v3-attn

The docker-compose.yml in this repo is the exact measured configuration:

hf download 0xSero/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid-Vision --local-dir ./glm52-vision
MODEL_DIR=./glm52-vision docker compose up

Needs 4× 96GB sm120 GPUs + ≈64 GB RAM. Serves DCP4, FP8 KV with a 509,467-token KV pool @ 250,000 max context (util 0.974; ≈85–88 GiB weights per GPU). Boots in ≈2–3 min (fastsafetensors, ≤2 GB shards). Single-stream decode ≈24–37 t/s on a PCIe-only (no-P2P) 4-card box. The compose mounts mxfp8_tier_nokvb.json — the measured overlay convention from the base card (kv_b_proj + shared_experts stay true BF16).

Vision requests use standard OpenAI image_url content parts against /v1/chat/completions; the image token budget comes out of the same context pool.

Differences from the text-only base serving

• MTP is off. The five-token MTP draft head generates drafts on this graft but the accepted-token counter stays at zero, so speculation is disabled rather than shipped broken.
• FP8 KV only. nvfp4_ds_mla KV is unsupported by the MLA attention backend for sm120 in this runtime.
• config.json carries the EOS fix applied 2026-07-22 (generation previously would not stop on the vision template).

Build it yourself

The image is madeby561/vllm-glm52-nvfp4-nf3-hybrid:v3 (public, Docker Hub) plus a pip install of the small plugin in serving/:

cd serving
docker build -t glm52-nf3-vision:v3-attn .

Text hybrid by madeby561; NVFP4 expert tier by Luke Alonso; vision tower NVFP4 by baseten; vision graft + serving plugin by 0xSero. Built to answer the follow-up question: can the 4-card full-model box also see? Smoke says yes — benchmarks pending.

Acknowledgements

• Z.AI for GLM-5.2 (MIT).
• madeby561 for the GLM-5.2-MXFP8-NVFP4-NF3-Hybrid hybrid-quant lineage.
• baseten for GLM-5.2-Vision-NVFP4.
• lukealonso for GLM-5.2-NVFP4.