GLM-5.2-Vision-MXFP8-NVFP4-NF3-Hybrid

GLM-5.2-Vision-MXFP8-NVFP4-NF3-Hybrid

🙏 Credit where it's due: this repo is a convenience merge of two other people's work, not an original model. The quantized text model — the NF3 format, the custom kernels, the expert allocation, all of it — is madeby561's GLM-5.2-MXFP8-NVFP4-NF3-Hybrid (text shards byte-identical). The vision head — the trained projector and the assembled MoonViT tower — is Baseten's GLM-5.2-Vision-NVFP4. All this repo adds is the glue: a merged config, a ~150-line vLLM plugin, and a serving image so the combination deploys in two commands. If this model is useful to you, their cards are where the interesting work is.

⚠️ Usable — read the hazards. Loads ONLY through the custom serving image below (in-house NF3 3-bit kernel + the glm5v_nf3 vLLM plugin — no official stack runs it). Serves on 4× 96GB sm120 GPUs with a 330k-token KV pool @ 262k max context at DCP4 + MTP-5 (measured 2026-07-23). Not the official model's 1M context, and not officially supported.

The full GLM-5.2 — 753B, all 256 experts, NF3-hybrid quantized — with eyes, on the same 4× 96GB cards. This is madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid (v3.6) with the vision head from baseten/GLM-5.2-Vision-NVFP4 grafted on: the frozen MoonViT tower (Kimi-K2.5 lineage, 27L/1152d, credit Moonshot AI) plus Baseten's trained PatchMerger projector (49.5M params, 1152→4608→6144), both kept BF16 (≈933 MB — quantizing vision towers measurably hurts OCR, so nobody does). The text tiers are byte-identical to the text-only hybrid: NVFP4 top-64 damage-allocated experts, NF3 long tail, BF16-stored / MXFP8-served non-expert layers.

The graft is possible because both parents share the exact same glm_moe_dsa text backbone (78L, hidden 6144, 256+1 experts, DSA sparse attention, MTP layer 78) and the projector targets hidden 6144 — the vision half bolts on without touching a single quantized byte.

What you get vs the text-only hybrid (A/B, same box, same day, both MTP-5)

Text behavior is unchanged by construction: same shards, same kernels, same sampling path. The only cost of vision is ≈23k KV-pool tokens. Vision quality: correctly describes layout/shapes/colors; OCR transcribed a ~11px-rendered text line character-exact in spot checks (not a benchmark run — MMMU/OCRBench pending).

ℹ️ Multi-image note — handled by this repo's chat template. The underlying model has no structural sense of image boundaries — it infers them from content dissimilarity (single-image projector training against a frozen text backbone that never learned to segment multimodal spans; the <|begin_of_image|>/<|end_of_image|> markers are reserved tokens with frozen embeddings). Raw adjacent image spans therefore fuse: similar images read as one, four images counted as "two", attributes scrambled across neighbors. This repo's chat_template.jinja fixes it by auto-numbering every image (Image 1:, Image 2:, … injected before each span — the only intervention that was stable across repeated probes; plain separators helped but flapped run-to-run). With the shipped template, our full probe battery — 4-image describe/count, identical-pair separation, identical-pair counting, similar-pair attribute binding — passes 24/24 across repeats at temperature 0. Implications: image numbering is part of the prompt (the model may refer to "Image 2"), and if you serve with a custom chat template, replicate the numbering or multi-image reliability regresses. Probes are synthetic (shapes/labels); MMMU/OCRBench still pending. --limit-mm-per-prompt caps images per prompt.

Serving

Ready-to-run image — everything from the text-only v3 image (NF3 kernel, hybrid loader, native mxfp8 tier) plus the glm5v_nf3 vLLM plugin baked in: chronarion/vllm-glm5v-nvfp4-nf3-hybrid:v1. The docker-compose.yml in this repo is the exact measured configuration:

hf download chronarion/GLM-5.2-Vision-MXFP8-NVFP4-NF3-Hybrid --local-dir ./glm5v-mxfp8
MODEL_DIR=./glm5v-mxfp8 docker compose up

Needs 4× 96GB sm120 GPUs + ≈64GB RAM. Boots in ≈4 min (fastsafetensors). Query it like any OpenAI-compatible VLM — content parts with image_url (data: URIs work); the chat template wraps each image as <|begin_of_image|><|image|><|end_of_image|>. --limit-mm-per-prompt '{"vision_chunk":8}' bounds the encoder reserve; raise it if you need more images per prompt and can spare KV pool.

The plugin (ships in ./plugins, ~150 lines)

The serving image's vLLM fork has the Kimi-K2.5 VLM in-tree (KimiK25ForConditionalGeneration = MoonViT + PatchMerger + DeepseekV2 text). glm5v_nf3 registers Glm5vForConditionalGeneration = that class with the text backbone swapped to GlmMoeDsaForCausalLM — the same one-swap trick as Baseten's SGLang plugin, ported to vLLM. The NF3/MXFP8/NVFP4 machinery needed zero changes: the expert bit-map keys on layers.(\d+), the mxfp8 tier matcher normalizes prefixes, and hybrid_bit_map rides in the top-level quantization_config — all robust to the language_model. nesting.

Three non-obvious fixes are in the plugin, all cases of runtime code assuming the model IS the text model (details in the source):

1. MTP draft-config derivation sees through the wrapper (hf_config_override patch; carries quantization_config + _name_or_path so the layer-78 experts stay NVFP4).
2. DSA indexer serving overrides are stripped from the draft config (they're baked into this repo's config.json; leaking them into the single-layer draft corrupts the heap).
3. lm_head is exposed on the wrapper so the speculator's target-head sharing finds it (without it: 0% draft acceptance, MTP worse than off).

Provenance / credits

• Text weights: madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid v3.6, byte-identical (which in turn credits lukealonso/GLM-5.2-NVFP4 for the NVFP4 expert tier and zai-org/GLM-5.2 for the base model, MIT).
• Vision tower + projector: baseten/GLM-5.2-Vision-NVFP4 (vision_tower.safetensors, mm_projector.safetensors, processor code, chat template — the projector is Baseten's training; the MoonViT tower follows Moonshot AI's Kimi-K2.5 architecture).
• Merged config.json: Glm5v wrapper (vision_config + this text_config + top-level quantization_config). Note vision_config.mm_hidden_size is 6144 here (vLLM reads it for the projector output dim; Baseten's SGLang config said 1152 and read text_hidden_size instead — don't "fix" it back).

By chronarion, building on madeby561's text-only hybrid — which asked whether a 4-card box can run the full model with no lobotomy. This one asks: can it also see? Text parity says yes, for ≈23k KV tokens.