GLM-5.2-NVFP4-AQLM-hybrid

GLM-5.2-Vision NVFP4+AQLM hybrid — image input + ~1M context on 4x RTX PRO 6000 (SM120, 384 GB)

Baseten's GLM-5.2-Vision graft on the NVFP4+AQLM hybrid quant — a 744B-param GLM-5.2 that now takes images, still serving a ~950K-token window on 4x 96 GB SM120 GPUs with lossless MTP speculative decode and disk-backed KV persistence.

Credits

• Baseten built the vision graft (baseten/GLM-5.2-Vision-NVFP4): the MoonViT vision encoder from Moonshot AI's Kimi-K2.6 bolted onto Z.ai's GLM-5.2 through a PatchMerger projector baseten trained (49.5M params, 1152→4608→6144) — without touching a single GLM weight. The vision tower (MoonViT-3d, 27 layers, 1152-dim, ~417M params) and projector in this repo are byte-identical BF16 copies of baseten's release.
• Z.ai — GLM-5.2 (MIT). Moonshot AI — Kimi-K2.6 / MoonViT (Modified MIT).
• This repo swaps baseten's frozen NVFP4 text backbone for our NVFP4+AQLM hybrid quant (292 GB, all 256 experts routable), which is what makes the ~1M window fit on 4x 96 GB instead of 8x B200.

Quickstart (4x RTX PRO 6000, SM120)

Not stock vLLM — use the pre-patched fork, and note the branch: vision support lives on vision-graft, not the text-only cudagraphs-v2 branch.

git clone -b vision-graft https://github.com/jarrelscy/vllm-glm52-sm120
cd vllm-glm52-sm120 && git checkout 57955eed9   # exact gated head
docker build -f Dockerfile.glm52-sm120 -t glm52-vision-sm120 .

docker run --gpus all --ipc=host -p 8001:8001 \
  -v /path/to/this-repo:/models/1m:ro \
  -e PARALLEL=tp4-1m-mtp \
  -e ENABLE_LMCACHE=1 -e UTIL=0.96 \
  glm52-vision-sm120 --trust-remote-code

Three things differ from the text-only card and all three are required:

1. Branch vision-graft (head 57955eed9, built on cudagraphs-v2). It adds the glm5v architecture (Glm5vForConditionalGeneration): MoonViT tower + projector wired into the fork's in-tree Kimi-K2.5 vision code, with the language model swapped to the hybrid GlmMoeDsaForCausalLM stack (DCP + native MTP + SM120 sparse-MLA fixes + NVFP4/AQLM MoE kernels). The text-only branch cannot load this repo's config.
2. --trust-remote-code — image preprocessing uses the Kimi-K2.5-style processor files shipped in this repo (kimi_k25_processor.py etc.).
3. UTIL=0.96 when LMCache is on — measured KV pool 991,727 tokens ≥ the 950K window. The entrypoint's LMCache default of 0.95 lands ~917K (boot failure); 0.97 leaves too little headroom for LMCache's ~1 GiB/GPU staging buffer and risks store-OOM. With ENABLE_LMCACHE=0 the default 0.97 is fine (KV 1,066,222).

The image build pins flashinfer to a known-good commit and stubs nvidia-cutlass-dsl's cute.experimental (the republished 4.5.2 wheel AttributeErrors at import flashinfer.comm), so a plain docker build is index-independent and reproducible.

Then send images through the standard OpenAI API:

curl http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "glm-5.2",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
    {"type": "text", "text": "Describe this image."}
  ]}]
}'

Validation (2026-07-23, 4x RTX PRO 6000)

Everything was gated combined — vision + 950K window + MTP + LMCache in one serving config — against the production text-only build as reference:

• Text regression: prompts without images produce output identical to the text-only production build (the graft leaves the text path untouched).
• Vision: single- and two-image prompts described correctly with correct per-image attribution, no hallucination; re-verified live on the exact baked production image.
• Long context + vision: needle-in-haystack with an image in context, exact at 130K tokens.
• MTP: acceptance 3.17–3.30 accepted/step (72–77%) with images present — inside the healthy envelope of the text-only production stack; spec decode stays lossless by rejection sampling.
• LMCache: cold-restart disk restore of a long mixed prompt in 3.4s vs 109s recompute (~32x), byte-identical output vs the cache-miss path.
• Cost of sight: tower+projector add ~320 MB/GPU at TP4 (~58K tokens of KV pool) — the 950K window still fits, at util 0.96 even with LMCache's buffers.

<details> <summary><b>Bare launch</b> — equivalent flags without the docker entrypoint</summary>

export NCCL_MAX_NCHANNELS=4 NCCL_BUFFSIZE=1048576 VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=192
export NCCL_P2P_LEVEL=SYS VLLM_MTP_INDEX_SHARE=1 GLM_MOE_LANE_ROWS=1 GLM_NVFP4_LUT256=1
vllm serve <repo-dir> --trust-remote-code \
  --tensor-parallel-size 4 \
  --decode-context-parallel-size 4 --dcp-comm-backend ag_rs \
  --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":3}' \
  --compilation-config '{"mode":3,"cudagraph_mode":"PIECEWISE"}' \
  --gpu-memory-utilization 0.97 --kv-cache-dtype fp8_ds_mla \
  --max-model-len 950000 --max-num-seqs 2 --max-num-batched-tokens 4096 \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --served-model-name glm-5.2 --port 8001

This bypasses the entrypoint, so no LMCache (use the docker run form for disk-backed KV; that path needs the patched jarrelscy/LMCache@glm52-dcp-dsa fork and util 0.96). cudagraph_mode must stay PIECEWISE for DCP+spec; do not add --enforce-eager.

</details>

Quantization & performance

The text backbone is exactly jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid (hot ~30% of experts NVFP4 @4.5 bpw, cold experts PV-tuned AQLM @2 bpw, attention/DSA/dense/shared/embeddings BF16, fp8_ds_mla KV) — see that card for the scheme, teacher-forced perplexity vs. other quants, GPQA Diamond 0.889, and measured throughput; text-path numbers carry over unchanged. Vision tower and projector run in BF16 (tower sharded across TP ranks, projector replicated).

Terminal-Bench 2.1

Agentic-coding eval on this exact serving stack (the hybrid quant above; text path, no images), using the harness's terminus-2 agent driving glm-5.2 over the OpenAI API at reasoning_effort: max, temperature: 1:

• pass@1 = the earliest-completed attempt per task, scored by the task's own verifier. 88 of the 89 tasks reached a verdict; 1 task (regex-chess) is excluded from the denominator because the agent never converged to a submission — it looped on tool-call exploration for hours without writing a solution file, so no verdict exists to count either way. Counting it as a failure instead gives 69/89 = 77.5%.
• The run was executed in two halves against the same model weights (an initial 47-task pass, then the remaining 42 re-run on the local 4x RTX PRO 6000 endpoint) and combined under one consistent pass@1 rule; each half scored ~78% independently (37/47 and 32/41).
• Per-task trajectories, verifier outputs, and the results table are published under traces/ in this repo.

Video input is untested. For text-only serving, the text card and its cudagraphs-v2 branch remain available, though this repo is a strict superset in capability.