GLM-5.2-MXFP8-NVFP4-NF3-Hybrid

GLM-5.2-MXFP8-NVFP4-NF3-Hybrid

Ready-to-run v20 Compose profiles are included for four 96 GiB SM120 GPUs, with validated context lengths from 400K through 775K using FP8 or calibrated NVFP4 MLA KV. Use the pinned v20 image and included launch files because the checkpoint's custom NF3 expert tensors require its native runtime support.

The full 753B-parameter GLM-5.2, with all 256 routed experts per MoE layer, compressed into 340.9 GiB (366,021,469,560 bytes) without expert pruning. Sixty-four damage-selected expert slots per layer retain Luke Alonso's NVFP4 tensors; the other 192 use the healed NF3 format and its SM120 Grid188 kernel. Attention, dense, and shared-expert tensors remain BF16 in the checkpoint. Eligible linears are converted once to MXFP8 at load according to the selected serving profile.

The weight tensors are v3.6 at immutable weight revision 68babde27a97a4c980c2494e830dd424975cd5a3. They contain 184 safetensor shards of at most 2 GB. The reviewed release bundle adds only documentation, Compose files, the launcher shim, and the calibrated KV-scale sidecar; it does not rewrite those weight shards.

Four v20 serving profiles

All profiles use TP4/DCP4, A16 expert activations, lossless BF16 DCP transport, and InstantTensor BUFFERED. The FP8 profile uses the accepted v19 online-MXFP8 membership. The two NVFP4 capacity profiles additionally convert shared experts online; their KLD is therefore certified separately rather than borrowed from the FP8 profile.

The final 775k profile allocated exactly 775,000 KV tokens and completed a token-counted 764,999-token prompt plus 16 generated tokens. It then completed a fresh request returning POST CAP READY, with zero engine restarts. It is a tight four-GPU pin and uses a small launcher shim plus two source patches applied only inside the ephemeral container. Read the KV-token count at startup and lower MAX_MODEL_LEN if your machine has less runtime headroom.

An exact-stack ceiling canary also completed a 774,999-token prompt against a 785,000-token pool and passed post-cap reuse, but its measured free-memory floor was only 94 MiB per GPU. The portable release profile therefore remains 775,000 rather than advertising the brittle ceiling.

The FP8-KV/MTP3 overlay allocated 450,255 KV tokens for a 450,000-token advertised limit. It completed a 449,935-token prompt with coherent generation and then a fresh 8k request, with zero restarts/OOMs and a 146 MiB measured free-memory floor. The explicit KV-byte pin avoids an automatically sized 467,627-token pool that booted but left only 6 MiB after lazy allocation.

Accuracy

GPQA-Diamond used temperature 1.0, top-p 0.95, and maximum reasoning effort. The v3.6 row is this exact checkpoint.

KLD is a distribution-drift instrument, not an intelligence score. The table above remains the direct quality check.

KLD method

The reported KLD uses the current shared GLM-5.2 ruler:

• source: zai-org/GLM-5.2 BF16 reference logits from 2026-07-08;
• one fixed 2,048-token Wikitext window, 2,047 scored positions;
• full 154,880-token vocabulary;
• direction KL(BF16 reference || candidate);
• TP4/DCP4, MTP0, A16, five fresh engine starts;
• exact 78-character GLM IndexCache pattern.

The final v20 review reruns each precision posture five times with a fresh engine and an isolated empty cache. KLD is target-only MTP0 even when the corresponding serving profile enables MTP3. The scale file is an audited, per-layer outer-scale artifact; it changes no checkpoint weights and adds no KV bytes.

The machine-readable aggregate is included as KLD_MATRIX.json. The published scale sidecar uses host-neutral provenance labels; its max_abs and scales arrays are byte-for-byte identical to the artifact used for every KLD receipt. KLD_MATRIX.json records both the receipt-artifact hash and a canonical numeric-payload hash.

Do not compare these values with the old card's FP8-as-reference, top-8192 drift table. That was a different reference, support, and direction.

Serve from the immutable v20 image

Exact image:

voipmonitor/vllm:gilded-gnosis-v20-vllm0c79e41-sie603f74-fi801d57a-cu132-20260726
sha256:10261c7d65101c8aba2ce1fb59eabe73aff9d35eca5043b330cc0ce76d3c98d0

A matched four-GPU A/B retained this July-26 image over the July-25 final image: the selected image measured 2,291/2,247 prompt tok/s at 64k/128k versus 2,264/2,217, and it contains the scale-file and launcher interfaces used by the reviewed profiles. A later policy A/B found that the July-26 helper's opt-in owner-top-k merge was the remaining prefill regression on this topology. The final 775k overlay disables that experimental route and retains the established replicated global-top-k oracle.

Download without Xet:

export HF_HUB_DISABLE_XET=1
hf download madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid \
  --local-dir ./GLM-5.2-MXFP8-NVFP4-NF3-Hybrid
cd ./GLM-5.2-MXFP8-NVFP4-NF3-Hybrid
export MODEL_DIR=$PWD

Do not pin the old weight revision for this serving recipe: that historical commit predates the calibrated scale sidecar. After publication, pin the final release-bundle commit if an immutable metadata checkout is required.

Choose one of the standalone Compose files included in this repository:

# Roughly 400k context, lowest measured KLD.
docker compose -f docker-compose.fp8.yml up -d

# 450k/MTP3, FP8 KV with shared experts converted online to MXFP8.
docker compose \
  -f docker-compose.fp8.yml \
  -f docker-compose.fp8-450k-mtp3.yml up -d

# 700k quality-first long-context pin, calibrated NVFP4 KV and BF16 RoPE.
docker compose -f docker-compose.nvfp4.yml up -d

# 775k/MTP3 capacity profile, calibrated NVFP4 KV and FP8 RoPE.
docker compose \
  -f docker-compose.nvfp4.yml \
  -f docker-compose.nvfp4-775k-mtp3.yml up -d

Wait for health before sending traffic:

curl -fsS http://127.0.0.1:5001/health
curl -fsS http://127.0.0.1:5001/v1/models

The two base files default to MTP0 so their advertised capacity is reproducible. The 450k and 775k overlays enable MTP3. Keep MTP disabled for target-only decode measurements or KLD work:

MTP=0 docker compose -f docker-compose.fp8.yml up -d --force-recreate

The v20 helper owns DCP query split, CKV gather, bounded prefetch, InstantTensor, NCCL, graph sizing, sparse attention, and parsers for the two base profiles. Leave those controls on auto there. Both MTP3 overlays pin the measured validation policy to avoid a fresh-cache calibration cleanup defect and to bound their CKV workspace. The 775k NVFP4 overlay additionally pins both DCP_TOPK_OWNER_MERGE=0 and VLLM_DCP_TOPK_OWNER_MERGE=0: the July-26 auto-policy otherwise enables an explicitly opt-in owner merge that regressed 64k/128k prompt throughput by about 19% on the measured topology. The overlays also apply the included MTP online-quant inheritance and sparse-indexer carry-fold patches before launch. No checkpoint tensor is changed. F8_DMA=0 is intentional: it keeps DCP transport lossless.

Why the Compose uses ONLINE_QUANT=custom

The image's convenient nf3-mxfp8 shorthand is a faster, more aggressive membership that also converts shared experts and fused q-a/kv-a projections. The FP8 profile uses this measured quality posture:

{"linear":{"weight":"mxfp8"},"ignore":["re:.*\\.fused_qkv_a_proj$","re:.*\\.q_a_proj$","re:.*kv_a_proj_with_mqa","re:.*\\.mlp\\.gate$","model.layers.78.eh_proj","lm_head"]}

The NVFP4 profiles use the same ignore list plus "shared_experts":{"weight":"mxfp8"} to recover the capacity required by their public pins. Their five-run KLD rows measure that exact membership.

The NVFP4 Compose also passes VLLM_NVFP4_MLA_SCALES_FILE explicitly. The final v20 image contains scale support, but checkpoint-relative auto-discovery is not part of this release. The generic vLLM environment scanner may still label this extension variable unknown; the stock v20 MLA writer reads it directly. Runtime verification observed non-identity latent_scale_identity=0 kernel signatures, proving that the calibrated values—not the identity fallback—reached the attention path.

v20 performance

Final review measurements on four RTX PRO 6000 Blackwell Max-Q GPUs:

The final 775k/MTP3 run measured a five-run coding peak of 121.6 tok/s mean / 124.3 tok/s maximum. The final prefill row is the second complete exact-token 64k/128k pass after warmup. The bounded carry-fold measured within about one percent of the unpatched owner-merge-off profile, while avoiding the stock fold's 112 MiB near-capacity allocation and engine OOM.

The 450k FP8 row combines the capacity certification with a separate warmed, exact-token 64k/128k prefill measurement. The older 180k row retains the matched decode measurements. No unmeasured decode number is borrowed for the new shared-expert precision posture.

VLLM_B12X_ABSORB_BMM=1 is active in the certified 775k posture. Load ordering matters: after the 184-shard target load, v20 logged that it was serving MLA absorbed projections directly from the B12X MXFP8 pack. The same-checkpoint MTP inheritance patch gives layer 78 the identical online-MXFP8 membership, so the four-shard draft also uses the direct pack and no post-draft fallback is logged.

The second patch selects SparkInfer's existing streaming carry fold for logical sparse-indexer output. Exact CUDA tests produced identical selected token sets at 131k and 524k. This removes page-table-sized transient candidate tensors that otherwise caused late OOMs even after a successful 775k boot. These are explicit runtime patches over the immutable image; this profile is not described as unmodified stock v20.

High-concurrency status

The table above intentionally reports C1 decode only. The July-26 v20 NF3 hybrid kernel has a known batch-dispatch crossover under investigation: Grid188 covers the smallest decode bucket and the optimized direct-top-k launch is currently bounded to eight routed rows; larger MTP verifier batches fall back to the packed route used for prefill. This does not affect correctness, KLD, advertised KV capacity, C1 decode, or the prefill results reported here, but it limits C2-C8 scaling. A lossless native dispatch fix is being prepared for an upstream runtime release; no unvalidated concurrency number or private kernel patch is included in this checkpoint release.

Credits

• Base model: zai-org/GLM-5.2
• NVFP4 tier: lukealonso/GLM-5.2-NVFP4
• v20 runtime and documentation: local-inference-lab/rtx6kpro
• Shared KLD ruler: GLM-5.2 KLD Evaluation

Built and evaluated by madeby561.