GLM-5.3-Flash-NVFP4-mtp-nvfp4-delta

GLM-5.3-Flash-NVFP4 with an NVFP4 MTP head (delta overlay)

This repo is not a complete checkpoint. It holds only the files that differ from nvidia/GLM-5.3-Flash-NVFP4 at revision 09b04e5e74bca08ca8549fc736d4cdd8624bfde3. Overlay it on that revision to get a checkpoint whose MTP (multi-token prediction) layer loads and runs as a speculative-decoding drafter in vLLM.

Shards 4–33 and all tokenizer / template files are byte-identical to nvidia's and are not re-uploaded.

Assemble

hf download nvidia/GLM-5.3-Flash-NVFP4 --revision 09b04e5e74bca08ca8549fc736d4cdd8624bfde3 --local-dir base
hf download stihahi/GLM-5.3-Flash-NVFP4-mtp-nvfp4-delta --local-dir delta
mkdir merged
cp -al base/. merged/                     # hardlinks, no extra disk
for f in delta/*.safetensors delta/*.json; do rm -f "merged/$(basename $f)"; cp "$f" merged/; done

The launch script, the vLLM patch it needs, the quantization script and benchmark numbers are at https://github.com/stihahi/glm53-flash-nvfp4-mtp-2x-dgx-spark.

Why

The official NVFP4 checkpoint ships the MTP layer's routed experts in BF16 (864 tensors, 13.5 GiB) but does not list layer 45 in its quantization ignore list, so vLLM builds FP4-sized parameters for them and the load fails (_load_w2 shape mismatch). Fixing only the ignore list makes it load, but on 2× DGX Spark (GB10, TP2) the BF16 experts push weights to 95.93 GiB per rank and the hosts run out of memory during profiling. With the experts in NVFP4 the weights are 91.08 GiB per rank, leaving room for a 6 GiB KV pool (673k tokens), and MTP runs with num_speculative_tokens=2.

Measured on 2× DGX Spark, vLLM TP2 over RoCE, 768 output tokens, reasoning on:

Draft acceptance 61–69%. Relative dequantization error of the re-quantized experts: ~9.5%.

Quantization method

For each expert, gate and up share one global scale (vLLM requires w13 to have a single weight_scale_2); down has its own. Block scales (FP8 E4M3, group 16) are computed with vLLM's scaled_fp4_quant. input_scale is copied from the same expert index in layer 44 (the W4A16 Marlin path used here ignores it). Script: tools/quant_mtp_experts.py in the GitHub repo.

License

MIT, same as the base models (zai-org/GLM-5.3-Flash, nvidia/GLM-5.3-Flash-NVFP4). This is an unofficial derivative, not affiliated with Z.ai or NVIDIA.