GLM-5.3-Flash-EXL3-TP4-Spark

GLM-5.3-Flash EXL3 on 4× DGX Spark — TP4 Serving Champion

Serving stack, quantization findings, drafter training evidence, and full negative-results ladder for running GLM-5.3-Flash with EXL3 quantization + DFlash2 speculative decoding on a 4-node NVIDIA DGX Spark (GB10) cluster over Mellanox SN2700 RoCE, vLLM TP4.

Champion result: 31.7 tok/s single-stream (c=1), 57.7 tok/s @ c=4, temp=0.

Drafter weights: cfontes/GLM-5.3-Flash-DFlash2-TR3-v3

This is the EXL3 lane of our two-lane GLM-5.3-Flash TP4 project. The NVFP4 lane champion is documented at cfontes/GLM-5.3-Flash-DFlash2-TP4-Spark (37.9 tok/s c1 / 94.2 c4). This repo documents how close EXL3 gets, why the remaining gap exists, and everything that didn't work.

Champion configuration

Launcher: launch-exl3-tr3-tp4.sh — set EXL3_MODEL, DFLASH_K, DFLASH_DRAFT, CUDA_GRAPHS env vars; run head on rank 0 and worker <1|2|3> on the rest.

Results (temp=0, llm-inference-bench, ctx=0)

Concurrency (champion config)

The full ladder (every lever we tried)

See results.tsv and results/ for the raw bench JSONs. Highlights:

• 4bpw eager (previous best): 27.1 tok/s
• CUDA graphs on (--enforce-eager removed): 30.7 (+11%, the single biggest serving-side win)
• 2bpw routed experts: neutral in eager mode (26.5) but wins under CUDA graphs (31.7) — 2bpw costs ~0.10 accept but the smaller dequant footprint composes with graphs
• K sweep under graphs: K2 > K3 > K4 > eager-best; K2 is the operating point
• FLASH_ATTN backend: 30.9 — TRITON_ATTN stays
• v1 (vendor) drafter A/B: 18.9 tok/s — our retrain is worth +68% on this stack

Drafter training (TR3 pipeline)

The served drafter (v3) is a retrain of the vendor incoai/GLM-5.3-Flash-DFlash2 checkpoint on 427,444 capture files (160 GB) of hidden-state data captured from the live EXL3-TR3-4bpw TP4 stack via the fork's DFLASH_SAVE_HIDDEN capture hook — i.e. trained on exactly the quantized target it serves against, no capture/target mismatch.

5-component loss (Phase-1 plan): 1.0·L_block_CE + 0.5·L_KL + 2.0·L_sel + 0.5·L_anchor + 0.1·L_hid, stage-A (train fc/hidden_norm/projection/codebooks, decoder frozen), position-weighted CE 0.85^k.

Results: 4-epoch retrain lifted served accept 1.71 → 1.83 (+7%) vs the v2 drafter at K2 (+68% throughput vs vendor v1 drafter). A 32-epoch run (val loss 4.21) moved served accept by only +0.02 — teacher-forcing saturates.

v4 (negative result)

A noise-augmented variant (σ=5% gaussian noise on input hidden states, 8 epochs) scored 31.1 tok/s / accept 1.75 — no gain. Retraining cannot close the gap to NVFP4; see findings below.

Findings

1. The EXL3 vs NVFP4 gap is quantization noise in mid-layer states, not drafter training

We measured the drafter's information ceiling directly: reconstructing target logits from the captured spec_hidden_states via the frozen lm_head (a linear probe of exactly what the drafter sees) yields only 30% top-1 agreement with the target's actual token. The v3 drafter's pos-0 accept is 53–58% — already +23–28 points beyond its own linear input ceiling. The NVFP4 lane's drafter hits 73% pos-0. The drafter is not the bottleneck; the quantized mid-layer hidden states are.

2. Step time on TP4 is collectives-bound, not weight-bound

EXL3-4bpw and EXL3-2bpw have identical steps/s in eager mode (15.3–15.4). Quantizing routed experts from 4→2 bits buys nothing until CUDA graphs remove launch overhead — then 2bpw pulls ahead (17.9 vs 17.0 steps/s). If you're CPU-launch-bound, quantize lighter; if you're NCCL-bound, quantizing is free acceptance loss.

3. --enforce-eager was costing 11%

The proven EXL3 recipe shipped with eager mode. Removing it (CUDA_GRAPHS=1 in our launcher) is a free +11% tok/s on this fork. (Note: this contradicts our NVFP4-lane PIECEWISE-graphs falsification — different fork, different graph implementation. Test both ways on your stack.)

4. Accept-length ceiling math

tok/s = accept × steps/s. At the measured 17.9 steps/s ceiling, 80 tok/s single-stream would need accept length 4.5 — 1.5× what the NVFP4 champion's drafter achieves on cleaner states. 80 tok/s is not reachable on EXL3 with any drafter trained on its hidden states. The honest comparison: EXL3-2bpw delivers ~84% of the NVFP4 lane's c1 throughput in under half the memory footprint.

What's in this repo

• launch-exl3-tr3-tp4.sh — champion launcher (env-driven: EXL3_MODEL, DFLASH_K, DFLASH_DRAFT, CUDA_GRAPHS)
• results.tsv — machine-readable results ladder
• results/*.json — every bench run cited above (llm-inference-bench format)
• MODEL_CARD.md — HuggingFace model card for the serving stack

Lineage & credit

We did not do this alone:

• zai-org/GLM-5.3-Flash — base model, Z.ai / Zhipu AI
• DFlash: Block Diffusion for Flash Speculative Decoding — Chen, Liang & Liu (2026), Z Lab — the speculative method
• incoai/GLM-5.3-Flash-DFlash2 — vendor drafter checkpoint (v1; starting weights for our retrains)
• @vcruz305 — the vllm-exl3 native kernels that make EXL3 usable on GB10/SM121 at all, and the receipts that MTP K=2 is the right operating point on this pack
• ZJY0516 — the vLLM fork runtime with the DFlash2 + EXL3 integration our stack runs on
• MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks — Mia (@MiaAI_lab). Our TP4 build is derived from her 2×Spark setup: the multi-node launch pattern, MNBT-7168 default, TRITON_ATTN guidance, and much of the serving recipe
• @Reederey — EXL3 fat-expert MoE kernel work that preceded vcruz305's native suite
• eg impulse / exllamav3 — the EXL3 quantization format & codebook design

Original work (this cluster): 4-node TP4 capture/training harness for DFlash2 on EXL3, the TR3 drafter retrains (v3 champion, v4 falsification), the 2bpw-routed-experts quant config evaluation, CUDA-graphs result on this fork, the hidden-state ceiling analysis, and all benchmark evidence.

Hardware

4× NVIDIA DGX Spark (GB10, 128 GB unified), rank 0–3, RoCE over Mellanox SN2700 (NCCL_IB_HCA=rocep1s0f1), vLLM fork runtime image (sm121a).