GLM-5.3-Flash-EXL3-TR3-3.5bpw

GLM-5.3-Flash EXL3/TR3 3.5bpw Mixed (per-tensor K3/K4)

First sub-4bpw GLM-5.3-Flash checkpoint that serves 1M context + MTP3 speculative decoding + CUDA graphs + thinking mode simultaneously on two RTX PRO 6000 (SM120, 96 GB each). At 4bpw that combination does not fit in 192 GB: the drafter KV plus a 1M-token pool exceeds the cards. The mixed rate frees exactly the gap. One serve does deep context AND fast decode; no separate read-lane and fast-lane serves.

GSM8K 96.89 / MATH-500 77.8 / HumanEval 82.3 raw are measured and published below - all at or above the 4bpw references on the same protocols. The deep-context decode curve is flat across 500K-950K, and the five-run KLD fidelity gate PASSED on 2026-09-02: mean 0.029654 nats vs the absolute 0.06 bar (4bpw reference 0.024555). Every gate below is measured on this exact weight set.

Also: speed/context claims below are from live serving receipts; the derivation is fully provenance-sealed (see How this was built).

Measured (2026-09-01/02, this exact weight set)

Reference on the same hardware: the 4bpw DFlash2 cell peaks at 78-91 tok/s with 192K-230K context. This checkpoint beats that decode rate at 5x the context.

Concurrency correction (2026-09-02): the first concurrency rows were measured on a serve booted with max_num_seqs=1 (streams serialized inside the engine; aggregate read flat ~108-110). The rows above are the corrected measurement on the batching profile (max_num_seqs=8, max_num_batched_tokens=4096, same forced-1024 protocol). The same bench's single stream reads 174-179 tok/s on that profile (repetition-favorable text lifts MTP acceptance).

DFlash2 drafter on this checkpoint (230K, method=dflash, depth 7, real batching): decode 214-226 tok/s single stream warm, acceptance 3.6-4.2, aggregate 167.5 at 6 streams / 168.2 at 8 (per-stream 46 / 31). Healthy, but MTP3 at 1M context beats it on aggregate too (227.5 / 246.4), so MTP3 remains the product profile. DFlash2 at 1M is pool-arithmetic-impossible (16.71 GiB KV needed per 1M request vs 7.84 available); its ceiling on this checkpoint is ~450K. Vision profile: swap --language-model-only for --limit-mm-per-prompt plus one rotary bind-mount patch (recipe repo).

Evidence files and the full serve recipe: the GitHub repo satindergrewal/GLM-5.3-Flash-EXL3-3.5bpw-Mixed-SM120-TP2. Real-session evidence: a multi-turn agentic session (tool calls + long reasoning) at 51-109 tok/s client-side with thinking on.

Quantization details (this exact weight set)

Why mixed instead of uniform 3.5: the codec has no uniform 3.5 rate. A per-tensor K3/K4 split at the exact 7/2 average is codec-native, needs zero kernel changes, and lets the allocator spend bits where calibration energy says they matter.

Requirements

• 2x RTX PRO 6000 (SM120), 96 GB each; docker
• Image built from the recipe repo Dockerfile (glm53-exl3:dflash2-mixed)
• ~150 GB disk for the checkpoint

Quick start

git clone https://github.com/satindergrewal/GLM-5.3-Flash-EXL3-3.5bpw-Mixed-SM120-TP2.git
cd GLM-5.3-Flash-EXL3-3.5bpw-Mixed-SM120-TP2
./start.sh          # port 8012, 1M context, graphs, tools; waits for health
./status.sh         # container + endpoint + GPUs at a glance
./stop.sh           # clean shutdown incl. the process-tree sweep

Boot is about 13 minutes: 120 shards, b12x compile, CUDA graph capture. Full env matrix and troubleshooting are in the recipe repo README.

Memory and context

Weights 145.4 GiB (~73 per card) leave room for a 1,985,915-token NVFP4 MLA KV pool: 1M context with 1.99x concurrency, or two 500K sessions, or deeper single requests with headroom.

Speculative decoding and tools

MTP3 drafter built in (acceptance ~2.4 mean, measured while serving with thinking on). Tool calling works through the glm45 parser with --enable-auto-tool-choice; verified end to end with OpenAI-format tools in a live agent session.

Quality gates (this exact weight set)

• GSM8K 5-shot: 96.89 (1319 problems, T=0.2, thinking on; 4bpw reference on the same hardware/protocol: 96.51).
• MATH-500: 77.8 raw (500 problems, T=0.2, boxed+normalized compare, budget retry; 4bpw raw on this class of protocol: 75.4).
• HumanEval: 82.32 raw (164 problems, T=0.2, canonical subprocess checks; 28 thinking-budget empties; budget-corrected class ~98.5. 4bpw same-protocol shape: 72.6 raw with 41 empties).
• Deep-context prefill curve: 2,982 t/s @500K, ~2,940 @750K, 2,880 @950K (correct answers at every rung; flat within 3% across depth, and within 6% of the 4bpw eager read-lane reference while also carrying MTP3 + CUDA graphs).
• KL divergence vs BF16 teacher: PASSED - mean 0.029654 nats vs the absolute 0.06 bar (2026-09-02). Five cold runs over the sealed 25-window final panel, 51,175 positions per run, full-vocab float64, teacher_to_student. Per-run means 0.02976 / 0.02975 / 0.02962 / 0.02969 / 0.02946 (stddev 0.000124); zero failing runs. Reader parity audited 27/27 samples (18 K3 + 9 K4 matrices; max abs error 0.0025 K3, 0.0036 K4 vs 0.025 tolerance). Sealed receipt 7c121f35fa434e8ab9d68ade46644e35004140e59531a6ca03fd77944a7fe4bf; evidence file in the recipe repo under docs/evidence/. 4bpw reference: 0.024555 (recorded, not gating).

Eval receipts (protocol, per-problem results) live in the recipe repo under docs/evidence/.

Charts

How this was built

Derived from the zai-org BF16 master with brandonmusic's published R10 encoder closure and teacher-logits calibration set, extended by a per-tensor mixed-rate allocation campaign: 45 encode units, energy-balanced K3/K4 assignment, sealed readiness/layer/pack receipts at every step. The artifact declares the mixed schema through the r7_routed_experts block in config.json; per-tensor widths ride in the trellis payload shapes, so no kernel changes are needed to read it. Derivation drivers and the patched runtime are in the recipe repo.

Honest limits

• Vision is qualified at 400K, at 499968 (gpu-memory-utilization 0.96), and at 1M context with up to 4 images per request (gmu 0.975): exact OCR + shapes/colors verified 2026-09-02/03, engine alive after, multi-image included. The original 499968 wedge was encoder headroom at gmu 0.985; the fix is the gmu drop. Video at 1M does not fit the KV pool (per-1M-request need 3.98 GiB vs the headroom math); video is a <=500K-profile capability (serve/serve-k35-vision.sh, image:4 + video:1). The 1M+vision profile is serve/serve-k35-vision-1m-multi.sh in the recipe repo.
• DCP=2 is not supported on 2-discrete-card hosts (A2A handshake wedge); DCP=1 is the qualified configuration.
• The 3.5bpw mixed rate is the first of its kind on this runtime; the KLD receipt above is the fidelity contract, and it certified this checkpoint on 2026-09-02 (0.029654 < 0.06, five cold runs).
• Prefill block size must be 64 or 32 (VLLM_EXL3_PREFILL_BLOCK_M); the image default 128 fails the FC2 route-subtile check.

License

The checkpoint and scripts inherit the upstream stack terms: base model per zai-org; TR3/MCG toolchain and calibration lineage per the ShapleyMCG License 1.0 (see license link above); see the recipe repo for the full third-party notice breakdown.

Credits

• zai-org: GLM-5.3-Flash base model
• brandonmusic: SM120 EXL3 runtime fork (v44-v84), the R10 encoder closure and the teacher-logits calibration set this derivation is built on
• tonyd2wild: the DFlash2 vLLM overlay this runtime's drafter path builds on
• local-inference-lab: b12x (mixed-rate trellis3 API)
• turboderp: EXL3/TR3 format and kernels