keys-GLM-5.3-EXL3

keys-GLM-5.3-EXL3 — full GLM-5.3 (753B MoE) as a 3-bit EXL3, quantized for a serving container

What this is: the complete zai-org/GLM-5.3 (78 layers, 256 routed experts, DeepSeek-style sparse MLA attention with indexer, one MTP block) with the routed experts quantized to 3-bit EXL3 trellis and everything else kept bf16/fp16. 308 GB in 41 safetensors shards. It serves on four NVIDIA DGX Spark (GB10) with TP=4 + decode-context-parallel 4 at a 1,000,000-token context, inside Mia AI Lab's vLLM container for the GLM-5.3-Flash EXL3 (ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3) plus the small patch set shipped in serving/.

Why we made it this way

Every previous EXL3 GLM-5.3 we tried to serve on DGX Spark cost hours-to-days of patching: the published quants were made for TabbyAPI/exllamav3 (full-model EXL3, 2-bit heads, mul1 everywhere) and vLLM has no full-model EXL3 — Mia's container only implements EXL3 for the routed experts, with attention/dense/head native. So this time we quantized to the container's ABI first:

• scope = routed experts only (the only thing the overlay's Exl3Config quantizes), uniform integer bits in the overlay's allowed set {3,4,5,6} → 3-bit, because 4-bit experts (≈101 GB/rank) do not fit four 128 GB GB10s next to a 1M KV cache; 3-bit ≈ 78 GB/rank does.
• -hb 16 -mb 16: head and MTP stay bf16 so the stock vLLM paths load them.
• suffix layout trellis/suh/svh/<codebook marker> exactly as the overlay's loader expects, quantization_config.json with quant_method=exl3, bits=3, scope=glm53_routed_experts_only.

It still needed a handful of patches (all in serving/), but they are small, mechanical, and now documented — the point of this repo.

How it was made (4× DGX Spark "encode farm")

One 753B model does not fit a single 128 GB Spark for quantization, and exllamav3 has no multi-GPU convert. We built a rolling pipeline (quant/orchestrate.py):

1. Rolling source shards. The FP8 source is 756 GB and not layer-ordered; the orchestrator keeps a 5-shard download window (sha-checked), runs convert.py for two layers at a time in a fresh container (--max_module L+1, then -r resume), and after each step rewrites finished shards dropping only model.layers.L.mlp.experts.* (norms/attention stay — the compile step needs them). Peak disk ≈ 660 GB; permanent shards = embed/lm_head, final norm, and the layer-78 MTP shard (must exist at every resume).
2. Remote expert-encoding farm (quant/exl3_farm.py + quant/patch_farm.py). exllamav3's per-layer H (Hessian) capture is serial, but the 768 expert matrices per layer are independent once H is final. The patch adds remote work threads to quantize_linears_parallel: it ships each matrix + its finalized H over TCP (length-prefixed torch.save) to worker containers on the other three Sparks, which return trellis/suh/svh/marker. Verified bit-identical to local encoding. One MoE layer: 49 min single-GPU → ~20–22 min on four. Whole model: ~27 wall-clock hours including a power-cut restart.
3. Resume-safe. A power interruption at 60/78 lost nothing: exllamav3 checkpoints after every layer; the in-flight layer is simply redone.
4. Compile. compile.py needs marisa_trie (not in the base image) — pip install marisa-trie in the quant container.

quant/recipe.yaml (58,167 tensor entries) is included; quant/build.sh + quant/patch_aarch64.py build exllamav3 1.4.5 on aarch64/CUDA 13 inside the Mia container image (AVX all-reduce/CPU-MoE stubs + pause intrinsic).

Codebook note (read before you quantize with exllamav3 ≥ 1.4)

convert.py -r (resume) lets the command-line default win over the stored args for the codebook (("codebook", True, "mul1") in the arg table). We passed -cb mcg on the first run only; every resumed 2-layer run silently quantized with mul1. Result: layer 3 is mcg, layers 4–77 are mul1. Quality is identical (both are 3-bit trellis codebooks), but Mia's overlay was mcg-only — which is why serving/exl3.py now detects the codebook per layer and drives the kernels' (mcg, mul1) flag pairs. If you resume an exllamav3 job, re-pass every quant flag.

Serving on 4× DGX Spark (what serving/ contains and why)

Base: ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3 + its start-tp4.sh launcher (github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks). Image layers we add (Dockerfiles included, build order dcp → mul1 → v145 → keys → keys3):

Launcher-level gotchas we hit (all in env.tp4.example comments): --ulimit nofile=1048576 (NCCL "Too many open files" at 4 ranks), a rank whose fabric IP sits on the second CX7 port needs its own WORKER_CX7_IF/IB, the NFS server node must bind-mount its own export at the shared path, gpu_memory_utilization above ~0.82 is refused by vLLM's startup guard (the worker already holds ~14 GiB of CUDA/NCCL state at check time) — use --kv-cache-memory-bytes to size the KV cache explicitly.

Measured (4× GB10, TP=4, DCP=4, CUDA graphs, fp8 KV, MTP k=3)

Production config: MTP k=3 speculator (the checkpoint's own bf16 MTP block), 1M context, idle serve, 256-token outputs, temperature 0.6 / top_p 0.95, thinking off. tok/s is decode only (completion tokens ÷ (wall − TTFT)); aggregate is summed across streams.

TTFT 0.6–2.0 s across C1–C8. Decode rises with depth as the fixed per-step DCP cost amortizes (8→12 tok/s from short context to 1M). Prefill ≈ 320–410 tok/s is the limiter at depth: a 979K prompt is ~52 min to first token. Speculative gain is acceptance-bound — code/list/essay accept 2.3–2.5 draft tokens/step, prose ~1.8 — so 25–30 tok/s single-stream is not reachable on prose with this MTP head.

DFlash2 draft (parked): the full-model DFlash2 draft (incoai/GLM-5.3-DFlash2) can be made to boot at DCP=4 (see serving/ patches: replicate the sliding-window KV per rank, --prefix-match-unit 16), but vLLM has no correct decode-context-parallel path for sliding-window attention, so it generates garbage. Full-model DFlash2 at DCP needs a DCP-aware SWA attention kernel (upstream), not a config change. MTP k=3 is the shipping speculator.

Quick start (4 Sparks)

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks glm53-mia && cd glm53-mia
python3 serving/patch_tp4_launcher.py start-tp4.sh          # LOCAL_MODEL_DIR + ABLIT_HOST_DIR knobs
# build the image chain on EVERY rank (see serving/Dockerfile.* ; ~1 min each on top of the Mia base)
cp serving/env.tp4.example .env.tp4                          # set rank IPs / NIC pins / LOCAL_MODEL_DIR=<this repo on shared storage>
SKIP_PULL=1 SKIP_SHIP=1 SKIP_BUILD=1 SKIP_DOWNLOAD=1 SKIP_SYNC=1 SKIP_OVERLAY_VERIFY=1 ./start-tp4.sh

OpenAI-compatible API on :8888, served model name GLM-5.3-EXL3. Sampling: temperature 1.0, top_p 0.95, thinking on (GLM defaults).

Lessons for "quantize with the serving container in mind"

1. Read the serving overlay's loader first: allowed scope, bits set, tensor suffixes, marker dtype/shape, codebook — and quantize to that ABI, not to the quantizer's defaults.
2. Keep anything the container serves natively (attention, dense, head, MTP) unquantized; the memory cost is small next to the experts and it removes whole classes of patches.
3. Verify tensor suffixes after the first resumed step, not at the end (see the codebook note).
4. Budget memory per rank for weights + KV + the ~14 GiB the process holds before vLLM measures anything; size KV explicitly.
5. A per-layer encode farm turns a multi-day single-GPU convert into an overnight job on the same boxes that will serve the model.

Credits

Zhipu / zai-org for GLM-5.3 · turboderp for exllamav3 · Mia AI Lab for the GB10 EXL3 vLLM container and launcher · Blackfrost-Research for the DERISKED GLM-5.3 (source of the optional ablit donors; not applied in these weights) · Quantized and served by @drowzeys on a 4× DGX Spark cluster, 2026-09.

License: the weights inherit the zai-org/GLM-5.3 license (MIT). Patches in serving/ are MIT; they modify Apache-2.0 vLLM/SGLang/exllamav3 code — see their headers.