GLM-5.3-W4A16

<p align="center"><img src="jangq-logo.png" width="160"/></p>

GLM-5.3-W4A16

GLM-5.3 — ~460 GB on disk (down from the ~1.5 TB BF16 source) — INT4 W4A16 quantization of the routed experts in compressed-tensors pack-quantized format. Loads and runs on Hopper (H100 / H200) via vLLM's Marlin kernels.

• Source: zai-org/GLM-5.3 (GLM-MoE-DSA architecture, 78 layers, 256 routed experts top-8, 1 shared expert, MLA + DeepSeek Sparse Attention, MTP head, 1M context)
• Quantization: INT4 symmetric, group-32, RTN, on the routed-expert weights only (incl. MTP) — packed with the compressed-tensors library's own pack_to_int32 for a bit-exact on-disk layout (weight_packed int32, weight_scale bf16, weight_shape int64)
• Why W4A16 (not NVFP4 / MXFP4): NVFP4 and MXFP4 are Blackwell-native formats and run through slow dequant emulation on Hopper. INT4 W4A16 → Marlin is the mature, fast 4-bit path on H100 / H200. A16 = activations, attention math, and KV cache all stay 16-bit — only expert weight storage is 4-bit
• Attention & router kept high-precision: MLA / DSA attention, dense MLP, the shared expert, router gates, norms, embeddings, and lm_head all stay BF16 — only the 256 routed experts per layer are INT4, and those are ~97% of the parameters
• MTP included: the MTP layer (model.layers.78.*) experts are quantized like every other layer
• Bundle size: ~460 GB
• Runs on: H100 / H200 with vLLM tensor-parallel

What's in the bundle

Dequant per group of 32: w = int4 * weight_scale (symmetric, scale = amax/7.5, range [-8, 7]).

Usage (vLLM)

vllm serve JANGQ-AI/GLM-5.3-W4A16 \
  --tensor-parallel-size 8 \
  --trust-remote-code

vLLM auto-selects the Marlin MoE kernel for compressed-tensors pack-quantized INT4 at group-32 on Hopper.

Method

Converted from the BF16 release with a streaming shard-by-shard quantizer (one tensor in memory at a time — no GPU, no large host RAM). Round-to-nearest INT4 is packed via compressed_tensors.pack_to_int32 so the layout is bit-exact with what vLLM expects; the format was ground-truthed against a known-working DeepSeek-family W4A16 model. This is a data-free RTN quant (no calibration). For maximum quality a calibrated GPTQ/AWQ pass on the routed experts can be run later on-device.

---

Quantized by JANGQ-AI.