GLM-5.3-Flash-AWQ-W4A16-aggr-w8

GLM-5.3-Flash AWQ-W4A16 + INT8 per-channel remainder (aggr profile)

Derived from wtdcode/GLM-5.3-Flash-AWQ-W4A16 @ abd7b07 (INT4 AWQ experts, compressed-tensors pack-quantized), itself a quantization of zai-org/GLM-5.3-Flash (MIT). This repo changes only the tensors listed under "What changed"; requant_info.json lists every quantized tensor.

Status: the base of JKeys LLC's production stack on 4x CMP 170HX (2026-09-15). Validated on 4x A100-SXM4-80GB (perplexity -0.16 % vs the source), measured daily on the 170HX box since.

What changed vs the source

Weight-only INT8, symmetric, per-channel (pack-quantized, group_1), on the BF16 remainder that dominates per-token weight traffic in single-stream decode: KDA projections (q,k,v,b,f_a,g_a,o), MLA projections (q_a/q_b/kv_a/kv_b/o) and the sparse-attention indexer's linears are left BF16 where the kernels require it, shared experts, dense MLPs of layers 0-2, lm_head. 407 tensors, 12.2 GB BF16 -> 6.1 GB. Routed INT4 experts, embeddings, norms, conv, router, hyper-connections and the MTP layer are untouched.

Why

Single-stream decode is weight-streaming bound: ~22 GB of weights per token on the source, only ~4.7 GB of it the INT4 experts. Shrinking the BF16 remainder removes a constant number of milliseconds from every decode step on pipeline-parallel topologies. It does not help tensor-parallel layouts, where the remainder is sharded and the step is all-reduce bound.

Measured

A100-SXM4 PP4, single stream, same host and harness: speculation off 52.9 -> 65.5 tok/s (+23.7 %). On the 170HX the MTP head of this checkpoint is BF16; for the fastest built-in speculation use -aggr-w8-mtpq8c, and with an external block drafter the MTP head is unused, so this checkpoint decodes like -mtpq8c (115.5 tok/s all-20 on the full stack before the mixA step, see the family table).

The family

Serving (what the numbers below were measured with)

Hardware: 4x NVIDIA CMP 170HX (GA100-class, sm_80, HBM2e unlocked from the firmware's 8 GB to 64 GB per card with the community cmpunlocker, memory clock 1728 MHz, PCIe Gen2 x4, 150 W cap), pipeline-parallel PP4, partition 13,12,12,8, one request at a time unless noted. Software: vLLM fork image ghcr.io/pixelml/club-170hx:vllm-glm53-sm80-pp-20260905 (vllm-openai:glm53-flash @ 487ecf187 with the PixelML sm80 patches) plus JKeys LLC's own env-gated patches, all of which default off: rolling speculative-k rule, sparse-attention split mode fill, packed pipeline hops, a compiled attention path, and a 6-stage Marlin MoE pipeline. Settings that matter on this hardware:

VLLM_DISABLED_KERNELS=AllSparkLinearKernel   # Marlin W8A16 for the INT8 projections (+1.7 % over AllSpark here)
VLLM_MLA_SPARSE_SPLIT_MODE=fill              # JKeys patch: the sparse-attention split kernel fills all 70 SMs (+3.6-4.2 %)
VLLM_PP_PACKED_HOP=1                         # JKeys patch: one PCIe transfer per pipeline hop instead of five (-1.07 ms/step)
VLLM_GLM5_COMPILE=1 VLLM_USE_BREAKABLE_CUDAGRAPH=0   # JKeys patch: torch.compile of the model with full CUDA graphs (-0.38 ms/step)
VLLM_PP_MAX_DECODE_REQS_PER_BATCH=2          # micro-batch cap: about concurrency / pipeline depth (2 for 8 sequences, 4 for 16)
--enable-prefix-caching --mamba-cache-mode align   # works on this hybrid model; hits land on whole 4608-token blocks

Speculative decoding: the checkpoint's own MTP head (num_speculative_tokens 3 with the rolling rule) needs nothing else. The faster numbers use the DFlash2 block drafter (incoai/GLM-5.3-Flash-DFlash2, CC BY-NC-ND 4.0); it is not included here and is not redistributed in any form; JKeys uses it under a separate written licence.

Quality

Weight-only, round-to-nearest, no calibration data. Teacher-forced perplexity on unseen text (22,168 tokens of code and 2026 prose) against the source checkpoint on the same host: aggr-w8 -0.16 %, mixA +0.58 %, kda-w8 +0.32 % (gate: below +1 %). The MTP-head variants change only the drafter, and rejection sampling keeps the target distribution, so their outputs are the source checkpoint's; only the draft acceptance rate can move. Known limit of the model, not the quantization: a single fact placed early in a ~23k-token prompt is often lost because the sparse attention selects the top 2048 tokens per query; retrieval that places the relevant lines near the question fixes it (12/12 vs 0/12 in our test), raising index_topk to cover the whole prompt also fixes it at ~2x prefill time.

Credits and licence

Model: Z.ai (zai-org/GLM-5.3-Flash, MIT). INT4 AWQ expert quantization: wtdcode. sm80 vLLM image: PixelML. Everything else in this repo (the INT8/INT4 remainder recipes, the MTP-head quantization, the measurements) is JKeys LLC, 2026, released under the same MIT terms as the base model. The DFlash2 drafter belongs to Inco and is not part of this repo. Tooling and receipts: contact JKeys LLC.