Downloads · 30 days
154
26% of all-time downloads
JJ48-24/GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq8c-int4g32-mixA
GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq8c-int4g32-mixA is a text generation model from JJ48-24. Use it when you need the model to write or continue text. The card lists the license as mit.
Derived from wtdcode/GLM-5.3-Flash-AWQ-W4A16 @ abd7b07 (INT4 AWQ experts, compressed-tensors pack-quantized), itself a quantization of zai-org/GLM-5.3-Flash (MIT). This repo changes only the tensors listed under "What…
Downloads · 30 days
154
26% of all-time downloads
All-time downloads
600
Public
Parameters
334B
175 GB on disk
Likes
3
Public
Click a slice to open those files.
.safetensors175 GB · 100%
How the weights are stored.
I32333B · 100%
From the Hugging Face model README
Derived from wtdcode/GLM-5.3-Flash-AWQ-W4A16 @ abd7b07 (INT4 AWQ experts, compressed-tensors pack-quantized), itself a quantization of zai-org/GLM-5.3-Flash (MIT).
This repo changes only the tensors listed under "What changed"; requant_info.json lists every quantized tensor.
Status: JKeys LLC's production checkpoint on 4x CMP 170HX since 2026-09-15. Requantized on the box from -aggr-w8-mtpq8c in
four minutes with the mixA recipe.
-aggr-w8-mtpq8cINT4, symmetric, group size 32 (group_2) on the shared experts of every MoE layer (129 tensors), the dense MLPs of layers 0-2 (9) and
lm_head (1), converted from the INT8 per-channel tensors. Everything else, including the INT8 per-channel MTP head, is -mtpq8c.
| configuration | tok/s all-20 | prose | code | math | json |
|---|---|---|---|---|---|
| DFlash2 k=7 + rolling rule, full stack, 6-stage Marlin MoE kernel (2026-09-16) | 122.1 | 71.7 | 132.7 | 133.3 | 157.9 |
| same with the shipped 4-stage kernel (2026-09-15) | 118.5 | ||||
| built-in MTP head, k=3 schedule + rule (no external drafter) | 97.9 | 74.2 | 105.3 | 100.4 | 117.4 |
static PP4 baseline the work started from (-mtpq8c, k=7 static, no patches) | 104.0 | 51.0 | 116.6 | 136.4 |
Public Spec-Bench suite (vllm bench serve --dataset-name spec_bench, 80 prompts, 256 output tokens, temperature 0), full stack:
single-stream 91.2 tok/s (TPOT p50 9.6 ms, TTFT p50 188 ms); concurrency 4: 142.7; 8: 164.7; 16: 163.9 with micro-batch cap 2 and 198.0
with cap 4. Prefill ~2.4k tok/s serial; ~8 concurrent 23k-token requests fit the KV pool.
Acceptance rate of the drafter and the degeneracy probe unchanged vs -mtpq8c; the mixA recipe's perplexity cost was measured on the
-aggr-w8 base at +0.58 %. Long-context single-fact recall is identical to -mtpq8c (a model property, see Quality below).
| repo | what is quantized beyond the source | role |
|---|---|---|
GLM-5.3-Flash-AWQ-W4A16-kda-w8-experimental | INT8 per-channel: KDA projections, shared experts, dense MLPs of layers 0-2 | first experiment, superseded |
GLM-5.3-Flash-AWQ-W4A16-aggr-w8 | INT8 per-channel: the whole BF16 remainder incl. MLA projections and lm_head (407 tensors) | base of the stack |
GLM-5.3-Flash-AWQ-W4A16-aggr-w8-int4g32-mixA | aggr-w8 + INT4 g32 shared experts, dense MLPs 0-2, lm_head | +2.6 % decode, ppl +0.58 % |
GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq | aggr-w8 + MTP-head (layer 45) experts INT4 g128 | superseded by -mtpq8c |
GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq8 | aggr-w8 + MTP-head experts INT8 g128 | superseded by -mtpq8c |
GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq8c | aggr-w8 + MTP-head experts INT8 per-channel (Marlin-friendly) | fastest built-in speculation |
GLM-5.3-Flash-AWQ-W4A16-aggr-w8-mtpq8c-int4g32-mixA | -mtpq8c + the mixA INT4 g32 set | the production checkpoint |
Hardware: 4x NVIDIA CMP 170HX (GA100-class, sm_80, HBM2e unlocked from the firmware's 8 GB to 64 GB per card with the community
cmpunlocker, memory clock 1728 MHz, PCIe Gen2 x4, 150 W cap), pipeline-parallel PP4, partition
13,12,12,8, one request at a time unless noted. Software: vLLM fork image ghcr.io/pixelml/club-170hx:vllm-glm53-sm80-pp-20260905
(vllm-openai:glm53-flash @ 487ecf187 with the PixelML sm80 patches) plus JKeys LLC's own env-gated patches, all of which default off:
rolling speculative-k rule, sparse-attention split mode fill, packed pipeline hops, a compiled attention path, and a 6-stage
Marlin MoE pipeline. Settings that matter on this hardware:
VLLM_DISABLED_KERNELS=AllSparkLinearKernel # Marlin W8A16 for the INT8 projections (+1.7 % over AllSpark here)
VLLM_MLA_SPARSE_SPLIT_MODE=fill # JKeys patch: the sparse-attention split kernel fills all 70 SMs (+3.6-4.2 %)
VLLM_PP_PACKED_HOP=1 # JKeys patch: one PCIe transfer per pipeline hop instead of five (-1.07 ms/step)
VLLM_GLM5_COMPILE=1 VLLM_USE_BREAKABLE_CUDAGRAPH=0 # JKeys patch: torch.compile of the model with full CUDA graphs (-0.38 ms/step)
VLLM_PP_MAX_DECODE_REQS_PER_BATCH=2 # micro-batch cap: about concurrency / pipeline depth (2 for 8 sequences, 4 for 16)
--enable-prefix-caching --mamba-cache-mode align # works on this hybrid model; hits land on whole 4608-token blocks
Speculative decoding: the checkpoint's own MTP head (num_speculative_tokens 3 with the rolling rule) needs nothing else.
The faster numbers use the DFlash2 block drafter (incoai/GLM-5.3-Flash-DFlash2,
CC BY-NC-ND 4.0); it is not included here and is not redistributed in any form; JKeys uses it under a separate written licence.
Weight-only, round-to-nearest, no calibration data. Teacher-forced perplexity on unseen text (22,168 tokens of code and 2026 prose)
against the source checkpoint on the same host: aggr-w8 -0.16 %, mixA +0.58 %, kda-w8 +0.32 % (gate: below +1 %). The MTP-head
variants change only the drafter, and rejection sampling keeps the target distribution, so their outputs are the source checkpoint's;
only the draft acceptance rate can move. Known limit of the model, not the quantization: a single fact placed early in a ~23k-token
prompt is often lost because the sparse attention selects the top 2048 tokens per query; retrieval that places the relevant lines near
the question fixes it (12/12 vs 0/12 in our test), raising index_topk to cover the whole prompt also fixes it at ~2x prefill time.
Model: Z.ai (zai-org/GLM-5.3-Flash, MIT). INT4 AWQ expert quantization: wtdcode. sm80 vLLM image: PixelML. Everything else in
this repo (the INT8/INT4 remainder recipes, the MTP-head quantization, the measurements) is JKeys LLC, 2026, released under the same MIT
terms as the base model. The DFlash2 drafter belongs to Inco and is not part of this repo. Tooling and receipts: contact JKeys LLC.