GLM-5.3-Flash-EXL3-Q8Core-Q3Q4Q6

GLM-5.3-Flash-EXL3-256K-Q8Core-Q3Q4Q6 (4.59 bpw)

This is a custom mixed-precision EXL3 conversion of zai-org/GLM-5.3-BF16, built for ExLlamaV3. It preserves selected high-sensitivity tensors at higher precision while fitting a 256K-token FP16 KV-cache deployment on a two-GPU system.

Overall quantization: 4.59 bits per weight (bpw), as reported by the EXL3 checkpoint metadata.

This is an EXL3 checkpoint. It is intended for ExLlamaV3 and compatible frontends such as TabbyAPI; it is not a Transformers or vLLM checkpoint.

Quantization

The complete tensor mapping is recorded in quantization_config.json. The model's reported average EXL3 bitrate is 4.59 bpw.

The Q4 routed up_proj promotions are layers 3, 11, 16-18, 21, 26, 28-32, 34-35, and 37-44. They include the final sparse layer and additional layers selected from the importance data used by the original mixed-precision plan.

Non-projection tensors retain their appropriate original storage formats. See quantization_config.json for the authoritative per-tensor representation.

Tested Deployment

This release targets a two-GPU, 192GB-class VRAM configuration. It is not a single-GPU release and it does not use tensor parallelism; ExLlamaV3 places consecutive layers across the two GPUs.

The placement budgets above are the explicit ExLlamaV3 split configuration, not a promise of a fixed nvidia-smi reading. Allocator and workspace usage vary by driver, prompt length, and request state.

Measured Interactive Performance

At the 450 W-per-GPU limit above, interactive requests on 69.8K-75.0K-token conversations with 98-100% prompt-cache hits produced:

The prefill number is the rate for the 714-1,708 newly added tokens in otherwise cached conversations. It is not an isolated cold-prefill benchmark and should not be read as a full-context prefill claim.

Runtime Notes

• The checkpoint's 256K profile uses an FP16 KV cache; FP16 is the cache format exposed by ExLlamaV3.
• The model supports vision. The tested deployment keeps vision weights in system RAM with vision_offload: true.
• GPU placement is hardware dependent. The split above was selected to retain long-prompt workspace on GPU 1.

An example TabbyAPI starting point is:

model:
  model_dir: /path/to/parent-directory
  model_name: GLM-5.3-Flash-EXL3-256K-Q8Core-Q3Q4Q6
  backend: exllamav3
  max_seq_len: 262144
  cache_size: 262144
  cache_mode: FP16
  vision: true
  vision_offload: true

draft_model:
  draft_mode: mtp
  draft_cache_mode: FP16
  draft_num_tokens: 3

License and Attribution

This is a derived quantization of Z.AI's GLM-5.3-BF16 release. The upstream license is included in LICENSE and governs use and redistribution of this checkpoint. Please cite and attribute the original GLM-5.3 release when using this model.

Conversion

Converted with ExLlamaV3 1.4.8. This repository contains the complete, self-contained checkpoint: 30 safetensor shards, model and tokenizer configs, and processor assets. There are no external weight links or runtime download requirements.