GLM-5.3-Int4-Int8Mix-Pollard-GPTQ-MTP-Draft

GLM-5.3 MTP Draft — layer 78, GPTQ int8 attention / int4 experts (standalone)

The model-author multi-token-prediction layer of zai-org/GLM-5.3 (layer 78: full MoE decoder block + eh_proj/enorm/hnorm + shared-head norm), packaged as a standalone speculative draft for vLLM method: mtp: 13 GB, compressed-tensors pack-quantized, with the embeddings and lm_head it needs to load on its own. Quantized with the Pollard-method GPTQ cook (full-Hessian error feedback, per-expert routed Hessians, in-domain calibration) at the same allocation as the int8/int4 production draft it replaces — attention int8 g128, routed experts int4 g128, norms/router/indexer bf16.

Companion body: bot-lab-21/GLM-5.3-Int4-Int8Mix-Pollard-GPTQ (the same file set ships inside it as mtp-draft-gptq-int8/; this repo exists so speculative-config.model can be an HF id).

Why this draft

Same body (int4/int8 GPTQ GLM-5.3), same day, same replay of real traffic, k = 5 adaptive, 8 × DGX Spark:

Hessian rounding on the draft layer alone is worth +22 % acceptance at identical bits, and a draft quantized the same way as its target beats an unquantized bf16 copy of the same weights. The probe drifts ~0.3 day to day on a fixed config; compare within a session. The stock (un-finetuned) head weights are used — six finetuned heads we trained all served worse than stock on this body.

Usage (vLLM ≥ 0.28, with the companion body)

vllm serve bot-lab-21/GLM-5.3-Int4-Int8Mix-Pollard-GPTQ --tensor-parallel-size 8 --trust-remote-code \
  --max-model-len 900000 --kv-cache-dtype fp8 \
  --speculative-config '{"model":"bot-lab-21/GLM-5.3-Int4-Int8Mix-Pollard-GPTQ-MTP-Draft","method":"mtp","quantization":"compressed-tensors","num_speculative_tokens":3,"draft_tensor_parallel_size":1}'

It should also draft for other int4/int8 GLM-5.3 bodies (the head weights are the author's); acceptance will differ with the body's quantization noise — measure with a replay of your own traffic.

Files

draft-00001…00004.safetensors (2,348 tensors: model.layers.78.*, model.embed_tokens.weight, lm_head.*, model.norm.weight), model.safetensors.index.json, config.json (text config, num_nextn_predict_layers: 1, compressed-tensors groups + ignore rules), generation_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja.

License / provenance

Derivative of zai-org/GLM-5.3 (GLM-5.3 License, © 2026 Z.AI). Quantization and measurement by bot-lab-21; quantizer core derived from Pollard Weights' pollard_gptq (Apache-2.0). Produced with the help of an AI assistant operating the cluster.