Downloads · 30 days
13
14% of all-time downloads
weili-0234/Qwen3.5-27B-MXFP4-GPTQ
Qwen3.5-27B-MXFP4-GPTQ is a text generation model from weili-0234. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
GPTQ (Hessian-corrected PTQ) MXFP4 quantization of Qwen/Qwen3.5-27B in vLLM compressed-tensors mxfp4-pack-quantized format. This is the strong PTQ baseline of the stage-3 W4A4-training study — same lattice and serving…
Downloads · 30 days
13
14% of all-time downloads
All-time downloads
94
Public
Parameters
15.9B
19 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors18.9 GB · 100%
How the weights are stored.
U812.9B · 81%
From the Hugging Face model README
GPTQ (Hessian-corrected PTQ) MXFP4 quantization of Qwen/Qwen3.5-27B in vLLM
compressed-tensors mxfp4-pack-quantized format. This is the strong PTQ
baseline of the stage-3 W4A4-training study — same lattice and serving path
as weili-0234/Qwen3.5-27B-MXFP4-RTN, but with GPTQ weight error correction
against a fixed calibration set. One artifact serves both modes: W4A4
(native MXFP4 activations, SM100+, default load) and W4A16
(--linear-backend marlin).
| Tooling | llm-compressor 0.12.0 (GPTQModifier), torch 2.10.0+cu128, transformers 5.9.0, Python 3.12.3 |
| Base model | Qwen/Qwen3.5-27B (dense qwen3_5, bf16) |
| Recipe | GPTQModifier(targets="Linear", scheme="MXFP4", ignore=["lm_head", "re:.*visual.*", "re:.*mtp.*", "re:.*embed_tokens.*"]) — identical module scope to the RTN and QAD checkpoints in this series (all attention + MLP linears; GDN conv1d untouched — GPTQModifier matches nn.Linear only) |
| Calibration | Fixed 512-conversation set calib_512.jsonl (md5 d1ff8cce1d785f8e51b71eb237fb7a71): random.Random(42).sample(range(n_rows), 512), sorted, from the study's OpenPerfectBlend-derived ChatML train corpus (train jsonl md5 0406bb3a7a482352360716a1bc5e9e04); rendered with the model's stock chat template, add_generation_prompt=False; max_seq_length=8192, num_calibration_samples=512 |
| Script | experiments/w4a4-qwen35-27b/jbom/gptq_27b.py in the study workspace — mirrors the stage-2 9B GPTQ job (experiments/mxfp4-qwen35-9b/k8s/gptq-mxfp4-job.yaml, which produced weili-0234/Qwen3.5-9B-MXFP4-GPTQ) |
| Hardware | 1×B200 (jbom-03), ~2.2 h wall |
| Role in the study | Strong-PTQ baseline row of the stage-3 Qwen3.5-27B 2×2 (W4A4-training vs W4A16-training × MXFP4 vs NVFP4). Companions: Qwen3.5-27B-MXFP4-RTN (weak PTQ), Qwen3.5-27B-MXFP4-QAD-W4A4-LR1e-5-s1000 (QAD). |
# gptq_27b.py MXFP4 (llmcompressor==0.12.0, transformers==5.9.0)
from datasets import load_dataset
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
model = Qwen3_5ForConditionalGeneration.from_pretrained("Qwen/Qwen3.5-27B", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-27B")
ds = load_dataset("json", data_files="calib_512.jsonl", split="train")
recipe = GPTQModifier(
targets="Linear",
scheme="MXFP4",
ignore=["lm_head", "re:.*visual.*", "re:.*mtp.*", "re:.*embed_tokens.*"],
)
oneshot(model=model, tokenizer=tokenizer, dataset=ds, recipe=recipe,
max_seq_length=8192, num_calibration_samples=512)
model.save_pretrained("Qwen3.5-27B-MXFP4-GPTQ", save_compressed=True)
tokenizer.save_pretrained("Qwen3.5-27B-MXFP4-GPTQ")
vllm serve weili-0234/Qwen3.5-27B-MXFP4-GPTQ # W4A4 (SM100+)
vllm serve weili-0234/Qwen3.5-27B-MXFP4-GPTQ --linear-backend marlin # W4A16
<!-- eval-results-kld -->
Mean top-20 token KL vs the BF16 teacher (Qwen/Qwen3.5-27B) over a held-out 256-conversation corpus (818,944 scored tokens), vLLM on 1×B200, W4A4 serving.
| checkpoint | KL vs BF16 | top-1 agree |
|---|---|---|
| MXFP4-QAD-W4A4-s1000 (1,000 steps QAD) | 0.0559 | 0.940 |
| this repo (MXFP4-GPTQ) | 0.0698 | 0.929 |
| MXFP4-RTN | 0.0907 | — |
GPTQ recovers ~23% of RTN's KL gap for MXFP4 with no training; W4A4 QAD training remains ahead.
<!-- eval-results-kld -->A 7-benchmark evaluation (RULER, AIME'25 avg@4, GPQA-Diamond, MMLU-Pro, HumanEval, IFEval, Pile-10k/WikiText perplexity) against the RTN and QAD checkpoints is in progress and will be added to this card.