Downloads · 30 days
27
23% of all-time downloads
weili-0234/Qwen3.5-27B-MXFP4-QAD-W4A4-LR1e-5-s2640
Qwen3.5-27B-MXFP4-QAD-W4A4-LR1e-5-s2640 is a text generation model from weili-0234. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
MXFP4 checkpoint of Qwen/Qwen3.5-27B after 1,000 steps of W4A4 quantization-aware distillation (QAD, weights and activations fake-quantized during training), exported to vLLM compressed-tensors format. One artifact se…
Downloads · 30 days
27
23% of all-time downloads
All-time downloads
118
Public
Parameters
15.9B
19 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors18.9 GB · 100%
How the weights are stored.
U812.9B · 81%
From the Hugging Face model README
MXFP4 checkpoint of Qwen/Qwen3.5-27B after 1,000 steps of W4A4
quantization-aware distillation (QAD, weights and activations fake-quantized
during training), exported to vLLM compressed-tensors format. One artifact
serves both modes: W4A4 (native MXFP4 activations, FlashInfer, SM100+, default
load) and W4A16 (--linear-backend marlin).
| Producing repo | tonyzhang-together/QATFactory branch weili/qwen35-27b @ 8e7cf1a |
| Base model | Qwen/Qwen3.5-27B (dense qwen3_5, bf16) |
| Training | QAD distillation from the BF16 base model as teacher: temperature-1.0 pure-KL loss, assistant-only masking, W4A4 fake-quant (MXFP4 weights E2M1 block-32 EVEN-rule E8M0 scales + dynamic MXFP4 activation fake-quant) on all attention + MLP linears. lr 1.0e-5 cosine→0 (1% warmup), global batch 32 (micro 4 × 8 GPU × accum 1), seq 8192, bf16 FSDP2 full-shard, seed 42, 2,640-step (≈1-epoch) schedule. Data: OpenPerfectBlend-derived ChatML distillation corpus, 100k conversations (train jsonl md5 0406bb3a7a482352360716a1bc5e9e04), held-out eval split disjoint. Hardware: 1×8 B200 (jbom-5c-01). |
| Step | Global step 2640 of 2640 — the completed arm (exit 0, 2026-07-24, ~22.3h on 1×8 B200). |
| Provenance | Clean end-to-end run (v4): every periodic save (s250..s2500) and the final save completed without incident; final held-out eval_fwd_kl 0.03163 vs the BF16 teacher. |
| Role in the study | s2640 waypoint of the MX-W4A4 arm of the stage-3 Qwen3.5-27B 2×2 study (W4A4-training vs W4A16-training × MXFP4 vs NVFP4). PTQ baselines: weili-0234/Qwen3.5-27B-MXFP4-RTN, weili-0234/Qwen3.5-27B-NVFP4-RTN. |
# on the training node, from the completed run's final model files:
python scripts/export_mxfp4_vllm.py \
--source /scratch/wxu/mxfp4/outputs/27b-mxfp4-w4a4-lr1.0e-5-2640 \
--model-assets Qwen3.5-27B \
--output exports/27b-mxw4a4-s2640 --device cuda:0
Quantized modules: all attention + MLP linears (q/k/v/o_proj, gate/up/down_proj); embeddings/lm_head/norms bf16. MXFP4 activation quantization at serve time is fully dynamic — no calibration state exists in this artifact.
vllm serve weili-0234/Qwen3.5-27B-MXFP4-QAD-W4A4-LR1e-5-s2640 # W4A4 (SM100+)
vllm serve weili-0234/Qwen3.5-27B-MXFP4-QAD-W4A4-LR1e-5-s2640 --linear-backend marlin # W4A16
<!-- eval-results-pending -->
Benchmark rows for this final checkpoint (KL vs BF16, GSM8K, GPQA-D, MMLU-Pro,
plus the 7-benchmark matrix: RULER, AIME'25 avg@4, HumanEval, IFEval,
Pile-10k/WikiText) are being measured and will be added here. The s1000
waypoint of this same run
(weili-0234/Qwen3.5-27B-MXFP4-QAD-W4A4-LR1e-5-s1000) already beats the RTN
baseline on every measured metric (KL 0.0559 vs 0.0907; GSM8K 92.8 vs 92.0;
GPQA-D 80.3 vs 79.3; MMLU-Pro 85.2 vs 83.3). Held-out training-eval KL of this
final checkpoint: 0.03163 (vs 0.0322 at s1000) on the distillation eval split.