Downloads · 30 days
10
18% of all-time downloads
Qwen3-8B-W4A16-AWQ
Rodder5/Qwen3-8B-W4A16-AWQ
Qwen3-8B-W4A16-AWQ is a machine learning model from Rodder5. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Qwen3-8B (pinned revision b968826) quantized with AWQ (llm-compressor, W4A16 scheme: 4-bit weights, 16-bit activations), produced for sm120-quant-bench: a study of where 4-bit damage actually lands, measured with froz…
All-time downloads
55
Public
Parameters
8.2B
6.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors6.1 GB · 100%
Parameter types
How the weights are stored.
I326.9B · 85%
Base models
Model card
From the Hugging Face model README
Qwen3-8B-W4A16-AWQ
Qwen3-8B (pinned revision b968826) quantized with AWQ (llm-compressor, W4A16 scheme: 4-bit weights, 16-bit activations), produced for sm120-quant-bench: a study of where 4-bit damage actually lands, measured with frozen splits, capability probes, and bootstrap 95% CIs on an RTX 5090. Write-up: "NVFP4 on a 5090 names two different machines".
Most quantized checkpoints ship with no damage documentation. This card tells you what this quantization costs, per capability, against the BF16 baseline measured on identical hardware with identical frozen evals.
| Metric | BF16 baseline | This checkpoint |
|---|---|---|
| Perplexity (wikitext-2 slice, lower better) | 17.92 | 19.73 |
| GSM8K strict-match | 91.8 [89.4, 94.2] | 88.8 [86.0, 91.4] |
| HumanEval pass@1 | 63.4 [56.1, 70.7] | 58.5 [51.2, 65.8] |
| MMLU STEM | 71.8 [68.6, 74.8] | 66.8 [63.5, 69.9] |
| MMLU humanities | 62.3 [58.9, 65.6] | 60.2 [56.8, 63.6] |
| Numeric fidelity probe | 78.7 [74.0, 83.3] | 70.0 [65.0, 75.0] |
| Long-context multi-needle (overall) | 97.8 [95.6, 99.4] | 97.2 [94.4, 99.4] |
| Tool-calling end-to-end (300 items) | 87.7 [84.0, 91.3] | 86.3 [82.3, 90.0] |
| Weights on disk (GB) | 16.4 | 6.1 |
| TTFT p50 ms (single-stream, RTX 5090) | 14.8 | 7.7 |
| ITL p50 ms/token | 11.1 | 4.7 |
Damage pools in computation, not structure. Full five-variant table with bootstrap CIs: the study.
Every number traces to a results JSON in the repo, and every eval run recorded which GPU kernel actually served it. This checkpoint's serving receipt on our stack (vLLM 0.26.0, sm_120):
Using MarlinLinearKernel for CompressedTensorsWNA16
Know what you are trading: by perplexity this checkpoint is identical to the GPTQ variant (19.73 vs 19.66). The capability probes disagree: it gives up ~6 points of HumanEval and ~7 of numeric fidelity relative to GPTQ. If your workload leans on code or arithmetic, prefer the GPTQ variant; if not, this one is equivalent and equally fast.
Part of sm120-quant-bench
One of three checkpoints quantized and evaluated together in sm120-quant-bench; siblings: Qwen3-8B-W4A16-GPTQ and Qwen3-8B-NVFP4.
These two checkpoints sit 0.06 perplexity apart and 6.7 points apart on a numeric-fidelity probe. That gap is the study.
Studies using these checkpoints
- NVFP4 on a 5090 names two different machines
- Your agent's tools survive 4-bit. Its token budget and its defaults don't.
How it was made
recipes/w4a16_awq.py (256 chat-templated ultrachat samples, max_seq 1024: the defaults OOM a 32 GB card). Calibration data is disjoint from every eval split by construction. Full recipe, harness, and raw results: the repo. Reproduction gets byte-identical eval items (seed-pinned, hashed manifest).
Serve
vllm serve <this-repo> --max-model-len 16384
Check the log for the kernel-selection line before benchmarking. If a number matters, its serving receipt matters.
- Type
- qwen3
- License
- apache-2.0
- Created
- Aug 8, 2026
- Updated
- Aug 11, 2026