Downloads · 30 days
13
17% of all-time downloads
Qwen3-8B-W4A16-GPTQ
Rodder5/Qwen3-8B-W4A16-GPTQ
Qwen3-8B-W4A16-GPTQ is a machine learning model from Rodder5. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Qwen3-8B (pinned revision b968826) quantized with GPTQ (llm-compressor, W4A16 scheme: 4-bit weights, 16-bit activations), produced for sm120-quant-bench: a study of where 4-bit damage actually lands, measured with fro…
All-time downloads
77
Public
Parameters
8.2B
6.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors6.1 GB · 100%
Parameter types
How the weights are stored.
I326.9B · 85%
Base models
Model card
From the Hugging Face model README
Qwen3-8B-W4A16-GPTQ
Qwen3-8B (pinned revision b968826) quantized with GPTQ (llm-compressor, W4A16 scheme: 4-bit weights, 16-bit activations), produced for sm120-quant-bench: a study of where 4-bit damage actually lands, measured with frozen splits, capability probes, and bootstrap 95% CIs on an RTX 5090. Write-up: "NVFP4 on a 5090 names two different machines".
Most quantized checkpoints ship with no damage documentation. This card tells you what this quantization costs, per capability, against the BF16 baseline measured on identical hardware with identical frozen evals.
| Metric | BF16 baseline | This checkpoint |
|---|---|---|
| Perplexity (wikitext-2 slice, lower better) | 17.92 | 19.66 |
| GSM8K strict-match | 91.8 [89.4, 94.2] | 88.4 [85.6, 91.2] |
| HumanEval pass@1 | 63.4 [56.1, 70.7] | 64.6 [57.3, 72.0] |
| MMLU STEM | 71.8 [68.6, 74.8] | 69.0 [65.8, 72.1] |
| MMLU humanities | 62.3 [58.9, 65.6] | 62.6 [59.2, 65.9] |
| Numeric fidelity probe | 78.7 [74.0, 83.3] | 76.7 [71.7, 81.3] |
| Long-context multi-needle (overall) | 97.8 [95.6, 99.4] | 96.7 [93.9, 98.9] |
| Tool-calling end-to-end (300 items) | 87.7 [84.0, 91.3] | 86.0 [82.0, 89.7] |
| Weights on disk (GB) | 16.4 | 6.1 |
| TTFT p50 ms (single-stream, RTX 5090) | 14.8 | 7.7 |
| ITL p50 ms/token | 11.1 | 4.7 |
Damage pools in computation, not structure. Full five-variant table with bootstrap CIs: the study.
Every number traces to a results JSON in the repo, and every eval run recorded which GPU kernel actually served it. This checkpoint's serving receipt on our stack (vLLM 0.26.0, sm_120):
Using MarlinLinearKernel for CompressedTensorsWNA16
Part of sm120-quant-bench
One of three checkpoints quantized and evaluated together in sm120-quant-bench; siblings: Qwen3-8B-W4A16-AWQ and Qwen3-8B-NVFP4.
These two checkpoints sit 0.06 perplexity apart and 6.7 points apart on a numeric-fidelity probe. That gap is the study.
Studies using these checkpoints
- NVFP4 on a 5090 names two different machines
- Your agent's tools survive 4-bit. Its token budget and its defaults don't.
How it was made
recipes/w4a16_gptq.py (512 chat-templated ultrachat samples, max_seq 2048, seed 3407). Calibration data is disjoint from every eval split by construction. Full recipe, harness, and raw results: the repo. Reproduction gets byte-identical eval items (seed-pinned, hashed manifest).
Serve
vllm serve <this-repo> --max-model-len 16384
Check the log for the kernel-selection line before benchmarking. If a number matters, its serving receipt matters.
- Type
- qwen3
- License
- apache-2.0
- Created
- Aug 8, 2026
- Updated
- Aug 11, 2026