Downloads · 30 days
16
3% of all-time downloads
Qwen3-4B-Instruct-2507-GPTQ-W4A16
Subalzt/Qwen3-4B-Instruct-2507-GPTQ-W4A16
Qwen3-4B-Instruct-2507-GPTQ-W4A16 is a text generation model from Subalzt. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A GPTQ W4A16 (4-bit weight, 16-bit activation, group size 128) quantization of Qwen/Qwen3-4B-Instruct-2507, produced with llm-compressor. Calibrated on 512 sequences of pile-10k
All-time downloads
486
Public
Parameters
4B
2.7 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors2.7 GB · 100%
Parameter types
How the weights are stored.
I323.6B · 90%
Try a prompt
Base models
Model card
From the Hugging Face model README
Qwen3-4B-Instruct-2507 — GPTQ W4A16 (INT4)
A GPTQ W4A16 (4-bit weight, 16-bit activation, group size 128) quantization of Qwen/Qwen3-4B-Instruct-2507, produced with llm-compressor. Calibrated on 512 sequences of pile-10k at seqlen 2048.
Quality (measured)
wikitext-2-raw-v1, ctx 2048, scored under transformers, paired per chunk:
| model | perplexity | vs BF16 | size |
|---|---|---|---|
| BF16 (base) | 10.0216 | — | 7.5 GB |
| GPTQ W4A16 (this) | 10.3261 | +3.04 % | 2.48 GB |
| AWQ W4A16 | 10.5907 | +5.68 % | 3.21 GB |
At 4-bit, expect a real quality cost (+3 %) in exchange for 3× smaller than BF16. In this study GPTQ beat AWQ (+3.0 % vs +5.7 %, paired p = 3e-35) and was the more robust method on 8 GB Blackwell — see the benchmark for the calibration caveat.
Speed & VRAM (RTX 5070 Laptop, 8 GB, sm_120)
| metric | value |
|---|---|
| VRAM to load | ~2.7 GB (incl. ~1.3 GB CUDA context) |
| weights on disk | 2.48 GB |
| decode speed | 58.0 tok/s (single stream, vLLM) |
The leanest and fastest of the three quants — smallest VRAM footprint and fastest decode (4-bit means fewer weight bytes to read per token). The choice when VRAM is tight and a few % perplexity is acceptable.
Usage (vLLM)
from vllm import LLM
llm = LLM("Subalzt/Qwen3-4B-Instruct-2507-GPTQ-W4A16")
print(llm.generate("Write a Python function for binary search.")[0].outputs[0].text)
Provenance & reproducibility
Full method, calibration ablation, and comparisons: github.com/Subalzt/qwen3-4b-quantized-fp8-gptq.
License
Apache-2.0, inherited from the base model. Attribution: Qwen team,
Qwen/Qwen3-4B-Instruct-2507.
- Task
- Text Generation
- Type
- qwen3
- License
- apache-2.0
- Languages
- en
- Created
- Aug 9, 2026
- Updated
- Aug 10, 2026