Downloads · 30 days
23
6% of all-time downloads
Qwen3-4B-GSQ-Q3_K_M-GGUF
d-alistarh/Qwen3-4B-GSQ-Q3_K_M-GGUF
Qwen3-4B-GSQ-Q3_K_M-GGUF is a machine learning model from d-alistarh. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Quantized with GSQ (Gumbel-Softmax Quantization) using the Q3KM mixed-precision recipe.
All-time downloads
397
Public
Repo size
2.1 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf2.1 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026
- SHA
- cc08c0f4
Base models
Model card
From the Hugging Face model README
Qwen3-4B GSQ Q3_K_M GGUF
Quantized with GSQ (Gumbel-Softmax Quantization) using the Q3_K_M mixed-precision recipe.
Benchmark: WikiText2 Perplexity (llama-perplexity, 512 context)
| Model | Size | PPL | Delta vs F16 |
|---|---|---|---|
| F16 baseline | 7.5 GB | 14.95 | — |
| GSQ (this model) | 1.93 GB | 14.97 | +0.02 (+0.1%) |
| Unsloth Dynamic Q3_K_M | 2.0 GB | 17.36 | +2.40 (+16.1%) |
| Standard Q3_K_M | 2.0 GB | 18.86 | +3.91 (+26.1%) |
Quantization Recipe
Per-projection assignment (uniform across all 36 layers):
- Q3_K: gate_proj, up_proj, q_proj, k_proj
- Q4_K: down_proj, v_proj, o_proj
- Q6_K: embed_tokens
Effective average: ~3.4 bits per weight.
Method
GSQ applies GPTQ initialization followed by 2000 steps of Gumbel-Softmax refinement per linear layer, optimizing quantized weights against the Hessian-weighted reconstruction loss.
- License
- apache-2.0
- Created
- Apr 30, 2026
- Updated
- Apr 30, 2026