Downloads · 30 days
58
16% of all-time downloads
Qwen3-8B-nanoquant-GGUF
arelath/Qwen3-8B-nanoquant-GGUF
Qwen3-8B-nanoquant-GGUF is a text generation model from arelath. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
- Status: completed - Model: Qwen/Qwen3-8B - Revision: b968826d9c46dd6066d109eabc6255188de91218 - Candidate run: /workspace/NanoQuant/evidence/029/029-compress-and-benchmark-qwen3-8b - Backend: llama.cpp - Wall time:…
All-time downloads
374
Public
Repo size
2.2 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf2.2 GB · 100%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Access
- Public
- Created
- Jul 25, 2026
- Updated
- Jul 25, 2026
- SHA
- eeb1427c
Try a prompt
Base models
Model card
From the Hugging Face model README
Experiment 29: Qwen/Qwen3-8B quality benchmark
- Status:
completed - Model:
Qwen/Qwen3-8B - Revision:
b968826d9c46dd6066d109eabc6255188de91218 - Candidate run:
/workspace/NanoQuant/evidence/029/029-compress-and-benchmark-qwen3-8b - Backend:
llama.cpp - Wall time: 1769.80 seconds
completed means all evaluators returned finite metrics; it is not a BF16-quality acceptance gate.
Protocol
- WikiText-2: 64 samples × 128 tokens, batch 8
- WikiText token hash:
sha256:e85e3ae8da01294229ac7dfc651f5d44f74eadaff974ad80a5a19779b58fbe0f - Tasks: piqa, arc_easy, arc_challenge, hellaswag, winogrande, boolq; first 200 rows, batch 4
- Tokenizer hash:
sha256:af4e928cfa630ef78563f4800a751d568307a7cd4d4f5fd5919de27b0fbfdbc4
Quality results
| Benchmark | Metric | BF16 | NanoQuant | Delta | Ratio |
|---|---|---|---|---|---|
| WikiText-2 | perplexity ↓ | 23.110475 | 40.599726 | +17.489251 (+75.68%) | 1.7568x |
| piqa | acc_norm ↑ | 0.8000 | 0.5350 | -0.2650 | 0.6687x |
| arc_easy | acc_norm ↑ | 0.8050 | 0.4000 | -0.4050 | 0.4969x |
| arc_challenge | acc_norm ↑ | 0.5350 | 0.3400 | -0.1950 | 0.6355x |
| hellaswag | acc_norm ↑ | 0.6350 | 0.4050 | -0.2300 | 0.6378x |
| winogrande | acc ↑ | 0.7050 | 0.5700 | -0.1350 | 0.8085x |
| boolq | acc ↑ | 0.8550 | 0.6600 | -0.1950 | 0.7719x |
Deployment storage
| Deployable representation | Bytes | Storage savings vs BF16 |
|---|---|---|
| BF16 checkpoint tensors | 16,381,516,776 | — |
| NanoQuant GGUF | 2,214,297,824 | 86.48% |
The packed NanoQuant quantized-layer payload is 884,765,160 bytes. The complete GGUF is the comparable deployable artifact and also includes embeddings, the output projection, norms, and metadata.
Quality execution
The BF16 reference uses Transformers. The NanoQuant candidate is evaluated only from the exported GGUF by the pinned llama.cpp runtime; no reconstructed dense or factorized PyTorch candidate is loaded.
| Runtime | Elapsed seconds | Peak dedicated GPU bytes | Peak host bytes |
|---|---|---|---|
| BF16 Transformers | 34.98 | 18,886,950,912 | 4,983,508,992 |
| NanoQuant llama.cpp GGUF | 1731.46 | unavailable | 2,550,489,088 |
- GGUF:
/workspace/NanoQuant/Results/029/qwen3-8b-nanoquant.gguf - GGUF SHA-256:
sha256:75d3f4f150195e0c6a7ccd2ce3372f919534fb7ec609f6b9ba52eb71af194253 - llama.cpp commit:
2927569f318311aded148d76be6b1109664066f5
Provenance
- Experiment config hash:
sha256:993eca2ef6c345d01926e4677e9cf9620c001fa7e61f1ffa207eeaad1e1803cb - Launcher:
experiments/029-compress-and-benchmark-qwen3-8b.py - Candidate identity:
{"config_hash":"sha256:9356ea7a7e46c6dc5286ce574c110add86e4fd77b9f0c92dc32f4646888f9dd9","model_hash":"sha256:db9a057024370dc56a8bba76b9f274692c2094615dc4da1497ce830d14707bc6","plan_hash":"sha256-bc87a0d973d42d9bc247fd72660dc76c590ffe29fbf918f1aa8db56e87c4dd8a"} - Global tuning:
None
- Task
- Text Generation
- License
- apache-2.0
- Created
- Jul 25, 2026
- Updated
- Jul 25, 2026