Downloads · 30 days
750
13% of all-time downloads
peasantsmith/Qwen3.8-Flash-Next-PS-IQ2_XXS-GGUF
Qwen3.8-Flash-Next-PS-IQ2_XXS-GGUF is a machine learning model from peasantsmith. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for gguf.
176.9B parameters in a single 75.2 GB file — 3.40 bits/param. An IQ2XXS-class GGUF of Qwen3.8-Flash-Next, built from the original Qwen/Qwen3.8-Flash-Next BF16 weights, with every component quantized to the precision i…
Downloads · 30 days
750
13% of all-time downloads
All-time downloads
6K
Public
Repo size
76.1 GB
Likes
16
Public
Click a slice to open those files.
.gguf76.1 GB · 100%
From the Hugging Face model README
176.9B parameters in a single 75.2 GB file — 3.40 bits/param. An IQ2_XXS-class GGUF of Qwen3.8-Flash-Next, built from the original Qwen/Qwen3.8-Flash-Next BF16 weights, with every component quantized to the precision it needs instead of crushed flat.
Please read: After running it for days non-stop I have concluded that this quantization is pretty good, but it does require certain care. I recommend running it with Thinking: Disabled or Low. You can definitely use xHigh or Medium, but I won't recommend them on hours of uninterrupted agentic work since you may risk loops or simply wasting time on overthinking. For short/medium tasks using thinking as xHigh is completely fine.
The model's single largest component is a per-layer-embedding (PLE) n-gram table: 51.2B params (~102 GB in BF16) — roughly 28% of the model. A naive 2-bit quant crushes it along with everything else, making the model's biggest component its weakest. This build takes the 2-bit win where it's safe and holds the PLE table at the quality floor the ecosystem agrees on:
| Model component | Params | Quant | Size |
|---|---|---|---|
| PLE n-gram table | 51.2B | IQ4_NL (4.5 bpw) | 26.8 GiB |
| Routed expert gate/up | — | IQ2_XXS (2.06 bpw) | 19.3 GiB |
| Routed expert down | — | Q4_0 (4.5 bpw) | 21.1 GiB |
| Attention (QKV) | — | Q4_K | 0.5 GiB |
| Token embeddings | — | Q6_K | 0.5 GiB |
| Output head | — | Q8_0 | 0.8 GiB |
| Routers / norms / biases / hash constants | — | F32 (kept) | — |
Vision encoder (mmproj, 334 tensors) | — | F16 | 0.9 GiB |
| Test | Result |
|---|---|
| Full-context generation (64k context window) | ✅ coherent at depth, no degradation |
| Long-generation stability | ✅ ≥3000 tokens continuous at steady t/s, no degradation |
Tool calling (get_weather JSON function call) | ✅ well-formed {"city":"Paris"} tool call returned |
| Long-context recall (~6k token prompt, needle) | ✅ number recalled correctly |
| LRU cache code task (7.5k token prompt) | ✅ complete, correct implementation |
| Greedy smoke ("The capital of France is") | ✅ "Paris" (finish=stop) |
| Vision via mmproj (image color ID) | ✅ correct |
| Perplexity (wikitext-2) | not measured — honesty note below |
Honesty note: PPL was not run — at this weight class the BF16 reference does not fit the same GPU stack, so no comparable baseline exists on this hardware. The task battery above is the quality evidence; 2-bit experts do carry a real quality delta vs BF16 by construction.
| File | Size | Notes |
|---|---|---|
Qwen3.8-Flash-Next-IQ2_XXS.gguf | 75.2 GB | text model (176.9B params) |
mmproj-Qwen3.8-Flash-Next-F16.gguf | 0.90 GB | vision encoder, F16, all 334 tensors — nothing stripped |
Generic llama.cpp invocation — tune the two offload values to your own hardware:
llama-server \
-m Qwen3.8-Flash-Next-IQ2_XXS.gguf \
--mmproj mmproj-Qwen3.8-Flash-Next-F16.gguf \
-ngl <layers that fit your VRAM> \
--n-cpu-moe <experts to spill to CPU RAM if VRAM is tight> \
--jinja
The model ships its chat template (--jinja). Requires a recent llama.cpp with qwen4exp architecture support.
Qwen/Qwen3.8-Flash-Next — official BF16 safetensors, original weights (no third-party quant or modified repo as source). Model credit: Qwen.convert_hf_to_gguf.pyllama-quantize importance-matrix quant with the per-component split above. Imatrix credit: unsloth.SHA-256 (text): 2e0f14e7eeddce8f80fc88cf96a9cc641b4f60549318fb53e30af85649883586
SHA-256 (vision): f5f125bdfa3997011693e7fe59fae1a5c068c40a627a0576d72734a9cdab9691
mmproj file