Downloads · 30 days
11
1% of all-time downloads
sokann/Qwen3.6-27B-GGUF-4.262bpw
Qwen3.6-27B-GGUF-4.262bpw is a machine learning model from sokann. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This is a 4.262 BPW quantized model for the GPU poors with 16 GiB of VRAM. It uses the SOTA IQK quants, and thus works in ikllama.cpp only.
Downloads · 30 days
11
1% of all-time downloads
All-time downloads
1.2K
Public
Repo size
14.3 GB
Likes
4
Public
Click a slice to open those files.
.gguf14.3 GB · 100%
From the Hugging Face model README
This is a 4.262 BPW quantized model for the GPU poors with 16 GiB of VRAM. It uses the SOTA IQK quants, and thus works in ik_llama.cpp only.
It was quantized using the simplest ever recipe – Q8_0 for the tiny ssm_alpha and ssm_beta tensors, IQ4_KS for the rest.
From local testing with llama-perplexity (wiki.test.raw, 580 chunks), it has the best quality and speed in the same size class:
| quant | this | bartowski Q3_K_M | unsloth UD-Q3_K_XL | mradermacher i1.IQ4_XS | bartowski IQ4_XS | unsloth IQ4_XS |
|---|---|---|---|---|---|---|
| Size (BPW) | 4.262 | 4.270 | 4.302 | 4.483 | 4.556 | 4.589 |
| Size (GiB) | 13.344 | 13.370 | 13.469 | 14.036 | 14.266 | 14.369 |
| VRAM usage (GiB) | 12.714 | 12.861 | 12.803 | 13.407 | 13.637 | 13.703 |
| Mean PPL(Q) | 7.005465 ± 0.046240 | 6.993009 ± 0.046208 | 6.995519 ± 0.046227 | 7.020660 ± 0.046587 | 6.996323 ± 0.046332 | 6.950126 ± 0.045846 |
| Mean PPL(base) | 6.908506 ± 0.045543 | 6.908506 ± 0.045543 | 6.908506 ± 0.045543 | 6.908506 ± 0.045543 | 6.908506 ± 0.045543 | 6.908506 ± 0.045543 |
| Cor(ln(PPL(Q)), ln(PPL(base))) | 99.29% | 98.52% | 98.82% | 99.30% | 99.32% | 99.38% |
| Mean KLD | 0.029103 ± 0.000627 | 0.058818 ± 0.000881 | 0.046348 ± 0.000841 | 0.027289 ± 0.000660 | 0.026270 ± 0.000653 | 0.024728 ± 0.000603 |
| Maximum KLD | 22.648993 | 24.616274 | 24.175169 | 18.568180 | 22.992002 | 21.687405 |
| 99.9% KLD | 2.372350 | 3.986622 | 3.614290 | 2.667850 | 2.385293 | 2.201674 |
| RMS Δp | 4.675 ± 0.053 % | 6.690 ± 0.059 % | 5.867 ± 0.060 % | 4.449 ± 0.057 % | 4.352 ± 0.057 % | 4.264 ± 0.056 % |
| Same top p | 92.884 ± 0.067 % | 90.350 ± 0.077 % | 91.829 ± 0.071 % | 93.903 ± 0.062 % | 93.888 ± 0.062 % | 93.997 ± 0.062 % |
With 16 GiB of VRAM, we can fit a context size of 75776 with quantized KV cache:
# ik_llama.cpp
-c 75776 -ctk q8_0 -khad -ctv q8_0 -vhad -wgt 1
For brave souls that seek the TurboQuant experience (see #1033, #1034, #1527), we can also fit a context size of 139264 with more heavily quantized KV cache:
# ik_llama.cpp
-c 139264 -ctk q4_0 -khad -ctv q4_0 -vhad -wgt 1
Size from llama-server output:
llm_load_print_meta: model size = 13.344 GiB (4.262 BPW)
llm_load_print_meta: repeating layers = 12.084 GiB (4.262 BPW, 24.353 B parameters)
...
llm_load_tensors: CUDA_Host buffer size = 645.09 MiB
llm_load_tensors: CUDA0 buffer size = 13018.97 MiB
<details>
<summary>Recipe</summary>
blk\..*\.attn_q\.weight=iq4_ks
blk\..*\.attn_k\.weight=iq4_ks
blk\..*\.attn_v\.weight=iq4_ks
blk\..*\.attn_output\.weight=iq4_ks
blk\..*\.attn_gate\.weight=iq4_ks
blk\..*\.attn_qkv\.weight=iq4_ks
blk\..*\.ssm_alpha\.weight=q8_0
blk\..*\.ssm_beta\.weight=q8_0
blk\..*\.ssm_out\.weight=iq4_ks
blk\..*\.ffn_down\.weight=iq4_ks
blk\..*\.ffn_(gate|up)\.weight=iq4_ks
token_embd\.weight=iq4_ks
output\.weight=iq4_ks
</details>
llama-sweep-bench result with a RTX 3090, with flags -ngl 99 -mqkv -muge -cuda graphs=1 -c 128000 -wgt 1 -wb:
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|---|---|---|---|---|---|---|
| 512 | 128 | 0 | 0.340 | 1503.86 | 2.808 | 45.58 |
| 512 | 128 | 10240 | 0.382 | 1341.67 | 2.975 | 43.02 |
| 512 | 128 | 20480 | 0.425 | 1203.93 | 3.060 | 41.83 |
| 512 | 128 | 30720 | 0.466 | 1098.45 | 3.152 | 40.61 |
| 512 | 128 | 40960 | 0.508 | 1007.98 | 3.267 | 39.18 |
| 512 | 128 | 51200 | 0.550 | 930.55 | 3.376 | 37.91 |
| 512 | 128 | 61440 | 0.595 | 860.55 | 3.465 | 36.94 |
| 512 | 128 | 71680 | 0.640 | 799.61 | 3.562 | 35.93 |
| 512 | 128 | 81920 | 0.681 | 751.90 | 3.682 | 34.76 |
| 512 | 128 | 92160 | 0.725 | 705.94 | 3.793 | 33.74 |
| 512 | 128 | 102400 | 0.769 | 666.22 | 3.881 | 32.98 |
| 512 | 128 | 112640 | 0.814 | 629.36 | 3.984 | 32.13 |
| 512 | 128 | 122880 | 0.857 | 597.34 | 4.101 | 31.21 |
This quant uses the imatrix from mradermacher. It performs well enough in long reasoning tasks and agentic tasks.