Downloads · 30 days
78
17% of all-time downloads
Qwen3.8-27B-GGUF
KissMyShinyArse/Qwen3.8-27B-GGUF
Qwen3.8-27B-GGUF is a machine learning model from KissMyShinyArse. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
UPDATE: Q5CR, Q6CR and Q8CR have been added to https://github.com/TheTom/llama-cpp-turboquant
All-time downloads
459
Public
Repo size
31.5 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf31.5 GB · 100%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 21, 2026
- Updated
- Nov 12, 2026
- SHA
- c5908b09
Base models
Model card
From the Hugging Face model README
UPDATE: Q5_CR, Q6_CR and Q8_CR have been added to https://github.com/TheTom/llama-cpp-turboquant
What it is
This quant is based on Unsloth's UD-Q8_K_XL quant, with the Q8 weights swapped for Q8_CR (ConvRot).
⚠️ Proof of concept - CUDA only.
| Quant | Size (GiB) | PPL (Q) | PPL Ratio | ΔPPL | Mean KLD | RMS Δp (%) | Same Top-p (%) |
|---|---|---|---|---|---|---|---|
| UD-Q8_K_XL-Q8_to_Q8_CR (this one) | 29.30 | 6.9565 | 1.00089 | 0.0062 | 0.00037 | 0.587 | 99.263 |
| Q8_CR | 27.05 | 6.9585 | 1.00118 | 0.0082 | 0.00043 | 0.598 | 99.099 |
| UD-Q8_K_XL (Unsloth, baseline) | 29.30 | 6.9537 | 1.00049 | 0.0034 | 0.00083 | 0.823 | 98.992 |
| Q8_0 | 27.05 | 6.9560 | 1.00082 | 0.0057 | 0.00095 | 0.942 | 98.742 |
How to use it
Build llama.cpp with the patch from this repo:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 749f688fcaa4c472ec034b08cb8a907c45cfaa02
patch -Np1 < /path/to/q8_cr.patch
cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON .
cmake --build build --config Release
This quantization is CUDA-only: inference requires an NVIDIA GPU. On a system
without CUDA, model loading fails with failed to find a compatible buffer type. For full speed, offload all layers to the GPU (-ngl 99). Partial
offload works, but every rotated matmul still executes on the GPU (activations
and weights are copied per layer), so generation becomes extremely slow.
Quantizing BF16 to Q8_CR
Quantize the BF16 GGUFs using the patched llama-quantize:
llama-quantize Qwen3.8-27B-BF16-00001-of-00002.gguf Qwen3.8-27B_Q8_CR.gguf Q8_CR
Research
See RESEARCH.md.
- License
- apache-2.0
- Created
- Aug 21, 2026
- Updated
- Nov 12, 2026