Downloads · 30 days
1.1K
3% of all-time downloads
lemonyins/Qwen3.6-27B-abliterated-i1-IQ4_XS-GGUF-Smaller
Qwen3.6-27B-abliterated-i1-IQ4_XS-GGUF-Smaller is a image-text-to-text model from lemonyins. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
♥♥♥♥ This model is specially prepared for your graphics card with 16GB VRAM.
Downloads · 30 days
1.1K
3% of all-time downloads
All-time downloads
40.6K
Public
Repo size
28.6 GB
Likes
16
Public
Click a slice to open those files.
.gguf28.6 GB · 100%
From the Hugging Face model README
** ♥♥♥♥ This model is specially prepared for your graphics card with 16GB VRAM. **
This model is equivalent to https://huggingface.co/lemonyins/Qwen3.6-27B-uncensored-abliterated-i1-IQ4_XS-GGUF-Smaller
This model refers to the fully optimized Qwen3.6-27B-i1-IQ4_XS, which has restored the attn_qkv layer to pure IQ4_XS. Furthermore, we introduce a novel hybrid precision quantization strategy: the FFN layer uses IQ3_S, which achieves significantly smaller file sizes while maintaining core inference capabilities through support from TurboQuant KV caching. Additionally, we use an Huihui abliterated version of the base model for quantization, making it convenient for users to conduct in-depth research.
The original llama.cpp quantization heuristics upgrade attn_qkv layers to q5_K under certain conditions (e.g., n_gqa >= 4), causing noticeable file bloat. cHunter789's fix restores attn_qkv to pure IQ4_XS, saving ~375 MiB.
Taking this further: FFN layers (ffn_down, ffn_up, ffn_gate) account for ~2/3 of total model parameters, yet they have higher redundancy than attention layers. Downgrading them from IQ4_XS to IQ3_S is a natural next step — it yields substantial size reduction with minimal quality impact, especially when attention layers (which dominate inference quality) remain at IQ4_XS.
attn_qkv, attn_k, attn_v, attn_output, output: IQ4_XSffn_down, ffn_up, ffn_gate: IQ3_SIQ4_XSOption A — Pure IQ4_XS baseline:
llama-quantize.exe \
--imatrix Huihui-Qwen3.6-27B-abliterated.imatrix.gguf \
Huihui-Qwen3.6-27B-abliterated-BF16.gguf \
Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS.gguf \
IQ4_XS
Option B — Recommended: IQ4_XS-FFN-IQ3_S (smaller + longer context):
llama-quantize.exe ^
--imatrix Huihui-Qwen3.6-27B-abliterated.imatrix.gguf ^
Huihui-Qwen3.6-27B-abliterated-BF16.gguf ^
Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS-FFN-IQ3_S.gguf ^
IQ4_XS ^
--tensor-type "blk.*.ffn_down" iq3_s ^
--tensor-type "blk.*.ffn_up" iq3_s ^
--tensor-type "blk.*.ffn_gate" iq3_s
Note on TurboQuant: This model is recommended to be used with llama.cpp (https://github.com/TheTom/llama-cpp-turboquant) that supports TurboQuant KV caching. TurboQuant allows the KV cache to use a separate, more compact quantization format (turbo4 / turbo3), dramatically reducing memory usage even when the model weights themselves remain at IQ4_XS. Of course, it is also possible to use vllm or other inference frameworks that support TurboQuant technology, but the author used llama.cpp for the test.
| Version | Context | KV Cache | VRAM Usage |
|---|---|---|---|
IQ4_XS (baseline) | 60K | turbo4 | 15.3 GB |
IQ4_XS (baseline) | 80K | turbo3 | 15.3 GB |
IQ4_XS-FFN-IQ3_S | 160K | turbo4 | 15.4 GB |
IQ4_XS-FFN-IQ3_S | 200K | turbo3 | 15.3 GB |
IQ4_XS, so the model's reasoning and instruction-following capabilities are largely preservedTested on NVIDIA RTX 4060 Ti 16GB:
| Scenario | Speed |
|---|---|
| Text-only inference | 18-20 tokens/s |
This model supports vision input when paired with the Vision Modality Projector (mmproj-BF16.gguf). If you load the vision module, the available context window will be reduced by approximately 10K to accommodate the vision encoder's memory footprint.
Example with vision support:
llama-server.exe ^
-m Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS-FFN-IQ3_S.gguf ^
-mmproj mmproj-BF16.gguf ^
-c 153840 ^
-ngl 99 ^
--flash-attn on ^
--cache-type-k turbo4 ^
--cache-type-v turbo4 ^
--host 0.0.0.0
Test using Chinese novel:
| Model Version | Perplexity (PPL) | Quality Drop |
|---|---|---|
| Q4_K_M | 13.1909 +/- 0.06037 | Baseline |
| IQ4_XS | 13.2138 +/- 0.06054 | 0.17% |
| IQ4_XS-FFN-IQ3_S | 13.6056 +/- 0.06159 | 3.14% |
Test with code:
| Model Version | Perplexity (PPL) | Quality Drop |
|---|---|---|
| IQ4_XS | 1.2217 +/- 0.00156 | Baseline |
| IQ4_XS-FFN-IQ3_S | 1.2324 +/- 0.00158 | 0.87% |
You need a TurboQuant-enabled llama.cpp build from TheTom/llama-cpp-turboquant.
Note: This model is quantized from an abliterated (uncensored) base model. The base model removes content restrictions for research and development purposes.
Recommended: IQ4_XS-FFN-IQ3_S version — 160K context on ~15 GB VRAM
llama-server.exe ^
-m Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS-FFN-IQ3_S.gguf ^
-c 163840 ^
-ngl 99 ^
--flash-attn on ^
--cache-type-k turbo4 ^
--cache-type-v turbo4 ^
--host 0.0.0.0
Alternative: Pure IQ4_XS version — shorter context, slightly larger file
llama-server.exe ^
-m Huihui-Qwen3.6-27B-abliterated-i1-IQ4_XS.gguf ^
-c 65536 ^
-ngl 99 ^
--flash-attn on ^
--cache-type-k turbo4 ^
--cache-type-v turbo4 ^
--host 0.0.0.0
attn_qkv → IQ4_XS fix and the original fully optimized IQ4_XS GGUF