Downloads · 30 days
581
8% of all-time downloads
lemonyins/Qwen3.8-27B-ULTIMATE-UNCENSORED-MTP-IQ4-GGUF-16GB
Qwen3.8-27B-ULTIMATE-UNCENSORED-MTP-IQ4-GGUF-16GB is a image-text-to-text model from lemonyins. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
♥♥♥♥ This model is specially prepared for your graphics card with 16GB VRAM.
Downloads · 30 days
581
8% of all-time downloads
All-time downloads
7.7K
Public
Repo size
14.9 GB
Likes
8
Public
Click a slice to open those files.
.gguf14.1 GB · 94%
From the Hugging Face model README
** ♥♥♥♥ This model is specially prepared for your graphics card with 16GB VRAM. **
This model is the IQ4_XS GGUF quantization of AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 (an uncensored / abliterated build of Qwen/Qwen3.8-27B), produced via mradermacher's i1 GGUF pipeline (mradermacher/Qwen3.8-27B-heretic-ara-i1-GGUF). The 16G variant is tuned to fit comfortably on a 16GB VRAM GPU (e.g. RTX 4060 Ti 16GB) when run with a TurboQuant KV cache.
This model refers to the fully optimized Qwen3.8-27B-i1-IQ4_XS, which has restored the attn_qkv layer to pure IQ4_XS. Furthermore, we introduce a novel hybrid precision quantization strategy: the FFN layer uses IQ3_S, which achieves significantly smaller file sizes while maintaining core inference capabilities through support from TurboQuant KV caching. Additionally, we use AEON-7's AEON-ULTIMATE-UNCENSORED abliterated version of the base model for quantization, making it convenient for users to conduct in-depth research.
The original llama.cpp quantization heuristics upgrade attn_qkv layers to q5_K under certain conditions (e.g., n_gqa >= 4), causing noticeable file bloat. cHunter789's fix restores attn_qkv to pure IQ4_XS, saving ~375 MiB.
Taking this further: FFN layers (ffn_down, ffn_up, ffn_gate) account for ~2/3 of total model parameters, yet they have higher redundancy than attention layers. Downgrading them from IQ4_XS to IQ3_S is a natural next step — it yields substantial size reduction with minimal quality impact, especially when attention layers (which dominate inference quality) remain at IQ4_XS.
attn_qkv, attn_k, attn_v, attn_output, output: IQ4_XSffn_down, ffn_up, ffn_gate: IQ3_SIQ4_XSllama-quantize.exe ^
--imatrix Qwen3.8-27B-heretic-ara.imatrix.gguf ^
Qwen3.8-27B-heretic-ara-BF16.gguf ^
Qwen3.8-27B-heretic-IQ4_XXS.gguf ^
IQ4_XS 8 ^
--tensor-type "blk.*.ffn_down" iq3_s ^
--tensor-type "blk.*.ffn_up" iq3_s ^
--tensor-type "blk.*.ffn_gate" iq3_s
Note on TurboQuant: This model is recommended to be used with llama.cpp (https://github.com/TheTom/llama-cpp-turboquant) that supports TurboQuant KV caching (release tqp-v0.3.0). TurboQuant allows the KV cache to use a separate, more compact quantization format (turbo4 / turbo3), dramatically reducing memory usage even when the model weights themselves remain at IQ4_XS. Of course, it is also possible to use vllm or other inference frameworks that support TurboQuant technology, but the author used llama.cpp for the test.
| Version | Context | KV Cache | VRAM Usage |
|---|---|---|---|
IQ4_XXS (baseline) | 100K-110K | turbo4 | 15.1-15.3 GB |
IQ4_XXS + MTP | 80K | turbo4 | ~15.3 GB |
draft-mtp): ~80K context, higher throughput thanks to multi-token predictionTested on NVIDIA RTX 4060 Ti 16GB:
| Scenario | Speed |
|---|---|
| IQ4_XXS | 20 tokens/s |
IQ4_XXS + MTP | 30 tokens/s |
This model supports vision input when paired with the Vision Modality Projector (mmproj-BF16.gguf). If you load the vision module, the available context window will be reduced by approximately 20K to accommodate the vision encoder's memory footprint.
You need a TurboQuant-enabled llama.cpp build from TheTom/llama-cpp-turboquant (release tqp-v0.3.0).
Note: This model is quantized from an abliterated / uncensored (AEON-ULTIMATE-UNCENSORED) base model (AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16). The base model removes content restrictions for research and development purposes.
Recommended run command (16GB VRAM, ~100K context):
set TURBO_AUTO_ASYMMETRIC=0
llama-turboquant-3.0\llama-server.exe -m Qwen3.8-27B-uncensored-abliterated-heretic-IQ4-GGUF-16G.gguf -b 2048 -ub 2048 -c 102400 -ngl 999 --flash-attn on -ctk turbo4 -ctv turbo4 --chat-template-file chat_template.jinja --host 0.0.0.0 --port 1234
Note: You must first use set TURBO-AUTO_SYMETRIC=0, otherwise KV will automatically increase to Q8_0.
Note: For visual support, please refer to the loading command in Qwen3.6 27B, but the context needs to be reduced by 20K.
Run command with MTP (speculative decoding, ~80K context):
llama-turboquant-3.0\llama-server.exe -m Qwen3.8-27B-uncensored-abliterated-heretic-IQ4-GGUF-16G.gguf -b 2048 -ub 2048 --parallel 1 --spec-type draft-mtp --spec-draft-n-max 2 -c 81920 -ngl 999 --flash-attn on -ctk turbo4 -ctv turbo4 --chat-template-file chat_template.jinja --host 0.0.0.0 --port 1234
About MTP: Enabling
--spec-type draft-mtp --spec-draft-n-max 2turns on multi-token prediction (speculative decoding) for higher throughput. Because the draft model shares part of the KV cache budget, the usable context window is reduced to ~80K (-c 81920) on a 16GB GPU.
Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16 uncensored / abliterated base model