Downloads · 30 days
30
6% of all-time downloads
georuffus/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF
Qwen3.8-27B-i1-IQ4_KS_KT-GGUF is a image-text-to-text model from georuffus. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This quantization was created to allow the entire model to fit into the memory of an NVIDIA graphics card with 16GB of VRAM.
Downloads · 30 days
30
6% of all-time downloads
All-time downloads
482
Public
Repo size
14.8 GB
Likes
1
Public
Click a slice to open those files.
.gguf14.8 GB · 100%
From the Hugging Face model README
This quantization was created to allow the entire model to fit into the memory of an NVIDIA graphics card with 16GB of VRAM.
This model has the token_emd.weight tensor bumped up to q8_0, which increases its download size but makes no difference in VRAM. By default, the token_emd.weight tensor is loaded entirely into system RAM. Upgrading it to q8_0 is therefore, in practice, a free upgrade to the model's quality, for example in interpreting non-English languages.
This repository contains GGUF format weights for the Qwen3.8-27B model, quantized using the ik_llama.cpp project.
This model was specifically created to run on consumer GPUs with 16GB VRAM. By utilizing q4_0 KV cache quantization, it allows pushing the context length up to 110k tokens.
Note: This model is designed exclusively for nVidia GPUs and is based on the advanced KS and KT quants developed by ikawrakow from the ik_llama.cpp repository.
GPU Only full 16GB VRAM the configuration below will give you a 105k context (110k is the max) (q4_0/q4_0). Decode speed starts at 45 t/s and falls to 25 t/s at the end, with a prefill speed above 1300 t/s (RTX 5070 Ti).
llama-server" \
-m "$MODEL_PATH" \
-a Qwen3.8-27B \
--ctx-size 105000 \
--n-gpu-layers 99 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--spec-type ngram-mod:n_max=2 \
--batch-size 512 \
--ubatch-size 128 \
--flash-attn on \
--host 0.0.0.0 \
--port 8081 \
--reasoning on \
--reasoning-format none \
--reasoning-budget 32000 \
-t 8 \
-tb 8 \
--parallel 1 \
--metrics \
--merge-qkv \
-khad \
-vhad \
--chat-template-kwargs '{"preserve_thinking": true, "reasoning_effort": "medium"}' \
--defrag-thold 0.4 \
--jinja \
--cont-batching \
--temp 0.6 \
--top-k 20 \
--min-p 0.05 \
--top-p 0.95 \
--presence-penalty 0.0 \
--repeat-last-n 512 \
--repeat-penalty 1.05
I compared the model by mradermacher with the model by cHunter789 using the War and Peace text file as a dataset.
Test dataset download:
wget [https://www.gutenberg.org/files/2600/2600-0.txt](https://www.gutenberg.org/files/2600/2600-0.txt) -O pg19.txt
Source: mradermacher/Qwen3.8-27B-i1-GGUF
./llama-perplexity \
-m Qwen3.8-27B.i1-IQ4_XS.gguf \
-f pg19.txt \
-c 65536 \
--chunks 32 \
-ngl 99 \
-ctk q4_0 \
-ctv q4_0 \
-khad \
-vhad \
--merge-qkv \
-fa 1 \
-b 512 \
-ub 128 \
-t 8
perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 77.21 seconds per pass - ETA 15.43 minutes
[1]7.7001,[2]8.0855,[3]8.2723,[4]8.3528,[5]8.5027,[6]8.3485,[7]8.3948,[8]8.3789,[9]8.4068,[10]8.4465,[11]8.4932,[12]8.2955,
Final estimate: PPL over 12 chunks for n_ctx=65536 = 8.2955 +/- 0.03221
llama_print_timings: load time = 2522.03 ms
llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: prompt eval time = 750310.64 ms / 786432 tokens ( 0.95 ms per token, 1048.14 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 1360215.26 ms / 786433 tokens
Source: cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF
./llama-perplexity \
-m Qwen3.8-27B.i1-IQ4_KT-attn_qkv-IQ4_KS.gguf \
-f pg19.txt \
-c 65536 \
--chunks 32 \
-ngl 99 \
-ctk q4_0 \
-ctv q4_0 \
-khad \
-vhad \
--merge-qkv \
-fa 1 \
-b 512 \
-ub 128 \
-t 8
perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 84.14 seconds per pass - ETA 16.82 minutes
[1]7.6895,[2]8.0682,[3]8.2542,[4]8.3333,[5]8.4853,[6]8.3416,[7]8.3890,[8]8.3734,[9]8.3999,[10]8.4386,[11]8.4857,[12]8.2890,
Final estimate: PPL over 12 chunks for n_ctx=65536 = 8.2890 +/- 0.03224
llama_print_timings: load time = 9369.56 ms
llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: prompt eval time = 783243.61 ms / 786432 tokens ( 1.00 ms per token, 1004.07 tokens per second)
llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 2323646.53 ms / 786433 tokens