Downloads · 30 days
58
6% of all-time downloads
merileijona/Qwen3.5-27B-IQ4_XS-GGUF
Qwen3.5-27B-IQ4_XS-GGUF is a machine learning model from merileijona. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Quantized GGUF version of Qwen/Qwen3.5-27B, converted and quantized to IQ4XS format (~13 GB) for CPU inference.
Downloads · 30 days
58
6% of all-time downloads
All-time downloads
894
Public
Repo size
14.8 GB
Likes
0
Public
Click a slice to open those files.
.gguf14.8 GB · 100%
From the Hugging Face model README
Quantized GGUF version of Qwen/Qwen3.5-27B, converted and quantized to IQ4_XS format (~13 GB) for CPU inference.
Quantized by @merileijona — GitHub: juhanimerilehto
| Property | Value |
|---|---|
| Format | IQ4_XS |
| Approx. size | ~13 GB |
| Base model | Qwen/Qwen3.5-27B |
| Converter | llama.cpp |
IQ4_XS is an importance-matrix quant (imatrix). It uses calibration data to allocate bits where they matter most, giving better quality at the same size compared to standard K-quants.
This quantization is intended for local, CPU-only inference on high-RAM workstations where GPU VRAM is insufficient to run the full model. It has not been formally benchmarked. The settings and usage notes below reflect the actual configuration used during testing.
from llama_cpp import Llama
llm = Llama(
model_path="model_IQ4_XS.gguf",
n_gpu_layers=0, # 0 = CPU-only
n_ctx=4096,
n_threads=16,
verbose=False,
)
response = llm(
"Your prompt here",
max_tokens=2048,
temperature=0.7,
top_p=0.9,
min_p=0.01,
)
print(response["choices"][0]["text"])
| Setting | Value |
|---|---|
n_gpu_layers | 0 (CPU-only) |
n_ctx | 4096 |
n_threads | 16 |
temperature | 0.7 |
top_p | 0.9 |
min_p | 0.01 |
max_tokens | 2048 |
Test hardware:
Token generation speed was not formally measured. The model ran stably at the settings above with no observed repetition or loop issues.
min_p=0.01 is recommended to prevent token loops at longer outputsn_gpu_layers to offload layers to VRAM