Qwen3-4B-GGUF

Llamacpp imatrix Quantizations of Qwen3-4B by Qwen

Using <a href="https://github.com/ggml-org/llama.cpp/">llama.cpp</a> at commit 9a3bf2b for quantization.

Original model: https://huggingface.co/Qwen/Qwen3-4B

All quants made using the imatrix option with a bilingual (English + Chinese) and code-heavy calibration dataset, so low-bit quants keep more of their Chinese and coding ability than a English-only calibration would.

Run them in LM Studio, Ollama, or directly with llama.cpp and any llama.cpp based project.

Prompt format

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Download a file (not the whole branch) from below:

The .imatrix file used to produce these quants is included in this repo, so anyone can reproduce or extend the quant set with the exact same importance matrix.

Downloading using the hf CLI

<details> <summary>Click to view download instructions</summary>

First make sure you have the CLI installed:

pip install -U "huggingface_hub[cli]"

Then target the specific file you want (do not clone the whole repo, it is large):

hf download qtum/Qwen3-4B-GGUF --include "Qwen3-4B-Q4_K_M.gguf" --local-dir ./

</details>

Which file should I choose?

<details> <summary>Click here for details</summary>

A great write up with charts showing various performances is provided by Artefact2 here.

The first thing to figure out is how big a model you can run. For that you need to know how much RAM and/or VRAM you have.

If you want the model running as FAST as possible, fit the whole thing in your GPU's VRAM: pick a quant with a file size 1-2GB smaller than your total VRAM.

If you want maximum quality, add your system RAM and your GPU's VRAM together, then pick a quant 1-2GB smaller than that total.

Next, decide between an 'I-quant' and a 'K-quant'.

If you don't want to think about it, grab a K-quant, in the format QX_K_X such as Q5_K_M.

If you want to dig deeper, see the llama.cpp feature matrix.

In short: if you are targeting below Q4 and running cuBLAS (Nvidia) or rocBLAS (AMD), look at the I-quants, in the format IQX_X such as IQ3_M. They are newer and offer better quality for their size. I-quants also work on CPU, but are slower than their K-quant equivalent, so it is a speed-vs-quality tradeoff.

</details>

Credits

Quantization pipeline built on llama.cpp by ggml-org.

Thanks to bartowski and kalomaze for establishing the imatrix calibration practice this pipeline follows.