Qwen2.5-Coder-Instruct-GGUF

Qwen2.5-Coder Instruct — GGUF, Q4_K_M

Q4_K_M GGUF builds of Qwen2.5-Coder-Instruct across four sizes, for running code generation locally with llama.cpp.

What these are

Code models. They write and complete source code from a description in plain language. Nothing here is fine-tuned, merged or otherwise altered: these are the original weights, converted and quantised, and they behave as the upstream models do.

Q4_K_M is the usual trade for local inference — close to the full model's quality at roughly a quarter of the memory.

How they were made

Reproducible with two commands from a checkout of llama.cpp:

python convert_hf_to_gguf.py Qwen2.5-Coder-1.5B-Instruct \
    --outtype f16 --outfile qwen2.5-coder-1.5b-instruct-f16.gguf

llama-quantize qwen2.5-coder-1.5b-instruct-f16.gguf \
    qwen2.5-coder-1.5b-instruct-q4_k_m.gguf Q4_K_M

Converted with llama.cpp at 4d91760.

Which size

Bigger writes better code and needs more memory. As a rough guide at Q4_K_M, 0.5B and 1.5B run comfortably on modest hardware, 7B wants around 8 GB free, and 14B around 12 GB.

The 3B is deliberately absent: it is published under the Qwen Research License rather than Apache-2.0, and everything here asks nothing of whoever downloads it.

Licence

Apache-2.0, unchanged from upstream. LICENSE is Qwen's own file, included as the licence requires. The original models are Qwen2.5-Coder by the Qwen team; all credit for the weights is theirs.