qwen2.5-7b-GGUF

Qwen2.5-7B-Instruct GGUF

Quantized GGUF versions of Qwen/Qwen2.5-7B-Instruct for local inference.

Available Quantizations

Usage

Via Ollama (Easiest)

ollama run qwen2.5:7b

Via llama.cpp

./llama-cli -m qwen2.5-7b-instruct-Q4_K_M.gguf -p "Your prompt here" -n 512

Via Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="paijo77/qwen2.5-7b-GGUF",
    filename="qwen2.5-7b-instruct-Q4_K_M.gguf",
    n_ctx=8192,
    n_gpu_layers=-1  # use GPU if available
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain quantum computing simply"}]
)
print(response["choices"][0]["message"]["content"])

Via Open WebUI

1. Download the GGUF file
2. In Open WebUI → Models → Add model
3. Point to local GGUF file

Why Qwen2.5-7B?

• Multilingual: English, Chinese, 29+ languages
• Long context: 128K tokens natively
• Coding: Excellent code generation
• Math: Strong mathematical reasoning
• Instruction following: Clean, structured outputs
• Size: Runs on 6GB VRAM or 8GB RAM (CPU)

Hardware Requirements

Support This Project

Quantization takes compute and time. If this helps you: 👉 https://www.tip.md/oyi77

License

Apache 2.0 — based on Qwen2.5 (Apache 2.0)