Downloads · 30 days
244
9% of all-time downloads
wanhin/qwen2.5-7b-instruct-q4-k-m-gguf
qwen2.5-7b-instruct-q4-k-m-gguf is a machine learning model from wanhin. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Qwen2.5-7B-Instruct model quantized to Q4KM (4-bit, medium quality)
Downloads · 30 days
244
9% of all-time downloads
All-time downloads
2.8K
Public
Repo size
4.7 GB
Likes
0
Public
Click a slice to open those files.
.gguf4.7 GB · 100%
From the Hugging Face model README
Qwen2.5-7B-Instruct model quantized to Q4_K_M (4-bit, medium quality)
wget https://huggingface.co/your-username/qwen2.5-7b-instruct-q4-k-m-gguf/resolve/main/qwen2.5-7b-instruct-q4-k-m-gguf.gguf
# With llama.cpp
./main -m qwen2.5-7b-instruct-q4-k-m-gguf.gguf -n 512
# With Python
python -c "
from llama_cpp import Llama
llm = Llama(model_path='./qwen2.5-7b-instruct-q4-k-m-gguf.gguf')
print(llm('Hello!', max_tokens=100)['choices'][0]['text'])
"
This quantized model provides a good balance between model quality and inference speed.