qwen1.5-1.8b-gguf

sultanali338/qwen1.5-1.8b-gguf

This repository contains GGUF quantized versions of Qwen/Qwen1.5-1.8B.

Quantized Models

Usage

Using llama.cpp

# Download the model
curl -L -O https://huggingface.co/sultanali338/qwen1.5-1.8b-gguf/resolve/main/Q4_K_M.gguf

# Run inference
./llama-cli -m Q4_K_M.gguf -p "Your prompt here" -n 128

Using Python

from huggingface_hub import hf_hub_download

# Download model
model_path = hf_hub_download(
    repo_id="sultanali338/qwen1.5-1.8b-gguf",
    filename="Q4_K_M.gguf"
)

# Use with llama-cpp-python
from llama_cpp import Llama

llm = Llama(model_path=model_path)
response = llm("Your prompt here", max_tokens=128)

Model Details

• Base Model: Qwen/Qwen1.5-1.8B
• Quantization Method: GGUF/llama.cpp
• Compatible Frameworks: llama.cpp, llama-cpp-python, text-generation-webui, and more

Performance Notes

• Q4_K_M: Recommended for most use cases (good balance of size and quality)
• Q5_K_M: Higher quality, slightly larger
• Q6_K: Near-original quality, larger size
• Q8_0: Highest quantized quality, largest size

Choose the quantization level based on your hardware constraints and quality requirements.