Qwen3-1.7B-GGUF

Qwen3-1.7B GGUF

This repository contains GGUF quantizations of the original Qwen3-1.7B model.

The models were converted from the original Hugging Face checkpoint into GGUF format and quantized using llama.cpp, making them compatible with local inference engines such as llama.cpp, llama-cpp-python, Ollama, LM Studio, Jan, and Open WebUI.

---

Original Model

• Model: Qwen/Qwen3-1.7B
• Developer: Qwen Team
• Original Repository: https://huggingface.co/Qwen/Qwen3-1.7B
• License: Apache-2.0

This repository contains only GGUF conversions of the original model.

---

Available Quantizations

---

Which Model Should I Download?

If you're unsure, download Qwen3-1.7B-Q4_K_M.gguf. It provides the best balance between quality, memory usage, and inference speed.

---

Download

Download the required .gguf file from the Files and versions section of this repository.

---

Usage with llama.cpp

llama-cli \
  -m Qwen3-1.7B-Q4_K_M.gguf \
  -p "Explain quantum computing in simple words."

---

Usage with Python (llama-cpp-python)

Install

pip install llama-cpp-python

Example

from llama_cpp import Llama

llm = Llama(
    model_path="Qwen3-1.7B-Q4_K_M.gguf",
    n_ctx=4096,
    n_threads=8,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "user",
            "content": "Explain quantum computing in simple words."
        }
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response["choices"][0]["message"]["content"])

---

Usage with Ollama

Create a file named Modelfile

FROM ./Qwen3-1.7B-Q4_K_M.gguf

Create the model

ollama create qwen3-1.7b-q4 -f Modelfile

Run it

ollama run qwen3-1.7b-q4

---

Compatible Software

These GGUF models can be used with:

• llama.cpp
• llama-cpp-python
• Ollama
• LM Studio
• Jan
• Open WebUI

Note: GGUF models are not compatible with the Hugging Face transformers library. For Transformers, download the original model from the Qwen repository.

---

Quantization Workflow

Original Hugging Face Model (.safetensors)
                │
                ▼
convert_hf_to_gguf.py
                │
                ▼
FP16 GGUF
                │
                ▼
llama-quantize
                │
                ▼
Q3_K_M
Q4_K_M
Q5_K_M
Q6_K
Q8_0

---

Credits

• Original Model: Qwen Team
• Original Repository: https://huggingface.co/Qwen/Qwen3-1.7B
• GGUF Conversion & Quantization: Sayujaya Sharma
• Quantization Tool: llama.cpp

---

License

This repository distributes quantized versions of the original Qwen3-1.7B model.

The original model is licensed under Apache-2.0, and this repository follows the same license.

Please refer to the original model repository for complete licensing details.

---

⭐ If you find this repository useful, consider giving it a ⭐ on Hugging Face.