Qwen3-0.6B-quantized

Qwen3-0.6B Quantized Models

This repository contains three quantized versions of the Qwen3-0.6B model, optimized for different use cases and hardware requirements.

Models Included

1. GGUF Q4_K_M (462 MB)

• Format: GGUF (llama.cpp compatible)
• Quantization: 4-bit K-quant (Q4_K_M)
• Best for: CPU inference, llama.cpp/prima.cpp, resource-constrained environments
• File: Qwen3-0.6B-GGUF/Qwen3-0.6B.Q4_K_M.gguf

2. GPTQ-Int4 (517 MB)

• Format: Safetensors (HuggingFace Transformers)
• Quantization: 4-bit GPTQ (group_size=128, symmetric)
• Best for: GPU inference with AutoGPTQ or Transformers
• Quantizer: gptqmodel 4.0.0
• Directory: Qwen3-0.6B-GPTQ-Int4/

3. GPTQ-Int8 (727 MB)

• Format: Safetensors (HuggingFace Transformers)
• Quantization: 8-bit GPTQ (group_size=128, symmetric)
• Best for: Higher accuracy with good compression
• Quantizer: gptqmodel 2.2.0
• Directory: Qwen3-0.6B-GPTQ-Int8/

Model Specifications

Usage

GGUF (llama.cpp / prima.cpp)

# Using prima.cpp
./llama-server -m Qwen3-0.6B-GGUF/Qwen3-0.6B.Q4_K_M.gguf --port 8080

# Using ollama
ollama run qwen3:0.6b

GPTQ (Transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Bopalv/Qwen3-0.6B-quantized",
    subfolder="Qwen3-0.6B-GPTQ-Int4",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "Bopalv/Qwen3-0.6B-quantized",
    subfolder="Qwen3-0.6B-GPTQ-Int4"
)

Quantization Details

Original Model

This is a quantized version of Qwen3-0.6B by Qwen Team.

License

Apache 2.0 (same as base model)