Qwen2-VL-2B-ternary

Qwen2-VL-2B-ternary

Qwen2-VL-2B-Instruct with the text/language backbone quantized to ternary weights. Vision backbone kept at FP16 for full image quality. 196 transformer layers quantized; weights stored as 2-bit packed ternary codes with per-group float16 scales. Load with ternary-quant for inference on GPU, CPU (auto BF16), or Apple Silicon.

Compression: 1.8× (text backbone only; vision stays FP16) vs FP16 | Quality: coherent output ✓ (image + text verified)

Usage

pip install ternary-quant

from ternary_quant.inference import load_ternary_model

# GPU — fastest (dequantize once, use cuBLAS)
model, tokenizer = load_ternary_model(
    "AsadIsmail/Qwen2-VL-2B-ternary", device="cuda", runtime_mode="cached"
)

# GPU — lowest VRAM (~1.5 GB (use runtime_mode='gemlite' or 'triton'))
model, tokenizer = load_ternary_model(
    "AsadIsmail/Qwen2-VL-2B-ternary", device="cuda", runtime_mode="gemlite"
)

# CPU — auto BF16 on Intel AVX2+
model, tokenizer = load_ternary_model(
    "AsadIsmail/Qwen2-VL-2B-ternary", device="cpu", runtime_mode="cached"
)

# Apple Silicon
model, tokenizer = load_ternary_model(
    "AsadIsmail/Qwen2-VL-2B-ternary", device="mps", runtime_mode="metal"
)

# Generate
from ternary_quant.inference import generate_text
output = generate_text(model, tokenizer, "Ternary quantization works by", max_new_tokens=100)
print(output)

Performance

Quantization details

Quantized with ternary-quant — the only PTQ tool that quantizes VLMs and seq2seq models.