Qwen3-1.7B-ternary

Qwen3-1.7B-ternary

Qwen3-1.7B quantized to ternary weights using ternary-quant broad track. Transformer layers quantized to ~4 effective bits per weight; embeddings stay FP16. Measured 97.5% average benchmark retention across ARC-Easy, ARC-Challenge, HellaSwag, and WinoGrande (0-shot lm-eval).

Compression: 2.7× vs FP16 | Quality: 97.5% avg lm-eval retain (ARC/HellaSwag/WinoGrande)

Usage

pip install ternary-quant

from ternary_quant.inference import load_ternary_model

# GPU — fastest (dequantize once, use cuBLAS)
model, tokenizer = load_ternary_model(
    "AsadIsmail/Qwen3-1.7B-ternary", device="cuda", runtime_mode="cached"
)

# GPU — lowest VRAM (~1.0 GB (use runtime_mode='gemlite' or 'triton'))
model, tokenizer = load_ternary_model(
    "AsadIsmail/Qwen3-1.7B-ternary", device="cuda", runtime_mode="gemlite"
)

# CPU — auto BF16 on Intel AVX2+
model, tokenizer = load_ternary_model(
    "AsadIsmail/Qwen3-1.7B-ternary", device="cpu", runtime_mode="cached"
)

# Apple Silicon
model, tokenizer = load_ternary_model(
    "AsadIsmail/Qwen3-1.7B-ternary", device="mps", runtime_mode="metal"
)

# Generate
from ternary_quant.inference import generate_text
output = generate_text(model, tokenizer, "Ternary quantization works by", max_new_tokens=100)
print(output)

Performance

Quantization details

Quantized with ternary-quant — the only PTQ tool that quantizes VLMs and seq2seq models.