SmolLM2-360M-Instruct-GLQ-4bpw

SmolLM2-360M-Instruct GLQ 4bpw

[!IMPORTANT] Superseded by xv0y5ncu/SmolLM2-360M-Instruct-GLQ-trellis-3inst-4bpw. This build predates block-diagonal RHT, so its effective rate is ~6.40 bpw — the pow2 padding was stored as real bits. The trellis (3INST) replacement is a true uniform 4 bpw (0.24 GiB loaded), the recommended codebook since v0.7, with measured quality: wikitext-2 PPL 13.09 vs 12.73 for bf16 (+2.7%). Even the 6 bpw trellis rung (0.31 GiB, PPL within +0.2% of bf16) costs fewer bits than this build's effective rate.

vllm serve xv0y5ncu/SmolLM2-360M-Instruct-GLQ-trellis-3inst-4bpw --quantization glq

These weights remain available and unchanged, so existing references keep working.

SmolLM2-360M-Instruct quantized using GLQ (Golay-Leech Quantization).

Note on effective bpw: This model was quantized with power-of-2 FHT padding. The labeled 4bpw refers to the codebook information content, but effective storage is ~6.4 bpw due to dimensional padding (hidden_size=960 padded to 1024). Quality benchmarks below reflect this effective rate.

Quality (lm-eval 5-task)

Tasks: arc_easy, hellaswag, piqa, winogrande, lambada_openai

Usage

python -m venv .venv
source .venv/bin/activate
pip install 'glq[vllm]'

import glq.hf_integration
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "xv0y5ncu/SmolLM2-360M-Instruct-GLQ-4bpw",
    device_map="cuda",
    dtype="float16",
)
tokenizer = AutoTokenizer.from_pretrained("xv0y5ncu/SmolLM2-360M-Instruct-GLQ-4bpw")

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Requirements

• transformers >= 5.0
• torch >= 2.0
• glq >= 0.2.8 (pip install glq)

License

Apache 2.0

---

🔗 GLQ on GitHub: https://github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.