SmolLM2-135M-Instruct-GLQ-4bpw

SmolLM2-135M-Instruct GLQ 4bpw

[!IMPORTANT] Superseded by xv0y5ncu/SmolLM2-135M-Instruct-GLQ-block-diagonal-4bpw. this build predates block-diagonal RHT, so its effective rate is ~10.07 bpw, not 4.0 — the padding was stored as real bits. The replacement is the same nominal rate with no waste.

vllm serve xv0y5ncu/SmolLM2-135M-Instruct-GLQ-block-diagonal-4bpw --quantization glq

These weights remain available and unchanged, so existing references keep working.

SmolLM2-135M-Instruct quantized using GLQ (Golay-Leech Quantization).

Note on effective bpw: This model was quantized with power-of-2 FHT padding. Effective storage is ~6.4 bpw due to dimensional padding (hidden_size=576 padded to 1024).

Usage

python -m venv .venv
source .venv/bin/activate
pip install 'glq[vllm]'

import glq.hf_integration
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "xv0y5ncu/SmolLM2-135M-Instruct-GLQ-4bpw",
    device_map="cuda",
    dtype="float16",
)
tokenizer = AutoTokenizer.from_pretrained("xv0y5ncu/SmolLM2-135M-Instruct-GLQ-4bpw")

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Requirements

• transformers >= 5.0
• torch >= 2.0
• glq >= 0.2.8 (pip install glq)

License

Apache 2.0

---

🔗 GLQ on GitHub: https://github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.