SmolLM2-135M-Instruct-GLQ-block-diagonal-4bpw

SmolLM2-135M-Instruct-GLQ-block-diagonal-4bpw

GLQ (E8-lattice) 4 bpw quantization of HuggingFaceTB/SmolLM2-135M-Instruct, using a block-diagonal randomized Hadamard transform — no power-of-2 FHT padding, so the stored size matches the nominal rate (a genuine 4 bpw, ~111 MB).

vs the earlier xv0y5ncu/SmolLM2-135M-Instruct-GLQ-4bpw: that checkpoint zero-padded non-power-of-2 dimensions up to the next power of two for the FHT, which inflated the effective rate to ~10 bpw (~191 MB). This block-diagonal version stores only the real weights — smaller, with an accurate bpw label. At equal storage, block-diagonal also yields higher SQNR, so it is a strict per-bit improvement.

Quality — lm-evaluation-harness 5-task

GLQ and bf16 measured on the same stack (vLLM backend, transformers 5.x).

Tasks: arc_easy, hellaswag, piqa, winogrande, lambada_openai (acc_norm where available, else acc). Per-task ratios slightly above 100% are within lm-eval noise.

Details

• Bits per weight: 4 bpw (uniform) — block-diagonal, so effective is approximately nominal.
• Checkpoint size: ~111 MB (bf16 original ~270 MB).
• Average SQNR: 18.6 dB.
• Calibration: WikiText-2, 128 samples, sequence length 2048.
• Method: E8 lattice codebook + randomized Hadamard transform (block-diagonal) + LDLQ.

Usage

python -m venv .venv
source .venv/bin/activate
pip install 'glq[vllm]'

import glq.hf_integration  # registers GLQ with transformers
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "xv0y5ncu/SmolLM2-135M-Instruct-GLQ-block-diagonal-4bpw"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, device_map="cuda", dtype=torch.float16)
ids = tok("The capital of France is", return_tensors="pt").to("cuda")
print(tok.decode(model.generate(**ids, max_new_tokens=20)[0], skip_special_tokens=True))

Serve with vLLM: vllm serve xv0y5ncu/SmolLM2-135M-Instruct-GLQ-block-diagonal-4bpw --quantization glq.

License

Apache 2.0. Original model: HuggingFaceTB/SmolLM2-135M-Instruct. Derivative quantization work; the base model and the GLQ tooling (github.com/cnygaard/glq) are both Apache 2.0.

---

🔗 GLQ on GitHub: https://github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.