LFM2.5-2.6B-GPTQ-hybrid-b-g32

LFM2.5-2.6B — GPTQ mixed-precision (hybrid-b-g32)

Weight-only quantization of LiquidAI/LFM2.5-2.6B: 5394 MB → 2142 MB (2.52×), VRAM for weights 5394 MB → ~2.1 GB.

Produced with block-sequential GPTQ, an LFM2-aware mixed-precision recipe, activation ordering, and an EN/JA-balanced calibration set.

⚠️ This is not a stock GPTQ checkpoint

AutoModelForCausalLM.from_pretrained will not load this repo. Weights use the forge-weightonly-v1 format — integer codes packed into a uint8 stream at exactly bits bits per weight, with per-layer widths in quant_config.json. Use the loader shipped in this repo:

from quant_loader import load_quantized          # needs qlinear.py, same repo

model, tok = load_quantized("aoiandroid/LFM2.5-2.6B-GPTQ-hybrid-b-g32")

msgs = [{"role": "user", "content": "量子化とは何ですか。"}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=256)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Requires transformers >= 5.2 (for Lfm2ForCausalLM), torch, safetensors. Weights stay packed in VRAM and are dequantized to BF16 inside forward(), so this saves memory and disk, not compute.

Recipe

LFM2 is a hybrid stack: 30 blocks, 22 short-convolution and 8 full-attention, each with a SwiGLU feed-forward — 166 quantized nn.Linear layers. The FFN is 81 % of the quantizable weight, so it sets the file size; the 8 attention blocks are 3.4 % of it and the only operators that mix across the whole sequence, so protecting them is nearly free.

Effective 5.31 bits/weight over the 2 434.8 M quantized parameters. The tied embedding stays BF16: it is an nn.Embedding shared with lm_head, and 524 MB of this repo is that one tensor.

Measurements

Paired against the BF16 baseline on identical token windows — 48 × 1024 English (wikitext-2 test) and 25 × 1024 Japanese. KL is mean KL(baseline ‖ variant) over the baseline's top-64 tokens; top-1 is argmax agreement with the baseline.

Downstream, 1000 questions each, single-token letter scoring:

Binomial standard error is ~1.5 pp at n = 1000, so differences under ~4 pp are not resolvable; every quantized variant tested sits within 3.5 pp of BF16.

Scoring note: LFM2.5's chat template ends its generation prompt with a literal <think>. A bare "Answer:" completion therefore never reaches a letter and scores at chance (16.7 % MMLU for the BF16 model). The numbers above close the block with </think> before the answer cue.

Calibration

70 396 tokens, 37.2 % Japanese — wikitext-2 validation plus a Japanese sentence corpus, disjoint from both evaluation sets. The language mix mattered more than the token count: a 12.6 %-Japanese set three times larger produced worse Japanese fidelity.

What did not work

Three standard methods were implemented and measured on this checkpoint, and none is in this recipe:

• ScatterQuant / ISQ (soliton–radiation low-rank split): the Marchenko-Pastur noise floor selects ranks of 1–26 capturing 0.5–12 % of the spectral energy, so the BF16 low-rank branch buys under 1 dB for 0.1–0.25 bits/weight. It degenerates to plain RTN.
• AWQ activation-aware scaling, folded exactly into the preceding RMSNorm (verified at 7e-7 relative difference): +4 % KL, i.e. slightly worse. GPTQ already exploits the same statistics through a second-order objective.
• Hadamard rotation (QuaRot-style): −8 % KL, real but not enough, and it costs a runtime rotation (~20 % of the FFN matmul) because LFM2's tied lm_head blocks the fused residual-stream form.

Finer groups bought 19 %, rotation 8 %, per-channel scaling nothing — the residual error on LFM2.5 is broadly distributed quantization noise, not activation outliers.

Changes from the base model

Per LFM Open License §4(b): all nn.Linear weights outside the embedding were replaced with packed integer codes and per-group scales/zero-points as tabulated above; model.safetensors is regenerated and quant_config.json is added. config.json, generation_config.json, tokenizer.json, tokenizer_config.json and chat_template.jinja are unmodified copies from the base model. No weights were fine-tuned; no training data was used beyond the calibration text described above.

License

LFM Open License v1.0, inherited from the base model — including its Commercial Use limitation (§5). Redistribution of this derivative carries the same terms.