LFM2.5-2.6B-GPTQ-hybrid-b-g32x

LFM2.5-2.6B — GPTQ mixed-precision, Japanese-first (hybrid-b-g32x)

Weight-only quantization of LiquidAI/LFM2.5-2.6B: 5394 MB → 2158 MB (2.50×).

This is the Japanese-leaning variant — the best Japanese fidelity of the whole sweep. For a balanced EN/JA build 16 MB smaller, see aoiandroid/LFM2.5-2.6B-GPTQ-hybrid-b-g32; the two differ only in group size on the conv and down projections.

⚠️ This is not a stock GPTQ checkpoint

AutoModelForCausalLM.from_pretrained will not load this repo. Weights use the forge-weightonly-v1 format — integer codes packed into a uint8 stream at exactly bits bits per weight, with per-layer widths in quant_config.json. Use the loader shipped in this repo:

from quant_loader import load_quantized          # needs qlinear.py, same repo

model, tok = load_quantized("aoiandroid/LFM2.5-2.6B-GPTQ-hybrid-b-g32x")

msgs = [{"role": "user", "content": "量子化とは何ですか。"}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=256)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Requires transformers >= 5.2 (for Lfm2ForCausalLM), torch, safetensors. Weights stay packed in VRAM and are dequantized to BF16 inside forward(), so this saves memory and disk, not compute.

Recipe

LFM2 is a hybrid stack: 30 blocks, 22 short-convolution and 8 full-attention, each with a SwiGLU feed-forward — 166 quantized nn.Linear layers.

Effective 5.37 bits/weight over the 2 434.8 M quantized parameters. The tied embedding stays BF16: it is an nn.Embedding shared with lm_head, and 524 MB of this repo is that one tensor.

Measurements

Paired against the BF16 baseline on identical token windows — 48 × 1024 English (wikitext-2 test) and 25 × 1024 Japanese. KL is mean KL(baseline ‖ variant) over the baseline's top-64 tokens; top-1 is argmax agreement with the baseline.

The extra 16 MB buys −2 % Japanese KL and +0.5 pp Japanese top-1 over the balanced build, with English essentially tied.

Calibration

70 396 tokens, 37.2 % Japanese — wikitext-2 validation plus a Japanese sentence corpus, disjoint from both evaluation sets. The language mix mattered more than the token count: a 12.6 %-Japanese calibration set three times larger produced worse Japanese fidelity. If your deployment is Japanese-first, calibrate on Japanese.

What did not work

Three standard methods were implemented and measured on this checkpoint, and none is in this recipe:

• ScatterQuant / ISQ (soliton–radiation low-rank split): the Marchenko-Pastur noise floor selects ranks of 1–26 capturing 0.5–12 % of the spectral energy, so the BF16 low-rank branch buys under 1 dB for 0.1–0.25 bits/weight. It degenerates to plain RTN.
• AWQ activation-aware scaling, folded exactly into the preceding RMSNorm (verified at 7e-7 relative difference): +4 % KL, i.e. slightly worse. GPTQ already exploits the same statistics through a second-order objective.
• Hadamard rotation (QuaRot-style): −8 % KL, real but not enough, and it costs a runtime rotation (~20 % of the FFN matmul) because LFM2's tied lm_head blocks the fused residual-stream form.

Finer groups bought 19 %, rotation 8 %, per-channel scaling nothing — the residual error on LFM2.5 is broadly distributed quantization noise, not activation outliers.

Evaluation note

LFM2.5's chat template ends its generation prompt with a literal <think>. In multiple-choice scoring, a bare "Answer:" completion therefore never reaches a letter and scores at chance (16.7 % MMLU for the BF16 model). Close the block with </think> before the answer cue.

Changes from the base model

Per LFM Open License §4(b): all nn.Linear weights outside the embedding were replaced with packed integer codes and per-group scales/zero-points as tabulated above; model.safetensors is regenerated and quant_config.json is added. config.json, generation_config.json, tokenizer.json, tokenizer_config.json and chat_template.jinja are unmodified copies from the base model. No weights were fine-tuned; no training data was used beyond the calibration text described above.

License

LFM Open License v1.0, inherited from the base model — including its Commercial Use limitation (§5). Redistribution of this derivative carries the same terms.