LFM2.5-1.2B-Thinking-ToMoE-INT8

LFM2.5-1.2B-Thinking-ToMoE-INT8

Weight-only INT8 quantization of the real ToMoE MoE — Nichonauta/LFM2.5-1.2B-Thinking-ToMoE (derived from LiquidAI/LFM2.5-1.2B-Thinking).

Unlike the GGUF companions (which are dense-equivalents because llama.cpp cannot represent the channel-MoE), this repository stores the actual pruned MoE weights quantized in safetensors format:

• LinearINT8 replaces the MLP/attention/conv Linear layers (per-channel symmetric for INT8; per-group of 64 with zero-point for INT4), packed 2-nibbles-per-byte for INT4.
• The routers, embeddings and norms stay fp32 (tiny and sensitive); weights dequantize lazily in the forward pass.
• Same custom model definition (modeling_lfm2_moe_final.py, trust_remote_code), plus the quantization flag in config.json.

Metrics vs the fp32 MoE

The 1.2B MoE is robust to weight-only quantization: INT8 and INT4 are within measurement noise of the fp32 MoE (raw PPL identical, chat within ~1.5%).

Usage (transformers, trust_remote_code)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Nichonauta/LFM2.5-1.2B-Thinking-ToMoE-INT8",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",
    attn_implementation="eager",
)
tokenizer = AutoTokenizer.from_pretrained("Nichonauta/LFM2.5-1.2B-Thinking-ToMoE-INT8")

The quantization field in config.json makes the custom modeling file build the quantized Linear layers automatically.

Files

• model.safetensors — INT8 weights + fp32 scales/zero-points + fp32 routers/embeddings/buffers
• modeling_lfm2_moe_final.py — custom model definition with LinearINT8 support (trust_remote_code)
• config.json — model configuration (quantization: "int8", auto_map)
• tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json, LICENSE

License

Derivative of LiquidAI/LFM2.5-1.2B-Thinking — released under the LFM Open License v1.0 (see LICENSE).