LFM2.5-230M-ToMoE-INT8

LFM2.5-230M-ToMoE-INT8

Weight-only INT8 quantization of the real ToMoE MoE — Nichonauta/LFM2.5-230M-ToMoE (derived from LiquidAI/LFM2.5-230M).

Unlike the GGUF companions (which are dense-equivalents because llama.cpp cannot represent the channel-MoE), this repository stores the actual pruned MoE weights quantized in safetensors format:

• LinearINT8 replaces the MLP/attention/conv Linear layers (per-channel symmetric for INT8; per-group of 64 with zero-point for INT4), packed 2-nibbles-per-byte for INT4.
• The routers, embeddings and norms stay fp32 (tiny and sensitive); weights dequantize lazily in the forward pass.
• Same custom model definition (modeling_lfm2_moe_final.py, trust_remote_code), plus the quantization flag in config.json.

Metrics vs the fp32 MoE

The 230M MoE is heavily degraded in both fp32 and quantized form (raw PPL ~800+), so INT8/INT4 are within noise of each other and of the fp32 MoE.

Usage (transformers, trust_remote_code)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Nichonauta/LFM2.5-230M-ToMoE-INT8",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",
    attn_implementation="eager",
)
tokenizer = AutoTokenizer.from_pretrained("Nichonauta/LFM2.5-230M-ToMoE-INT8")

The quantization field in config.json makes the custom modeling file build the quantized Linear layers automatically.

Files

• model.safetensors — INT8 weights + fp32 scales/zero-points + fp32 routers/embeddings/buffers
• modeling_lfm2_moe_final.py — custom model definition with LinearINT8 support (trust_remote_code)
• config.json — model configuration (quantization: "int8", auto_map)
• tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json, LICENSE

License

Derivative of LiquidAI/LFM2.5-230M — released under the LFM Open License v1.0 (see LICENSE).