LFM2.5-2.6B-GGUF

LFM2.5‑2.6B GGUF Quantizations

Overview

This repository contains the GGUF‑formatted quantized versions of the LiquidAI LFM2.5‑2.6B model. Each variant is named according to the LFM2.5-2.6B-<quant>.gguf scheme and is optimized for different trade‑offs between size and inference speed.

IMPORTANT FIX: It was not possible to disable thinking in LiquidAI/LFM2.5-2.6B-GGUF, these quants don't have that issue.

All quantizations were generated using the imatrix quantization pipeline. Note: the token_embd.weight layer was not quantized in any of these files.

imatrix calibration file is from bartowski

Written by LFM2.5-2.6B-Q8_0: This model card is written by the Q8_0 quant.

Quantization Table

Notes

• Imatrix Pipeline: All quantizations were created with the imatrix quantization tool. This method provides high‑quality compression while preserving model accuracy.
• Token Embeddings: The token_embd.weight layer remains in full FP16 (or the original precision) and was not quantized. This ensures that embedding look‑ups remain accurate during inference.
• KL Divergence Test: A KL‑divergence evaluation will be added to this repository shortly. Stay tuned for the updated benchmark results.

Usage

To load any of the quantized models, simply use the standard GGUF loader (e.g., llama.cpp or auto-gptq):

# Example with llama.cpp
./llama-server -m "./LFM2.5-2.6B-Q4_K_M.gguf" -c 128000

Replace the model file with any of the entries from the table above.

License

All models and quantizations are released under the LiquidAI LFM2.5‑2.6B license (see the model card for details).

---

For further questions or feedback, feel free to open an issue.