LFM2.5-350M-heretic-GGUF

🤖 LFM2.5-350M-heretic — GGUF

This repository hosts GGUF weights for LFM2.5-350M-heretic, quantized from the source floating-point tensors provided by coder3101/LFM2.5-350M-heretic.

🔄 Sister Repository: Check out the Imatrix Sister Repository for enhanced precision at lower bit fractions.

[!NOTE] If you plan on using 4-bit or 5-bit variants, consider the imatrix sister repository instead — importance matrix calibration improves logic retention at those bit depths. This repository is best suited if you want the near-lossless Q8_0 build.

ℹ️ Model Profile & Core Features

LFM2.5-350M is the smallest text-only model in Liquid AI's Liquid Foundation Model 2.5 series, built for extreme on-device and edge deployment. It shares the family's hybrid architecture of double-gated LIV (Liquid, Input-adaptive, Value-selective) convolution layers interleaved with GQA (Grouped Query Attention) layers, pre-trained on 28 trillion tokens with large-scale reinforcement learning post-training. Despite its size, it is tuned for instruction following, lightweight tool calling, and structured data extraction, with day-one support across llama.cpp, MLX, vLLM, SGLang, ONNX, and OpenVINO.

The heretic suffix denotes post-processing via the Heretic v1.3.0 method performed by coder3101, which removes refusal conditioning while preserving the model's lightweight instruction-following behavior.

📋 Technical Specifications

🛠️ Heretic Overrides (ARA)

📊 Refusal Bypass Metrics

[!NOTE] The metrics below are self-reported by the original model author (coder3101) and have not been independently reproduced.

🧮 Numerical & Tensor Formats

📦 Available Model Files

Main model weights

🎛️ Component Pairing Guide

Download exactly one main weights file:

• Q4_K_M: Balanced 4-bit format suitable for most everyday use.
• Q5_K_M: Higher-fidelity mid-range format recommended as a general default.
• Q8_0: Near-lossless 8-bit format for when memory is not a constraint.

⚡ Deployment & Execution Commands

[!NOTE] Liquid AI recommends the following generation parameters for best results: temperature: 0.1, top_k: 50, repetition_penalty: 1.05.

[!TIP] Swap the -m filename below for either quantized file depending on your size/quality trade-off preference.

llama.cpp CLI

./llama-cli \
  -m LFM2.5-350M-heretic-Q4_K_M.gguf \
  -c 8192 \
  -ngl 99 \
  --temp 0.3 \
  --top-k 40 \
  --repeat-penalty 1.05 \
  -p "<|im_start|>system\nYou are a concise, helpful assistant.<|im_end|>\n<|im_start|>user\nState the capital of Italy and one interesting fact about it.<|im_end|>\n<|im_start|>assistant\n"

OpenAI-Compatible API Server

./llama-server \
  --host 0.0.0.0 \
  --port 8080 \
  -m LFM2.5-350M-heretic-Q4_K_M.gguf \
  -c 16384 \
  -ngl 99 \
  --flash-attn

💬 Chat Templates & Prompt Design (ChatML)

<|im_start|>system
You are a capable assistant. Follow instructions precisely.<|im_end|>
<|im_start|>user
Your task or query here.<|im_end|>
<|im_start|>assistant

⚠️ Safety & Operational Notes

• This model is abliterated and will generate content that standard aligned models refuse. Use responsibly and in compliance with applicable laws.
• This is a text-only model — it has no vision encoder and cannot process images.
• Despite its small footprint, IFBench and structured-extraction benchmarks show substantial generational gains over LFM2 predecessors.
• Best suited for constrained hardware: CPUs, NPUs, and edge devices rather than complex reasoning workloads.
• For better output quality at this quantization level, consider the imatrix variant in the companion repository.