LFM2.5-8B-A1B-AWQ-INT4-agentic

LFM2.5-8B-A1B AWQ INT4 (agentic/chat calibration)

AWQ INT4 checkpoint of LiquidAI/LFM2.5-8B-A1B, calibrated on agentic/chat-style prompts for vLLM serving. Uses compressed-tensors format — auto-detected by vLLM 0.24+.

Model Details

Quantization

Conv layers, MoE gates, dense FFN in layers 0–1, and lm_head kept in higher precision. Post-quant export renames linearized MoE expert keys for vLLM compatibility.

Benchmarks

Hardware: NVIDIA GeForce RTX 5060 Ti (16 GB), vLLM 0.24, CUDA 13.3, flash-attn

Speed (single request, 256 output tokens)

Concurrency Scaling (128 output tokens)

Peak aggregate throughput: 1,742 tok/s at concurrency 20, zero errors across all levels.

Context Length Scaling (single request, 128 output tokens)

Decode speed stays flat across context lengths — TTFT scales linearly with prompt size as expected.

Quality

LFM2.5-8B-A1B is a reasoning-only model (it always emits a <think> chain of thought — there is no non-thinking mode), so it was not run through this project's local chat_core lighteval suite: those numbers are collected in native mode across a mix of thinking and non-thinking models, and a reasoning-only model isn't apples-to-apples there, nor tractable at full sample count on a single 16 GB GPU.

For quality, refer to LiquidAI's published benchmarks for the base model (LiquidAI/LFM2.5-8B-A1B): IFEval 91.84, IFBench 56.47, Multi-IF 79.93, MATH500 88.76, AIME25 42.53, BFCLv3 64.79. This INT4 (W4A16) checkpoint targets vLLM serving throughput; expect small quantization deltas from those BF16 figures. Speed/concurrency for this checkpoint is benchmarked below.

vLLM Usage

vllm serve LostGentoo/LFM2.5-8B-A1B-AWQ-INT4-agentic \
  --trust-remote-code \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.75

No --quantization flag needed — vLLM auto-detects compressed-tensors format.

OpenAI-compatible test

curl http://127.0.0.1:8012/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"LFM2.5-8B-A1B-AWQ-INT4-agentic","messages":[{"role":"user","content":"ping"}],"max_tokens":32}'

Troubleshooting

• CUDA/nvcc errors: Set CUDA_HOME=/opt/cuda (flashinfer requires CUDA toolkit)
• ninja not found: Install ninja-build (flashinfer JIT dependency)
• flashinfer JIT failures: Set VLLM_ATTENTION_BACKEND=FLASH_ATTN to bypass
• Model config mismatch: This model uses compressed-tensors quant format — do NOT pass --quantization awq

License

Same as base model: LiquidAI/LFM2.5-8B-A1B License