LFM2.5-8B-A1B-NVFP4

LFM2.5-8B-A1B-NVFP4

NVFP4 (W4A4) quantisation of LiquidAI/LFM2.5-8B-A1B, built and verified on NVIDIA GB10 Blackwell (sm_121a).

✅ The first NVFP4 build of any LFM2.5 checkpoint. Checked 2026-08-22: the only other FP4-family LFM2.5 build is mlx-community/LFM2.5-2.6B-mxfp4, which is MXFP4 for Apple MLX — a different format and a different runtime. Repository-content comparison only.

This is real W4A4, not weight-only

A weight-only NVFP4 export will not serve: SGLang's _is_fp4a4_nvfp4 returns False when input_quant is None, and there is no weight-only float4 scheme in the stack. Real W4A4 requires a calibrated input_global_scale per quantised Linear. This export has them:

format: nvfp4-pack-quantized      quant_method: compressed-tensors
weights:            4-bit float, tensor_group, group_size 16, static
input_activations:  4-bit float, tensor_group, group_size 16, dynamic="local"

The file

Ignore list — quantised: expert Linears, attention, conv projections and dense FFN. Left at bf16: lm_head (tied), embeddings, all 22 routers, all expert_bias buffers, and every norm. Calibrated with moe_calibrate_all_experts=True so every expert sees every calibration token — with top-4-of-32 routing a naive pass starves most experts. All 2,112 expert Linears received scales, with zero "calibrated without statistics" warnings.

Note: llmcompressor 0.13.0 has no lfm2_moe entry in ARCH_TO_2D_MAPPINGS, so it writes experts as gate_proj/up_proj/down_proj while the vLLM/SGLang lfm2_moe loader expects w1/w2/w3. The mapping was derived from Lfm2MoeExperts.forward and applied (8,448 tensors renamed, 0 leftovers).

Measured throughput

Sample output

Chat completion with the model's default sampling. 225 completion tokens, TTFT 0.054 s.

The sky appears blue because molecules in the atmosphere scatter short-wavelength (blue) light far more efficiently than longer wavelengths, so the scattered light dominates the colour we see. This preferential scattering of blue light makes the overall hue of the sky blue.

Requirements

Needs a runtime with compressed-tensors NVFP4 W4A4 support on Blackwell — SGLang v0.5.18-cu130 was used here; vLLM nightly also registers Lfm2MoeForCausalLM.

python3 -m sglang.launch_server --model-path <this-repo> \
  --host 0.0.0.0 --port 30000 --tp-size 1 --mem-fraction-static 0.45

Provenance

llmcompressor oneshot with QuantizationModifier(targets="Linear", scheme="NVFP4") — not model_free_ptq, which cannot calibrate activations and can only emit the unservable weight-only form. Calibration: HuggingFaceH4/ultrachat_200k (train_sft) via the model's own chat template. Source revision b9aebfcbe28b6cb374042f495d733037550ab146. Licence inherited from the base model.