LFM2.5-Embedding-350M-FP8

LFM2.5-Embedding-350M-FP8

FP8 quantization of LiquidAI/LFM2.5-Embedding-350M using transformers FineGrainedFP8 (e4m3 weights, 128x128 block scales, dynamic activations). The matmul-heavy Linear layers are 8-bit; token embeddings and norms stay bf16. ~556 MB.

Quality

Versus the bf16 model on a multilingual query/document probe: cosine ~= 0.997 (min ~0.995), and the retrieval rank order is preserved (nearest-document matches identical to bf16). Across the full NanoBEIR Multilingual Extended benchmark (13 datasets x 11 languages, 143 tasks), this FP8 checkpoint scores 0.575 NDCG@10 versus 0.577 for the bf16 base reported by LiquidAI: a -0.002 (-0.35% relative) difference, within the <1% quality target.

Per-language average (over the 13 datasets): en 0.642, fr 0.586, pt 0.582, it 0.577, de 0.577, es 0.576, ja 0.571, ko 0.567, sv 0.563, no 0.559, ar 0.525. Measured with transformers AutoModel (CLS pooling + L2 normalization).

Usage (load with AutoModel, NOT SentenceTransformer)

Do not load this with SentenceTransformer(...) - its loader does not apply the FP8 quantizer to this custom architecture and returns incorrect embeddings. Use transformers.AutoModel (below) or vLLM.

import torch, torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

repo = "racineai/LFM2.5-Embedding-350M-FP8"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True, device_map="cuda").eval()

def embed(text):
    ins = tok(text, return_tensors="pt", truncation=True, max_length=512).to("cuda")
    with torch.no_grad():
        h = model(**ins).last_hidden_state
    return F.normalize(h[:, 0], dim=-1)          # CLS pooling (token 0) + L2-normalize

q = embed("query: what is the capital of France?")
d = embed("document: Paris is the capital and most populous city of France.")
print((q @ d.T).item())

Use the asymmetric prompts "query: " and "document: ", CLS pooling (position 0), and L2-normalization, same as the base model.

Maximum-throughput CUDA engine (built by Racine AI)

We built an open-source, standalone C++/CUDA inference engine for this base model: https://github.com/RacineAI-comp/lfm2-embedding-fp8-engine

We fused the entire forward pass into custom kernels: fused RMSNorm / SwiGLU FP8 epilogues, block-scaled MXFP8 GEMMs (per-tensor FP8 + tensor-core attention on H100), a bidirectional GQA attention kernel, varlen packing, and CLS pooling. On a single H100, measured against LiquidAI's own published QPS curve for this model, our engine reaches ~2.5x throughput at concurrency 32 at statistically matched retrieval quality, with single-request latency ~0.97 ms (vs ~1.5 ms for the bf16 reference).

Hardware support. The engine runs on any NVIDIA GPU with FP8 tensor cores:

• Blackwell (sm_100 / sm_120) - B200, B300, GB200 and the RTX 50-series / RTX PRO 6000 Blackwell. Uses the native block-scaled MXFP8 path (CUDA 12.8+).
• Hopper (sm_90) - H100, H200, GH200. Uses the per-tensor FP8 + FAST_ACCUM path (LFM2_PLAIN=1).
• Ada Lovelace (sm_89) - L40S, L40, L4, RTX 6000 Ada, RTX 4090/4080. Same per-tensor FP8 path.

Benchmarked on H100 (sm_90) and RTX 5070 (sm_120). Not supported on Ampere (A100, RTX 30-series) or older - those have no FP8 tensor cores.

This checkpoint vs that engine. They are two different FP8 formats for the same base model and are not interchangeable. This repo is a drop-in FineGrainedFP8 checkpoint you load with AutoModel; our engine uses its own per-tensor-FP8 + MXFP8 weight layout that it regenerates from the bf16 base model (via the repo's export_weights.py), so it does not load this checkpoint directly. Use this repo for plug-and-play transformers use; use our engine for maximum H100 / Blackwell throughput.

License & attribution

Derived from LiquidAI/LFM2.5-Embedding-350M, under the LFM Open License v1.0. All original license terms and attribution to Liquid AI apply to this quantized derivative.