lfm2-colbert-350m-onnx-fp16

LFM2-ColBERT-350M (ONNX float16)

ONNX float16 export of LiquidAI/LFM2-ColBERT-350M, a multilingual late-interaction (ColBERT) retriever, packaged for ONNX Runtime (CPU / edge) with weights in float16: 708 MB instead of 1.41 GB for the float32 export (half the size), near-lossless for retrieval.

Packaged by Webagil. Base model, architecture and weights are © Liquid AI, Inc.

⚠️ Modifications vs the base model

Per Section 4 of the LFM Open License v1.0 (marking of modifications):

• Exported to ONNX (opset 18, dynamic input shapes): LFM2 backbone + 1_Dense projection (1024→128) + per-token L2 normalization fused into the graph. Output output = one L2-normalized vector per token, shape [batch, seq, 128].
• Weights quantized to float16 (inputs/outputs kept float32, keep_io_types). Numerically near-lossless: per-token cosine ≥ 0.9999 vs float32, end-to-end nDCG@10 within ~0.4 % (see Benchmarks).
• Added onnx_config.json specifying the query/document prefixes and length bounds, so integrators do not have to guess them.

No change to the model's behavior, training or architecture. The base model is the work of Liquid AI, Inc.

Model description

Files

• model.onnx: float16 weights, float32 I/O (single file, ~708 MB).
• tokenizer.json, onnx_config.json, config_sentence_transformers.json.
• LICENSE (LFM Open License v1.0), NOTICE (attribution + modifications).

Usage (ONNX Runtime)

Inputs: input_ids, attention_mask (int64, [batch, seq]). Output: output (float32, [batch, seq, 128], already L2-normalized per token). Prepend the [Q] / [D] prefixes: each one (whitespace included) is a single registered special token ([Q] = id 64400, [D] = id 64401), and the model was trained with them, so omitting them degrades retrieval. Score with MaxSim.

import numpy as np, onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL  # see note below
sess = ort.InferenceSession("model.onnx", so, providers=["CPUExecutionProvider"])

def encode(text, is_query):
    prefix, max_len = ("[Q] ", 32) if is_query else ("[D] ", 512)  # base model's query/doc lengths
    ids = tok.encode(prefix + text).ids[:max_len]
    a = np.array([ids], dtype=np.int64)
    out = sess.run(["output"], {"input_ids": a, "attention_mask": np.ones_like(a)})[0][0]
    return out  # (n_tokens, 128), L2-normalized; single items need no padding

def maxsim(q, d):  # late interaction
    return float((q @ d.T).max(axis=1).sum())

# French example; usage is identical in all 8 supported languages
q = encode("Où se repose le chat ?", is_query=True)
d = encode("Le chat dort sur le canapé du salon.", is_query=False)
print(maxsim(q, d))

Performance note (important): use the maximum ONNX Runtime graph optimization level: ORT_ENABLE_ALL in Python. In the Rust ort crate (rc.12), use GraphOptimizationLevel::All, not Level3: in that crate Level3 maps to ORT_ENABLE_LAYOUT, which leaves the MatMul/Cast fusions un-applied and makes the float16 model ~3–4× slower on CPU.

Benchmarks

NanoBEIR-en (13 datasets, mean nDCG@10): this float16 ONNX artifact, evaluated full-corpus with plain MaxSim (public harness: onnxruntime + tokenizers + mteb):

The small gap is mostly harness-related (PyLate evaluation pipeline vs raw MaxSim over the full corpus): in controlled same-harness comparisons, float16 costs ≤ 0.003 nDCG@10 vs float32 (see below).

Quantization quality (MTEB-fr, nDCG@10): the controlled float16 vs float32 comparison. Both columns are measured with the same public harness (onnxruntime with ORT_ENABLE_ALL, MaxSim late interaction, nDCG@10), so float16 vs float32 is apples-to-apples and reproducible by anyone:

Absolute scores come from this minimal brute-force eval (one vector per token; documents truncated at the model's 512-token limit, no long-document chunking), not tuned leaderboard runs. What matters for a quantization card is the float16 ↔ float32 gap: ≤ 0.4 % relative on both datasets (0.0029/0.8172 ≈ 0.35 %; 0.0003/0.4097 ≈ 0.07 %).

float16 vs float32 parity: per-token embeddings stay cosine ≥ 0.9999 (worst 0.99993) vs float32 over lengths 7–511. End-to-end, this is near-lossless: nDCG@10 is within ~0.4 % of float32; the sub-1e-3 per-token differences only reorder documents that are already near-tied in MaxSim score.

Latency: single-item encode, ORT_ENABLE_ALL, median of 11, measured on a 2019 MacBook Pro (CPU Intel i9-9880H, GPU AMD Radeon Pro 5500M for the WebGPU column). CPU latency scales with sequence length; on GPU it grows much more slowly, so the speed-up widens with longer inputs:

Throughput (same 256-document FiQA protocol, mixed lengths). CPU batch=1 throughput is similar across machines (~4 docs/s: an Apple M2 Pro measured 3.8 docs/s, the 2019 Intel 4.4); the real speed-up comes from the WebGPU EP, not from a faster CPU. WebGPU rows use length-sorted batching (see Batching guidance below):

MaxSim scoring itself (the late-interaction Q · Dᵀ then max-reduce) is a plain NumPy op, orders of magnitude cheaper than encoding. This repo ships only the encoder (see Files above): build whatever index suits you (brute-force for small corpora, or a PLAID/PQ index for large ones) on top of the per-token vectors.

Size: 708 MB (float16) vs 1.41 GB (float32): ×2 smaller. On CPU, float16 runs at parity with float32 speed (the gain is size, not compute; weights are up-cast to float32 internally).

GPU acceleration (WebGPU EP)

The model runs unmodified on ONNX Runtime's WebGPU Execution Provider (Dawn); see the latency and throughput tables above (×2 to ×6 vs CPU). Fidelity is preserved: per-token cosine GPU↔CPU ≥ 0.99997, zero NaN, including on this float16 graph, which the CoreML EP rejects (dynamic input shapes). PyPI ships onnxruntime-webgpu wheels for macOS arm64, Windows x86-64 and Linux x86-64; macOS x86-64 and Windows arm64 require a source build with --use_webgpu.

Batching guidance (important)

• Padding is safe. LFM2 is a conv+attention hybrid, but padded batching does not contaminate the convolutions when the attention mask is set: per-token cosine batch-32 vs batch-1 ≥ 0.99999 (measured). Pad with pad_token_id = 7 and mask the padding.
• Sort by length before batching (bucketing). Naive batching wastes compute on padding and can be slower than batch=1 on GPU; length-sorted buckets of 4–8 (GPU-dependent) give the throughput above.
• On CPU, batch=1 is as fast as batching: keep it simple.

Storage tips for the per-token vectors

• Token pooling ×2 (hierarchical/Ward barycenters; Clavié et al. 2024, Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling) halves the stored vectors per document and is quality-neutral to slightly positive (denoising) in our nDCG@10 measurements; ×3 stays neutral, ×4 starts to cost.
• The L2-normalized 128-d token vectors quantize to int8 (per-vector scale) with no measurable retrieval loss, a further ×4 storage reduction on top of pooling.

Limitations

• CPU inference of float16 is not faster than float32 (no native float16 compute on most CPUs); the benefit is size. For speed-ups use the WebGPU EP (measured above) or another accelerator.
• GPU gains shrink with sequence length: a single short query (~20 tokens) only gains ~×2 over CPU (dispatch + copies dominate), vs ×6 for a 512-token document and ×3.8–×4.9 on batched ingestion throughput.
• Indexed/retrieved documents are untrusted input: do not treat their content as instructions (indirect prompt injection).
• Same capabilities/limitations as the base model.

License & attribution

Licensed under the LFM Open License v1.0: see LICENSE and NOTICE. Base model: LiquidAI/LFM2-ColBERT-350M, © Liquid AI, Inc.

Commercial use is permitted while the user's total annual revenue is below USD 10 M; above that, contact Liquid AI, Inc. (license).

Citation

Cite the original model:

@misc{lfm2-colbert-350m,
  title  = {LFM2-ColBERT-350M},
  author = {Liquid AI, Inc.},
  year   = {2025},
  howpublished = {\url{https://huggingface.co/LiquidAI/LFM2-ColBERT-350M}}
}