LateOn-regularized-onnx

LateOn-regularized-onnx

ONNX export of lightonai/LateOn-regularized (Apache-2.0) for fastlate and any ONNX Runtime client. The weights are unchanged; this repo adds the graph, an INT8 dynamic quantization of it, and the ColBERT settings the model was trained with.

Files

• model.onnx: transformer, the PyLate Dense projection layer(s), and per-token L2 normalization, in one graph. Inputs input_ids, attention_mask (int64, dynamic batch and length); output embeddings of shape (batch, seq, 128). Opset 17.
• model_int8.onnx: onnxruntime.quantization.quantize_dynamic of the above, QInt8 weights.
• onnx_config.json: prefix ids, lengths, skiplist, expansion and padding settings copied from the PyLate model, plus fde_center.
• tokenizer.json: the source tokenizer, with [Q] /[D] as added tokens.

Settings

Query prefix [Q] (id 50368), document prefix [D] (id 50369), inserted right after the sequence-start token. Query length 32, document length 300. do_query_expansion=False: queries are not padded. Punctuation tokens (skiplist_words) are dropped from document embeddings after encoding, as in PyLate.

fde_center=True: whether to subtract the corpus mean token vector before building MUVERA fixed-dimensional encodings. Measured on a 3,000-chunk code/notebook corpus with FDEs of 4,096 dims: this model needs centring (recall of the exact top-20 within 1,000 candidates went from 0.61 to 0.94), matching the evaluation notes on the source model card.

Validation

fp32 graph vs PyTorch forward: max abs diff 4.8e-07; int8 graph vs PyTorch: 0.017; int8 graph vs PyLate encode on a sample text: documents 0.016, queries 0.022.

Exported with PyLate 1.6.0, transformers 5.3.0, torch 2.14.0, onnxruntime 1.2x, on 2026-09-18.