lightonai-lateon-Q8_0

LateOn Q8_0 for litembeddings

A Q8_0 GGUF conversion of lightonai/LateOn, a 149M-parameter ModernBERT ColBERT retrieval model. This package is built for litembeddings and includes the model's complete three-stage PyLate projection folded into one 768-to-128 matrix.

Files

Both files are required. The GGUF encoder alone produces 768-dimensional token vectors; litembeddings applies and normalizes the supplied projection to produce the model's intended 128-dimensional token vectors.

litembeddings usage

.load ./litembeddings

SELECT lembed_model(
  '/path/to/LateOn-Q8_0.gguf',
  json_object(
    'colbert_projection', '/path/to/LateOn-Q8_0.projection',
    'ctx_size', 300,
    'batch_size', 300
  )
);

-- Full-precision token vectors.
SELECT lembed_tokens('What is SQLite WAL mode?');

-- Compact int8 token vectors for storage and MaxSim search.
SELECT lembed_tokens_quantized('What is SQLite WAL mode?');

SELECT lembed_maxsim(
  lembed_tokens('What is SQLite WAL mode?'),
  lembed_tokens('WAL mode lets SQLite readers coexist with one writer.')
);

LateOn was trained for queries up to 32 tokens and documents up to 300 tokens. Use the same preprocessing and retrieval conventions as the source model when comparing quality results.

Conversion provenance

• Source: lightonai/LateOn
• Source revision: c01907b70557ee5c7753680d4819a5cce1674b83
• Converted: 2026-07-14
• llama.cpp revision: 6eddde06a4f25d55d538b5d15628dcc2b6882147
• Quantization: Q8_0
• Projection: W3 @ (W2 + R2) @ (W1 + R1), preserving both learned dimension-changing PyLate residuals

An end-to-end parity check against the source FP32 Transformers/PyLate pipeline produced per-token cosine similarity min=0.999883, mean=0.999951 for the validation text. Small differences are expected from Q8_0 quantization.

License and attribution

The source model is released under Apache 2.0. See the LateOn model card for training details, evaluation results, intended use, limitations, and citation information. This repository is an independent conversion and is not affiliated with LightOn.