lightonai-lateon-Q4_0

LateOn Q4_0 for litembeddings

A compact Q4_0 GGUF conversion of lightonai/LateOn, a 149M-parameter ModernBERT ColBERT retrieval model. This speed/size variant is built for litembeddings and includes the model's complete three-stage PyLate projection folded into one 768-to-128 matrix.

Q4_0 is intended for high-throughput packed indexing and retrieval. It is approximately 40% smaller than the Q8_0 conversion and reached approximately 29.9k tokens/s at packed saturation on an Apple M4 Max, versus approximately 20.9k tokens/s for Q8_0 under the same benchmark. Very short single requests are dominated by dispatch overhead and are not consistently faster; use Q8_0 when maximum numerical fidelity matters more than size or packed throughput.

Files

Both files are required. The GGUF encoder alone produces 768-dimensional token vectors; litembeddings applies and normalizes the supplied projection to produce the model's intended 128-dimensional token vectors.

litembeddings usage

.load ./litembeddings

SELECT lembed_model(
  '/path/to/LateOn-Q4_0.gguf',
  json_object(
    'colbert_projection', '/path/to/LateOn-Q4_0.projection',
    'ctx_size', 300,
    'batch_size', 300
  )
);

-- Full-precision token vectors.
SELECT lembed_tokens('What is SQLite WAL mode?');

-- Compact int8 token vectors for storage and MaxSim search.
SELECT lembed_tokens_quantized('What is SQLite WAL mode?');

SELECT lembed_maxsim(
  lembed_tokens('What is SQLite WAL mode?'),
  lembed_tokens('WAL mode lets SQLite readers coexist with one writer.')
);

LateOn was trained for queries up to 32 tokens and documents up to 300 tokens. Use the same preprocessing and retrieval conventions as the source model when comparing quality results.

Validation

An end-to-end parity check against the source FP32 Transformers/PyLate pipeline over 331 output tokens produced per-token cosine similarity min=0.994162, mean=0.998347.

A 216-document FastAPI retrieval evaluation with 50 queries retained the same measured ranking metrics as exact search with this artifact: MRR 0.8284, Hits@1/5/10 74% / 94% / 96%, and weighted score 92/100. This is a small validation corpus, not a replacement for task-specific evaluation; Q4 quantization can affect other corpora or near-tied rankings.

Conversion provenance

• Source: lightonai/LateOn
• Source revision: c01907b70557ee5c7753680d4819a5cce1674b83
• Converted: 2026-07-14
• llama.cpp revision: 6eddde06a4f25d55d538b5d15628dcc2b6882147
• Quantization: Q4_0 from the F16 GGUF conversion
• Projection: W3 @ (W2 + R2) @ (W1 + R1), preserving both learned dimension-changing PyLate residuals

License and attribution

The source model is released under Apache 2.0. See the LateOn model card for training details, evaluation results, intended use, limitations, and citation information. This repository is an independent conversion and is not affiliated with LightOn.