jina-colbert-v2-int8-onnx

jina-colbert-v2-int8-onnx

Dynamic INT8 quantization of jinaai/jina-colbert-v2, produced for fast/enterprise-friendly deployment of KOMPAS-3D_GUARD's optional ColBERT retrieval backend. This is a derivative artifact, not an official Jina AI release.

Why this exists

The upstream onnx/model.onnx + onnx/model.onnx_data is FP32, ~2.24 GB combined, and requires an external-data file. That is a slow, unreliable download for CI/enterprise networks and inflates install size. This repo hosts a single self-contained INT8 ONNX file quantized from that same upstream checkpoint, so it can be fetched, verified, and cached deterministically.

Provenance / reproducibility

• Base checkpoint: jinaai/jina-colbert-v2, revision 4552c4dc1ffd7d7a635b6a41a1077fe9c9cdd974 (onnx/model.onnx + onnx/model.onnx_data, verified against upstream SHA256 before quantizing).
• Pipeline: onnxruntime.quantization.shape_inference.quant_pre_process (shape inference, skip_symbolic_shape=True, external data preserved) then onnxruntime.quantization.quantize_dynamic(weight_type=QuantType.QInt8).
• Tool versions: onnxruntime==1.27.0, onnx (current at build time), Python 3.14.
• No calibration data / static quantization was used (dynamic weight-only INT8); activations remain computed in floating point at inference time.

Files and SHA256

config.json/tokenizer*.json/special_tokens_map.json are copied verbatim from the upstream revision above (unchanged, same hashes as upstream).

Measured quality delta (quick sanity check, not a full eval)

Per-token cosine similarity between FP32 and this INT8 model's late-interaction embeddings on a short KOMPAS API sentence: mean 0.975, min 0.963 (single sample, CPU InferenceSession). This is a smoke check, not a retrieval-quality (hit@k) evaluation; validate on your own eval set before relying on it for ranking decisions.

License

The base model jinaai/jina-colbert-v2 is licensed CC-BY-NC-4.0 (non-commercial). This quantized derivative inherits the same license: non-commercial use only, with attribution to Jina AI's original model. If you need to use this in a commercial/enterprise product, you must obtain a commercial license from Jina AI for the base model — quantization does not change the licensing terms.

Usage

from fastembed import LateInteractionTextEmbedding

LateInteractionTextEmbedding.add_custom_model(
    model="dwnmf/jina-colbert-v2-int8-onnx",
    sources={"hf": "dwnmf/jina-colbert-v2-int8-onnx"},
    model_file="model_int8.onnx",
    dim=128,
    description="INT8 dynamic-quantized jina-colbert-v2 ONNX",
    license="cc-by-nc-4.0",
    size_in_GB=0.56,
)
model = LateInteractionTextEmbedding("dwnmf/jina-colbert-v2-int8-onnx")