Octen-Embedding-0.6B-ONNX-INT4-FULL

Octen-Embedding-0.6B — INT4-Full ONNX (INT4 MatMul + INT8 Gather)

Smallest variant of cstr/Octen-Embedding-0.6B-ONNX at ~434 MB — both transformer weights (MatMul) and the embedding table (Gather) are quantized.

Quantization details

Quantizing the Gather reduces the ~621 MB FP32 embedding table to ~155 MB (INT8). Quantizing MatMul to INT4 (block_size=32) gives ~275 MB. Combined total: ~434 MB vs 896 MB for INT4-MatMul-only.

Model details

Inference

import onnxruntime as ort
import numpy as np
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
tokenizer.enable_padding(pad_id=0, direction="right")
tokenizer.enable_truncation(max_length=512)

session = ort.InferenceSession("model.int4_full.onnx", providers=["CPUExecutionProvider"])

texts = ["semantic search example", "another sentence"]
enc  = tokenizer.encode_batch(texts)
ids  = np.array([e.ids            for e in enc], dtype=np.int64)
mask = np.array([e.attention_mask for e in enc], dtype=np.int64)

lhs        = session.run(None, {"input_ids": ids, "attention_mask": mask})[0]
seq_lens   = mask.sum(axis=1) - 1
embeddings = lhs[np.arange(len(texts)), seq_lens]
norms      = np.linalg.norm(embeddings, axis=1, keepdims=True)
embeddings = embeddings / np.maximum(norms, 1e-8)
print(embeddings.shape)  # (2, 1024)

Variants

License

Apache 2.0 — same as the base Octen model.

Provenance and EU AI Act Art. 53 note

• Upstream model: Octen/Octen-Embedding-0.6B — published by Octen.
• Upstream licence: apache-2.0. This repository redistributes under the same terms; it grants no rights the upstream licence does not.
• What was done here: format conversion and/or quantisation only (ONNX, INT4 precision). No training, no fine-tuning, no merging, no distillation, no change to architecture, vocabulary or capability. Only the numeric representation of the upstream weights differs.
• Training data: documented — where it is documented at all — by the upstream provider; see the upstream model card. No training data was used, added or selected by this repository. No training-content summary was found on the upstream model card at the time of writing; that documentation gap is upstream's and is not filled here.
• Provider status: under Regulation (EU) 2024/1689 the upstream authors remain the provider of this model. Converting the serialisation format does not make this repository the provider of a new general-purpose AI model, and no such claim is made. Questions about training content, copyright policy or model capability belong upstream.