all-MiniLM-L6-v2-onnx-quantized

all-MiniLM-L6-v2 — ONNX INT4 & INT8 QDQ Quantized

CPU-generic quantized ONNX exports of sentence-transformers/all-MiniLM-L6-v2 that run everywhere — x86_64, ARM64/NEON, any CPU — without ISA-specific kernels.

Variants

Both variants quantize all ops — MatMul and Gather (word embeddings) — so the full model fits in CPU cache. Unlike the ISA-specific INT8 QOperator builds in the source repo (AVX2, AVX-512, ARM64), these are CPU-generic and work on any architecture with ONNX Runtime ≥ 1.20.

When to use which

• model_qint4.onnx — prefer when throughput matters. Half the FP32 latency, acceptable accuracy loss from block-wise quantization.
• model_qint4_qdq.onnx — prefer when embedding quality matters (e.g. semantic search, clustering). Runs at FP32 speed but preserves higher cosine similarity through per-tensor uint8 quantization of embedding lookups.

Usage

import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

# INT4 (speed-first)
session = ort.InferenceSession("model_qint4.onnx")

# INT8 QDQ (accuracy-first)
# session = ort.InferenceSession("model_qint4_qdq.onnx")

tokens = tokenizer(["hello world"], return_tensors="np", padding=True, truncation=True)
outputs = session.run(None, {
    "input_ids": tokens["input_ids"],
    "attention_mask": tokens["attention_mask"],
    "token_type_ids": tokens["token_type_ids"],
})
embedding = outputs[1]  # pooled output (1 × 384)

Quantization details

Generated with ONNX Runtime's weight-only quantization tools:

• model_qint4.onnx: MatMulNBitsQuantizer with block_size=32, is_symmetric=True, accuracy_level=4, op_types_to_quantize=("MatMul", "Gather"). Produces MatMulNBits and GatherBlockQuantized ops (contrib, opset 21).
• model_qint4_qdq.onnx: quantize_dynamic with weight_type=QuantType.QUInt8, op_types_to_quantize=["MatMul", "Gather"]. Wraps MatMul/Gather in standard DequantizeLinear/QuantizeLinear pairs.

Reproducing

Both variants are generated from the FP32 model.onnx in sentence-transformers/all-MiniLM-L6-v2.

# Install dependencies
pip install onnx onnxruntime

# Download source + generate both quantized variants
./generate.sh

# Or generate and upload to Hugging Face
# ./generate.sh --upload