modernbert-ja-130m-int8-onnx

modernbert-ja-130m-onnx

ONNX export of sbintuitions/modernbert-ja-130m (132.5M params, 19 layers, RoPE, local+global attention), converted for the feature-extraction task so it outputs last_hidden_state for building sentence embeddings via mean pooling.

Two variants are provided in onnx/:

Benchmark

Measured on a CPU-only sandbox (onnxruntime CPUExecutionProvider), single-sequence inference, over a 20-sentence Japanese set covering paraphrase pairs, related pairs, and unrelated pairs. Quality is reported two ways against the original PyTorch fp32 model: mean cosine similarity of individual embeddings, and Spearman rank correlation of the full pairwise similarity matrix (i.e. does the variant preserve which sentences rank as more/less similar — what actually matters for a sentence-similarity use case).

Takeaways:

• Plain ONNX export (onnx/model.onnx) is a free win: 2.4x lower latency with zero quality loss (cosine similarity 1.0000 against the original PyTorch outputs). Use this when fidelity matters most and storage isn't the constraint.
• Dynamic INT8 quantization (onnx/model_quantized.onnx) adds a further 2.2x latency reduction on top of ONNX fp32 (5.3x vs. the original PyTorch model) and cuts file size ~4x, at the cost of some embedding precision (mean cosine similarity 0.990, ranking correlation 0.933 rather than a perfect 1.0). In our test this did not flip the relative order of clearly-similar vs. clearly-dissimilar sentence pairs, but the ranking correlation is noticeably below the fp32 variants, so it's worth validating against your own similarity/retrieval eval set before relying on it for fine-grained ranking.
• An fp16 export was also tested; it kept near-perfect quality (cosine 0.9999997, ranking correlation 0.99997) but showed no latency benefit in this CPU-only sandbox (fp16 speedups require GPU tensor cores) and CPU fp16 kernels in onnxruntime/PyTorch are numerically less mature. It isn't included here — happy to add onnx/model_fp16.onnx if you plan to serve on GPU.

Recommendation: use onnx/model_quantized.onnx if you need the smallest footprint and fastest CPU inference and can tolerate ~1-2% embedding drift; use onnx/model.onnx if you want the latency win from ONNX with no measurable quality change.

Usage

With onnxruntime directly

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("<this-repo>")
session = ort.InferenceSession("onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])

def embed(sentences):
    enc = tokenizer(sentences, padding=True, return_tensors="np")
    last_hidden_state, = session.run(
        None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]}
    )
    mask = enc["attention_mask"][..., None].astype(np.float32)
    summed = (last_hidden_state.astype(np.float32) * mask).sum(axis=1)
    counts = np.clip(mask.sum(axis=1), 1e-9, None)
    return summed / counts  # mean-pooled sentence embeddings

embs = embed(["今日はいい天気ですね。", "本日は晴れて気持ちがいいです。"])
cos_sim = (embs[0] @ embs[1]) / (np.linalg.norm(embs[0]) * np.linalg.norm(embs[1]))
print(cos_sim)

With optimum

from optimum.onnxruntime import ORTModelForFeatureExtraction
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("<this-repo>")
model = ORTModelForFeatureExtraction.from_pretrained("<this-repo>", file_name="onnx/model_quantized.onnx")

Conversion details

• Exported with optimum (optimum.exporters.onnx), task feature-extraction, opset 18.
• Dynamic INT8 quantization via onnxruntime.quantization.quantize_dynamic (QInt8 weights).
• No pooling head is baked into the graph — apply mean pooling (or your pooling of choice) over last_hidden_state using the attention mask, as shown above. The base model has no officially recommended pooling strategy since it's released as a masked-language-model checkpoint rather than a tuned sentence-embedding model; mean pooling is the standard default for this class of encoder.

Base model card, license (MIT), and full training details: sbintuitions/modernbert-ja-130m.