medembed-small-v0.1-onnx-int8

MedEmbed-small-v0.1 — reproducible ONNX INT8 export

This repository contains a compact ONNX Runtime export of abhinand/MedEmbed-small-v0.1 for local semantic search in Smart Search for Anki — Medical.

It is an independently produced derivative, not an official export from the upstream author. The source revision, every source and output checksum, the complete conversion recipe, and a numerical parity sanity check are included in this repository.

Artifact

The model accepts input_ids, attention_mask, and token_type_ids with dynamic batch and sequence dimensions. It returns one sentence_embedding tensor with 384 dimensions. The maximum configured sequence length is 512 tokens.

The graph already applies the upstream Sentence Transformers pipeline:

1. use the encoder's CLS token (last_hidden_state[:, 0, :]);
2. L2-normalize the 384-dimensional vector; and
3. dynamically quantize weights to signed INT8 for compact local inference.

The FP32 ONNX intermediate is reproducible but intentionally omitted to reduce the download by approximately 127 MB.

Source and lineage

• Immediate source: abhinand/MedEmbed-small-v0.1
• Immutable source revision: 40a5850d046cfdb56154e332b4d7099b63e8d50e
• Source model's declared license: Apache-2.0
• Base model reported by the source model card: BAAI/bge-small-en-v1.5
• Base model's declared license: MIT

See PROVENANCE.json for source file hashes, transform parameters, tool versions, and validation results. See NOTICE.md and the included license files for attribution and modification notices.

Reproduce the artifact

Use CPython 3.11 on macOS arm64 to reproduce the verified release environment:

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-model-export.txt
python export_medembed_onnx.py --output-dir build

The script downloads only the pinned upstream revision, verifies every required source file, checks the upstream pooling and normalization configuration, exports the FP32 graph, performs the documented INT8 quantization, runs ONNX validation and numerical parity checks, and refuses to complete unless both output files match their expected byte sizes and SHA-256 digests.

The exact verified INT8 digest is:

accb5b9e914356d01190c9f208ac822b345b24daeee4aa0fb345dfcc89a871d5

Bit-for-bit reproduction was confirmed in the pinned environment recorded in PROVENANCE.json. Other operating systems, Python patch versions, or package builds may produce a numerically equivalent graph with different serialized bytes; such an output is not the release artifact unless its digest matches.

Numerical parity sanity check

Twelve short, synthetic medical and general-language sentences were embedded through the same Rust tokenizers plus ONNX Runtime path used by the add-on. The resulting vectors were compared with:

• the pinned upstream PyTorch encoder using the exact same token IDs, its configured CLS pooling, and L2 normalization;
• the FP32 ONNX export; and
• the INT8 ONNX export, followed by the add-on's final L2 normalization.

Both exports returned finite 12 × 384 arrays. After the same final normalization used by Smart Search, the maximum INT8 unit-norm error was 5.96e-8. This is an export-integrity sanity check on a small synthetic set. It is not a clinical evaluation, a retrieval benchmark, or evidence that quantization preserves every ranking on every collection.

Minimal local inference

import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
tokenizer.enable_truncation(max_length=512)
tokenizer.enable_padding()

encodings = tokenizer.encode_batch(["What causes elevated creatinine?"])
feeds = {
    "input_ids": np.asarray([item.ids for item in encodings], dtype=np.int64),
    "attention_mask": np.asarray(
        [item.attention_mask for item in encodings], dtype=np.int64
    ),
    "token_type_ids": np.asarray(
        [item.type_ids for item in encodings], dtype=np.int64
    ),
}

session = ort.InferenceSession(
    "onnx/model_int8.onnx",
    providers=["CPUExecutionProvider"],
)
embedding = session.run(["sentence_embedding"], feeds)[0]

Intended use

This export is intended for local English-language semantic retrieval, such as finding related material within a user's own study notes. It produces text embeddings; it does not answer clinical questions or provide medical facts.

Limitations and safety

• The model can miss relevant text or rank unrelated text highly.
• INT8 quantization changes embeddings slightly and can change close rankings.
• The source model card reports medical retrieval training, but this export has not been independently clinically validated.
• The upstream model card names several training/evaluation datasets, and the upstream project describes synthetic training triplets derived from PubMed Central material. This conversion does not redistribute those datasets and did not independently audit their exact snapshots, provenance, or license chain.
• Performance outside English and beyond the 512-token configuration is not established here.
• Do not use embeddings as medical advice, diagnosis, treatment guidance, medication reconciliation, or an emergency decision tool.
• Do not infer that the upstream authors, BAAI, Hugging Face, Anki, or any named dataset endorses this export or Smart Search.

Privacy

The model files contain no Anki collection data. In Smart Search, embedding inference runs locally. Downloading these public files may expose ordinary network metadata, such as an IP address and user agent, to the hosting provider; card text and search queries are not included in model download requests.

Credits

• Abhinand Balachandran, MedEmbed: Medical-Focused Embedding Models
• Beijing Academy of Artificial Intelligence, BGE embedding models
• Reproducible ONNX export and quantization: Saleh Mostafa / MedBrevia

Prepared for the open-source Smart Search for Anki — Medical add-on. This repository is not affiliated with or endorsed by Anki or its authors.