bge-reranker-v2-m3-mlx-int8

bge-reranker-v2-m3 · MLX Swift 8-bit

A multilingual cross-encoder reranker for on-device search on Apple Silicon, at 0.64 GB.

Intended use

Re-ranking retrieval candidates and judging whether two texts are about the same thing, in 100+ languages, fully on-device. Feed (query, passage) as a sentence pair; the single output logit (sigmoid for a 0–1 score) ranks candidates.

How it was made

The original fp32 weights were loaded with an MLX Swift BertModel (MLXEmbedders naming, after its sanitize step) and quantized in memory with MLXNN.quantize (affine, 8-bit, group size 64, default Linear and Embedding layers). The quantized module parameters were saved unchanged as model.safetensors; quantization.json records the scheme.

Verification

An item-linking benchmark of four multilingual packs (English, Czech and a multilingual parliament set) uses this model together with two companion models from the same family of repos (embedder, reranker, NLI). It produced byte-identical outputs with these 8-bit files and with the original fp32 weights.

Loading (read this first)

These are MLX Swift weights, not a Transformers checkpoint: transformers and sentence-transformers cannot load them. Use the original repo for those. To load in MLX Swift:

1. Build the MLXEmbedders BertModel from config.json (with the sequence-classification head).
2. Call quantize(model: model, groupSize: 64, bits: 8, mode: .affine) on it before loading weights. Quantized layers store weight / scales / biases.
3. Load model.safetensors into the module and use the tokenizer files as usual.

quantization.json ({"bits": 8, "groupSize": 64, "mode": "affine"}) is the marker a loader can check to take step 2.

Files

License and attribution

Apache License 2.0, the same as BAAI/bge-reranker-v2-m3; the text is in LICENSE. All credit for the model goes to BAAI. The change here is quantization only.