jina-reranker-v3.5-mlx-q8

jina-reranker-v3.5-mlx-q8

8-bit quantized version of jinaai/jina-reranker-v3.5-mlx, a domain-ready multilingual listwise reranker optimized for Apple Silicon.

All Linear layers (attention + MLP) are quantized to 8-bit affine with group_size=64. Embeddings and the projector stay in bf16 for accuracy. 0.78 GB vs 1.19 GB (~35% smaller) with no measurable quality loss for reranking.

Benchmark (NDCG@10)

Measured against the bf16 original on MTEB reranking tasks (identical candidates across models):

q8 matches bf16 within noise (delta ≤ 0.0003). See BENCHMARK.md for methodology. A 4-bit variant is also available.

Differences from the original MLX repo

Loading

The quantized weights use the standard MLX quantization format. Loaders that support MLX safetensors (mlx-lm, oMLX) will dequantize on the fly:

from mlx_lm import load

model, tokenizer = load("underlotus/jina-reranker-v3.5-mlx-q8")

Note: the standalone rerank.py / MLXReranker helper in the original repo does not support quantized weights (its modeling.py has no dequantization logic). Use mlx-lm or oMLX to load this model.

Usage with oMLX

1. Download this repo into your oMLX model directory.
2. Keep a directory name containing reranker (e.g. jina-reranker-v3.5-mlx-q8) so oMLX routes it to its reranker engine.
3. Load it from the oMLX app. The Jina v3.5 sliding-window patch is applied automatically.

Quantization details

Quantization was performed on Apple Silicon with a small script:

import mlx.nn as nn
from mlx.utils import tree_flatten

# only Linear layers, 8-bit affine
nn.quantize(
    model,
    group_size=64,
    bits=8,
    class_predicate=lambda p, m: isinstance(m, nn.Linear),
)

Citation

If you find jina-reranker-v3.5 useful in your research, please cite the original paper:

@misc{nasika2026jinarerankerv35efficientlistwisereranker,
      title={jina-reranker-v3.5: An Efficient Listwise Reranker with Hybrid Attention and Self-Distillation},
      author={Christina Nasika and Feng Wang and Antonis Krasakis and Han Xiao},
      year={2026},
      eprint={2607.18152},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2607.18152},
}

License

CC BY-NC 4.0. For commercial usage inquiries, please contact Jina AI.