gte-reranker-modernbert-base-GGUF

gte-reranker-modernbert-base-GGUF

GGUF conversions of Alibaba-NLP/gte-reranker-modernbert-base, a ModernBERT-based cross-encoder reranker, for use with llama.cpp.

All files were converted with convert_hf_to_gguf.py from upstream llama.cpp (which supports the ModernBertForSequenceClassification architecture and its reranker classification head) and validated against the original model's published reference scores — see the table below.

Files

At only ~150M parameters the absolute size savings from aggressive quantization are modest, and quantization erodes the fine-grained score discrimination that is the point of a reranker. F16 or Q8_0 is recommended unless you are tightly memory-constrained.

Usage

# Build llama.cpp for your backend (CUDA: -DGGML_CUDA=ON, Metal is on by default on macOS)
llama-server -m gte-reranker-modernbert-base-f16.gguf --reranking

# Then POST to /rerank (or /v1/rerank):
curl -X POST http://127.0.0.1:8080/rerank -H "Content-Type: application/json" -d '{
  "query": "what is the capital of China?",
  "documents": ["Beijing", "Shanghai"]
}'

Note on score scale: llama.cpp's /rerank returns the raw logit, not the 0–1 score that sentence-transformers' CrossEncoder.predict() returns. To match the original model's 0–1 scores, apply a sigmoid: score = 1 / (1 + exp(-logit)). Ranking order is identical either way (sigmoid is monotonic).

Validation

Each quant was checked against the original model card's two published reference pairs (after applying sigmoid to the GGUF logits) and against a 4-document ranking whose ground-truth order from transformers is [0, 2, 1, 3].

All quants preserve the correct ranking on the test cases; score fidelity degrades with quantization as expected.

License & attribution

Apache-2.0, inherited from the base model Alibaba-NLP/gte-reranker-modernbert-base. All credit for the model itself goes to the GTE team at Alibaba-NLP (arXiv:2308.03281). These are format conversions only.