bge-reranker-base-nq-ft

bge-reranker-base fine-tuned on Natural Questions (train split)

A cross-encoder reranker for RAG retrieval: BAAI/bge-reranker-base fine-tuned on hard-negative groups mined from the Natural Questions train split. It was produced as Stage 8 of a controlled ablation study of RAG chunking, where the diagnosis was explicit - at the best chunk size the answer was already inside the BGE top-20 pool for ~96% of questions while Recall@1 sat at ~0.63, so the bottleneck was ranking within the pool.

Training

Every evaluation below uses the NQ validation split, disjoint from training.

Results

In-domain (NQ, 1000 docs / 1032 questions). delta vs the off-the-shelf reranker, with all arms reranking one identical BGE top-20 pool (2 SE = 0.030):

At fixed 15/0 this closes about a third of the gap between dense R@1 (0.628) and the pool ceiling (0.964). R@3 and R@5 rise too; no metric trades down.

Cross-dataset (TriviaQA rc.wikipedia, 472 docs / 300 questions, 2 SE = 0.054). Cross-dataset limitations:

On the tested TriviaQA bench, the off-the-shelf reranker lowered R@1 by 0.03...0.07 relative to dense retrieval. This model returned to approximately the dense baseline. Its difference from the off-the-shelf reranker was positive at 5/5 configs and above the approximate 2 SE band at 4/5 configs, but the cross-dataset result remained approximately equal to dense retrieval.

The reported improvement is specific to NQ. The TriviaQA result does not establish expected behavior on other datasets.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("sfczaa/bge-reranker-base-nq-ft", max_length=512)
scores = model.predict([(question, passage) for passage in candidates])
ranked = [c for _, c in sorted(zip(scores, candidates), reverse=True)]

Drop-in replacement for BAAI/bge-reranker-base - same architecture, tokenizer and 512-token truncation. Reranking costs ~0.56 s/question at depth 20 on a T4, unchanged by fine-tuning.

Caveats

• Trained and validated on Natural Questions; questions are split-disjoint, but popular Wikipedia pages can appear in both splits' corpora (standard for NQ).
• The cross-dataset evaluation uses distant-supervised TriviaQA gold (answer string match on entity pages), weaker than NQ's annotated gold; absolute recall is not comparable between the two benchmarks.
• Evaluated only on English Wikipedia-style passages at 6-15 sentence chunks.

License

MIT, matching the BAAI/bge-reranker-base base model.