ms-marco-MiniLM-L-6-v2-4bit-nf4

MiniLM-L6-v2 Cross-Encoder (4-bit NF4 Quantized)

A 4-bit NF4 quantized version of cross-encoder/ms-marco-MiniLM-L-6-v2 for passage reranking, using bitsandbytes quantization.

Quantization Details

Evaluation

Evaluated on three IR benchmarks using a BM25 (top-100) + neural reranking pipeline.

LitSearch (Academic Literature Search)

SciFact (Scientific Fact Verification)

NFCorpus (Biomedical IR)

Summary

4-bit NF4 quantization preserves near-identical quality across all three benchmarks:

Usage

With transformers

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model = AutoModelForSequenceClassification.from_pretrained(
    "MO7YW4NG/ms-marco-MiniLM-L-6-v2-4bit-nf4",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "MO7YW4NG/ms-marco-MiniLM-L-6-v2-4bit-nf4",
)

query = "What is the impact of climate change on coral reefs?"
passage = "Rising ocean temperatures cause widespread coral bleaching events..."

inputs = tokenizer(
    query, passage,
    return_tensors="pt",
    truncation=True,
    max_length=512,
    padding=True,
).to(model.device)

with torch.no_grad():
    score = model(**inputs).logits.squeeze().item()
print(f"Relevance score: {score:.4f}")

With sentence-transformers CrossEncoder

from sentence_transformers.cross_encoder import CrossEncoder

model = CrossEncoder(
    "MO7YW4NG/ms-marco-MiniLM-L-6-v2-4bit-nf4",
    max_length=512,
)

query = "What is the impact of climate change on coral reefs?"
passages = [
    "Rising ocean temperatures cause widespread coral bleaching events...",
    "The history of marine biology dates back to ancient Greece...",
]

pairs = [[query, p] for p in passages]
scores = model.predict(pairs)
print(scores)

Technical Notes

• The classifier head is kept in fp16 (not quantized) to maintain output precision.
• Requires bitsandbytes and a CUDA-capable GPU at inference time.
• Model size on disk: ~17 MB (vs ~88 MB for fp32).

Citation

Base model:

@misc{ms-marco-MiniLM-L-6-v2,
  title={MS MARCO Cross-Encoder MiniLM-L-6-v2},
  author={Nils Reimers},
  url={https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2},
}