bert-dense-retriever

---

license: mit language:

• en library_name: transformers pipeline_tag: feature-extraction tags:
• bert
• dense-retrieval
• semantic-search
• information-retrieval
• ms-marco
• faiss

---

BERT Dense Retriever

A dense semantic retrieval model fine-tuned on the MS MARCO Passage Ranking dataset using BERT-base-uncased.

The model encodes natural language queries and passages into dense vector embeddings that can be indexed with FAISS for efficient semantic search.

This repository contains the complete Hugging Face compatible model including tokenizer, configuration, and custom model implementation.

---

Model Details

Backbone

• BERT-base-uncased

Pooling

• Mean Pooling

Embedding Normalization

• L2 Normalization

Similarity Metric

• Cosine Similarity

Training Objective

• CrossEntropy Loss over the similarity matrix (InfoNCE-style retrieval objective)

---

Training Configuration

---

Evaluation Results

The fine-tuned model substantially improves retrieval quality on the evaluation set compared with the untuned BERT-base encoder.

---

📊 BEIR Evaluation Results

The table below compares the proposed retrieval pipeline against the BM25 baseline reported by the original BEIR benchmark.

Note: BM25 scores are taken from the original BEIR benchmark and are included as the lexical retrieval baseline for comparison.

---

🔍 Key Findings

• Strong zero-shot generalization across multiple retrieval domains.
• Significant improvements over the BEIR BM25 baseline on FEVER, Quora, HotpotQA, and FiQA.
• Hybrid retrieval (BM25 + Dense Retriever) improves retrieval quality for specialized biomedical documents in TREC-COVID.
• Cross-Encoder re-ranking substantially enhances the final ranking quality by leveraging full query-document interactions.

---

Usage

from transformers import AutoTokenizer, AutoModel

model_name = "Innovatewithapple/bert-dense-retriever"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
)

inputs = tokenizer(
    "What is deep learning?",
    return_tensors="pt"
)

embeddings = model(**inputs)

print(embeddings.shape)

---

Intended Use

This model is designed for:

• Dense semantic retrieval
• Semantic search
• Question-passage retrieval
• Retrieval-Augmented Generation (RAG)
• Information retrieval research

---

Source Code

GitHub Repository:

https://github.com/Innovatewithapple/dense-semantic-retrieval

---

Author

Mihir Vyas