fever-dpr-query-encoder-modernbert-base

Model Description

This model is the Query Encoder of a Dual-Encoder Dense Passage Retriever (DPR). Built upon answerdotai/ModernBERT-base, it is specifically designed to map input claims or queries into 768-dimensional dense vectors. The model was fine-tuned on the FEVER dataset to perform high-precision cosine similarity matching with corresponding evidence vectors in the vector space.

Data Processing

The training data originates from the pietrolesci/nli_fever dataset, with the following targeted processing applied:

• Sample Filtering: Retained only the samples with label == 0 (Entailment/Supports) in the train set.
• Positive Pair Construction: Used the premise column as the Query and the corresponding hypothesis column as the Positive Passage.
• Negative Sample Construction: Other irrelevant passages within the same batch automatically served as In-Batch Negatives.

Hyperparameters & Training Setup

• Model Architecture: Dual-Tower architecture (Query and Passage encoder weights are completely independent). The output sequence undergoes Mean-Pooling and L2 Normalization.
• Loss Function: InfoNCE Loss (In-Batch Negatives Cross-Entropy), Temperature = 0.05.
• Optimizer: AdamW
• Learning Rate: 2e-5
• Weight Decay: 0.01
• Warmup: Warmup Ratio = 0.1
• Batch Size: 32
• Epochs: 3
• Mixed Precision: bf16
• Hardware Environment: NVIDIA RTX 4090D (24GB VRAM)

Training Results

By the end of training (Epoch 3), the model demonstrated extremely strong convergence in pushing apart negative samples and pulling together positive samples:

• Final Loss: 0.0002
• Diag Similarity (Positive Diagonal Similarity): 15.5625
• All Similarity (Global Average Similarity): 1.7031

Note: The diagonal score is significantly higher than the global average score, proving that the model successfully constructed a highly discriminative vector space without experiencing representation collapse.

Citation

@misc{yuu-xie2026modernbert-base-dpr-fever,
  author = {Yuu-Xie},
  title = {fever-dpr-query-encoder-modernbert-base},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Yuu-Xie/fever-dpr-query-encoder-modernbert-base}}
}