lightretriever-llama3.2-3b

This model was trained by Dhruv as part of a RAG research internship at IIT Bombay for an IBM-affiliated project

LightRetriever Llama-3.2-3B (Full Training Run)

A LoRA-fine-tuned dense retriever built on Llama-3.2-3B, reproducing LightRetriever — an LLM adapted into a bi-encoder for query/passage embedding via contrastive learning.

• Base model: meta-llama/Llama-3.2-3B
• Type: Dense retriever (bi-encoder), LoRA adapter
• Language: English
• License: Inherits Llama 3.2 Community License

Uses

Encode queries/passages into embeddings for retrieval or RAG pipelines. Not a generation/chat model.

Training

• Data: ~42GB subset (18 English domains) of lightretriever/lightretriever-finetune-data — includes msmarco, nq, hotpotqa, fever, trivia, and others. 1 query + 1 positive + 7 hard negatives per example.
• Method: LoRA (r=16, alpha=32, dropout=0.1) contrastive fine-tuning, InfoNCE-style loss, in-batch + cross-device negatives
• Trainable params: 12,242,944 (adapter only, ~97MB)
• Steps: 5,000 (full run), batch size 32 (2 GPUs × 16), lr 2e-5 cosine, bf16, seq len 512
• Hardware: 2× RTX A6000 (48GB)

Evaluation

MS MARCO dev set (8.84M passages, 6,980 queries) via MTEB retrieval benchmark (mteb v1.39.7).

Evaluation time: ~14.1 hours (single GPU, corpus chunk size 2M).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B")
model = PeftModel.from_pretrained(base_model, "dhruvv45/lightretriever-llama3.2-3b-full")
tokenizer = AutoTokenizer.from_pretrained("dhruvv45/lightretriever-llama3.2-3b-full")

Note: adapter only — full inference needs the lightretriever source for pooling/prompt logic.

Framework versions

• PEFT 0.20.0
• Transformers 4.57.6
• PyTorch 2.5.1+cu121

Author

Dhruv Agarwal