Qwen2.5-3B-Instruct-NQ-RAG-DPO-LoRA

Introduction

This repository contains a LoRA adapter fine-tuned using Direct Preference Optimization (DPO) over a Retrieval-Augmented Generation (RAG) evaluation pipeline built on the Natural Questions validation set.

Training Pipeline

1. Base Model: Qwen/Qwen2.5-3B-Instruct

2. RAG responses generated over NQ validation split

3. Responses scored using custom reward signals:

• Faithfulness

• Citation usage

• Hallucination detection

• Refusal detection

1. Preference pairs constructed using margin filtering

2. LoRA fine-tuning using DPO

Dataset Lineage

This model is trained and evaluated using:

Dataset repository:

AnjanSB/NQ-RAG-DPO-Evaluation

Configurations used:

• rag_responses (base + trained generations)

• responses_scores (reward signals)

• dpo_train_data (preference dataset)

• comparison_metrics (evaluation results)

Evaluation Summary

The Train Data which is used to train base model indicates small in size(228 Total records) and high quality(2.18 a significant jump of mean response margin between choosen and rejected responses of a same prompt).

The trained model shows:

• Significantly improved responses margin which indicates stable responses.

• Improved citation usage

• Reduced hallucination

• Slightly improved faithfulness

• Stable refusal behavior

👤 Author

AnjanSB

Experiment Repo : https://dagshub.com/AnjanSB/RAG-DPO-PEFT-LLMOPS

Profile : https://www.linkedin.com/in/anjansb/