ReframeBot-DPO-Llama3.1-8B

ReframeBot-DPO-Llama3.1-8B

LoRA adapter for meta-llama/Meta-Llama-3.1-8B-Instruct, further aligned with Direct Preference Optimisation (DPO) on top of the SFT checkpoint (ReframeBot-SFT-Llama3.1-8B).

DPO training steered the model towards empathetic, open-ended Socratic responses and away from direct advice, dismissiveness, or unsafe content. This is the production adapter used in the ReframeBot system.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

base = "meta-llama/Meta-Llama-3.1-8B-Instruct"
adapter = "Nhatminh1234/ReframeBot-DPO-Llama3.1-8B"

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

Training Details

Dataset: 1,400 preference pairs {prompt, chosen, rejected} generated with GPT-4. Chosen responses demonstrate empathy + open-ended questioning; rejected responses contain direct advice, dismissiveness, or unsafe content.

Evaluation

Intended Use

Designed as a component in the ReframeBot system — not a standalone mental-health tool. Must not be used for clinical intervention or crisis support without human oversight.

Project

GitHub: ReframeBot