NPCAlign-DPO

NPCAlign DPO — NPC Quest Dialogue LoRA (SFT + DPO)

LoRA adapter further fine-tuned via Direct Preference Optimisation (DPO) on top of the SFT model. Trained to generate more natural conversation endings and diverse NPC responses.

This adapter is applied on top of the merged SFT model, not directly on the base Llama model. See Usage section.

Model Details

• Base: meta-llama/Meta-Llama-3.1-8B-Instruct + SFT weights merged
• Method: DPO with LoRA (rank 16)
• Preference data: 1,341 (chosen, rejected) pairs generated from SFT model outputs, scored by Gemma 4 26B judge on 5 criteria
• Beta: 0.1

Usage

Note: The base model meta-llama/Meta-Llama-3.1-8B-Instruct is a gated model. You must accept Meta's license and set your HF_TOKEN before loading.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "HermitQ/NPCAlign-DPO")
tokenizer = AutoTokenizer.from_pretrained("HermitQ/NPCAlign-DPO")

Step 1: Load base + SFT, merge

sft = PeftModel.from_pretrained(base, "HermitQ/NPCAlign-SFT")
merged = sft.merge_and_unload()

Step 2: Apply DPO adapter on merged model

model = PeftModel.from_pretrained(merged, "HermitQ/NPCAlign-DPO")

DPO Training Details

Evaluation vs SFT Baseline

Self-BLEU decrease indicates more diverse generation.

GitHub

GitHub link: Hermit888/NPCAlign