serana-dpo

serana-dpo

"Serana" and The Elder Scrolls are property of Bethesda/ZeniMax. This adapter is a non-commercial engineering portfolio artifact, not an official product, and is not affiliated with Bethesda/ZeniMax.

Stage: DPO, part of a CPT -> SFT -> DPO post-training pipeline for persona consistency, built end-to-end on one 24GB GPU (NVIDIA L4). Full pipeline, data sourcing, evaluation design, and GPU engineering: https://github.com/MachuEngine/serana-post-training

Continues serana-sft with DPO on 837 RLAIF preference pairs (judge picks between two SFT-sampled replies per prompt). Note: in this project's own evaluation, DPO showed no CI-confirmed quality gain over SFT on any metric -- see the repo's results tables. Shipped anyway, as the honest result of the pipeline, not because it won.

Built on top of machu8/serana-sft, not the base model directly.

Usage

This is a LoRA adapter, not a standalone model -- it requires the base model (Qwen/Qwen3-8B) to be loaded first.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(base, "machu8/serana-dpo")

See the repo's PROMPTS.md §1 for the exact persona system prompt this was trained/evaluated with -- results are only comparable when it's reused as-is.

Results

Both the quality table (PCS / PRS / style similarity / knowledge-boundary accuracy / mean reply length, all with 95% CIs) and the hardware table (VRAM, KV-cache, throughput, AWQ vs bf16) are in the repo's artifacts/runs/results_quality.md and results_hardware.md, generated directly from this adapter's real eval runs.