RLHF-Flant5

RLHF Flan-T5: Low-Toxicity Dialogue Summarization

A fine-tuned version of google/flan-t5-base trained to produce concise, low-toxicity summaries of conversational dialogues. Training followed a two-stage pipeline: supervised fine-tuning with LoRA, followed by reinforcement learning from human feedback (RLHF) using PPO with a toxicity classifier as the reward signal.

Training Pipeline

Stage 1 — Supervised Fine-Tuning with LoRA

The base model was first fine-tuned on the DialogSum dataset using PEFT LoRA.

Stage 2 — RLHF with PPO

The LoRA-adapted model was wrapped with a value head and trained with PPO using a hate-speech detection model as the reward signal.

How to Use

Load the RLHF adapter (recommended)

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

base_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")
model = PeftModel.from_pretrained(base_model, "YashNagraj75/RLHF-Flant5/Flant5-RLHF")
tokenizer = AutoTokenizer.from_pretrained("YashNagraj75/RLHF-Flant5")

dialogue = "Person A: ...\nPerson B: ..."
prompt = f"Summarize the following conversation.\n\n{dialogue}\n\nSummary:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Load the SFT-only adapter

model = PeftModel.from_pretrained(base_model, "YashNagraj75/RLHF-Flant5/flan-t5-lora")

Repository Contents

Framework Versions

• PyTorch, Transformers, PEFT, TRL, Evaluate

License

MIT