flan-t5-base-dialogsum-qlora

FLAN-T5-Base Fine-tuned for Dialogue Summarization

This model is a fine-tuned version of google/flan-t5-base on the knkarthick/dialogsum dataset.

Training Details

• Base Model: google/flan-t5-base
• Dataset: knkarthick/dialogsum
• Training Method: QLoRA (4-bit quantization + LoRA adapters)
• LoRA Rank: 16
• Training Steps: 1000
• Final Validation Loss: 0.396802

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

# Load base model and tokenizer
base_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")

# Load the fine-tuned LoRA adapter
model = PeftModel.from_pretrained(base_model, "RaamBhakt2609/flan-t5-base-dialogsum-qlora")

# Generate summary
dialogue = "Your dialogue text here..."
input_text = f"summarize: {dialogue}"
input_ids = tokenizer(input_text, return_tensors="pt", max_length=512, truncation=True).input_ids

with torch.no_grad():
    outputs = model.generate(input_ids, max_length=128, num_beams=4, early_stopping=True)

summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(summary)

Training Configuration

• LoRA Alpha: 32
• LoRA Dropout: 0.05
• Target Modules: ["q", "v", "o", "wi", "wo"]
• Batch Size: 4 (per device)
• Gradient Accumulation Steps: 2
• Learning Rate: 3e-4
• Optimizer: paged_adamw_8bit