flan-t5-base-xsum-lora

FLAN-T5 Base XSum LoRA Adapter

This repository provides a LoRA fine-tuned adapter for
:contentReference[oaicite:1]{index=1}
trained on the
:contentReference[oaicite:2]{index=2}
for abstractive text summarization.

The adapter uses parameter-efficient fine-tuning (PEFT) to adapt the base model while keeping most of its parameters frozen.

---

Model Details

• Base model: google/flan-t5-base
• Task: Abstractive Summarization
• Fine-tuning method: LoRA
• Dataset: EdinburghNLP/xsum
• Framework: Transformers + PEFT

---

Summary

---

Training Procedure

The model was fine-tuned on the XSum dataset, which contains BBC news articles paired with a single-sentence summary.

Training used LoRA adapters applied to the attention projection layers while keeping the base model frozen.
This reduces the number of trainable parameters while maintaining strong summarization performance.

---

How to Use

Install dependencies:

pip install transformers peft accelerate


# Load the adapter:
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")

model = PeftModel.from_pretrained(base_model, "A7med-Ame3/flan-t5-base-xsum-lora")

text = "summarize: Artificial intelligence is transforming many industries..."

inputs = tokenizer(text, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=60)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Next Steps

# لإry target_modules=["q", "k", "v", "o"] for potentially better ROUGE scores

# Increase LoRA rank to r=32 for more expressive adapters

# Use google/flan-t5-large for higher-quality production models

# Tune beam search parameters to improve inference quality