T5-base-finetuned-Xsum

T5-base Fine-tuned on XSum

This repository provides a T5-base model fine-tuned on the XSum dataset for abstractive summarization.
Given a document, the model generates a concise one-sentence summary.

---

Dataset

• Name: XSum (EdinburghNLP/xsum)
• Task: Extreme summarization — generating a concise one-sentence summary from a longer news article.
• Data Splits:
  • train: ~204k samples
  • validation: ~11k samples
  • test: ~11k samples
• Fields:
  • document: news article (input).
  • summary: one-sentence summary (target).
  • id: BBC article ID.

---

Model & Method

• Base model: flax-community/t5-base-cnn-dm
• Fine-tuning task: Seq2Seq summarization
• Loss function: Cross-entropy with label smoothing (default in HF Seq2SeqTrainer)
• Training strategy: Full fine-tuning (no adapters)

---

Training Setup

• Epochs: 10
• Train batch size: 16
• Eval batch size: 16
• Learning rate: 3e-5
• Optimizer: adamw_torch_fused
• Scheduler: linear with warmup ratio 0.1
• Gradient accumulation: 4
• Weight decay: 0.01
• Mixed precision: FP16 enabled
• Gradient checkpointing: Enabled
• Evaluation strategy: per epoch
• Prediction strategy: generate during eval
• Logging: TensorBoard (./logs)

---

Results

Final evaluation results after 10 epochs:

---

How to Use

Summarization Inference

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "mia-project-2025/T5-base-finetuned-Xsum"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

document = """The 2019 Rugby World Cup final was played in Yokohama, Japan, 
where South Africa defeated England 32-12 to win their third title."""

inputs = tokenizer("summarize: " + document, return_tensors="pt", max_length=768, truncation=True)

summary_ids = model.generate(inputs["input_ids"], num_beams=4, max_length=128, early_stopping=True)
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)

print("Generated Summary:", summary)