t5-small-custom

Model Card for T5-Small Summarization Model

Model Details

• Model Name: T5-Small (Text-to-Text Transfer Transformer)
• Model Type: Summarization and Text Generation
• Developer: Google Research
• Architecture: Transformer-based encoder-decoder model, part of the T5 (Text-to-Text Transfer Transformer) family.
• Size: Small version with approximately 60 million parameters.
• Languages: Primarily trained on English text.

Intended Use

T5-Small is designed to handle various text-to-text tasks, including:

• Summarization
• Translation
• Text classification (reframed as a text generation task)
• Question answering

This model is best suited for environments with limited computational resources, as its small size enables faster training and inference times compared to larger models in the T5 family.

Training Data

The T5-Small model was pre-trained on the C4 (Colossal Clean Crawled Corpus), which is a large dataset composed of cleaned and filtered web text. This dataset represents a broad range of topics, styles, and writing forms, which allows the model to generalize well across various NLP tasks.

• Corpus Size: The C4 dataset contains approximately 750 GB of English text.
• Task Formulation: All NLP tasks, including summarization, were framed as text-to-text tasks, where both input and output are represented as natural language text.

Training Procedure

• Pretraining Objective: The model was trained using a denoising autoencoder objective, where the input text is corrupted and the model is tasked with reconstructing the original text. The corruption strategy involves masking spans of text, which aligns closely with summarization tasks.
• Summarization Fine-Tuning: The model can be fine-tuned on specific summarization datasets such as CNN/Daily Mail or other custom datasets by converting them into text-to-text format.
• Optimization: Trained using the Adam optimizer with standard hyperparameters (learning rate, batch size, etc.).

Training Configuration:

• Batch Size: Typically 64–128 for fine-tuning on summarization tasks.
• Learning Rate: A range of 1e-5 to 3e-5, depending on the specific task and dataset.
• Epochs: Usually 3–5 epochs for summarization tasks, but this can vary.

How to Use

You can easily use the T5-Small model for summarization using the HuggingFace transformers library.

from transformers import T5ForConditionalGeneration, T5Tokenizer

# Load the T5-small model and tokenizer
model = T5ForConditionalGeneration.from_pretrained('t5-small')
tokenizer = T5Tokenizer.from_pretrained('t5-small')

# Example text input for summarization
text = 
The Palestinian Authority officially became the 123rd member of the International Criminal Court on Wednesday, 
a step that gives the court jurisdiction over alleged crimes in Palestinian territories.
The formal accession was marked with a ceremony at The Hague, in the Netherlands, where the court is based.

# Preprocess the input text
input_ids = tokenizer.encode("summarize: " + text, return_tensors='pt', truncation=True)

# Generate the summary
summary_ids = model.generate(input_ids, max_length=50, min_length=20, length_penalty=2.0, num_beams=4, early_stopping=True)

# Decode the summary
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print(summary)

Hyperparameters to Tune

• Max Length/Min Length: Controls the maximum and minimum lengths of the generated summaries.
• Num Beams: Controls the number of beams for beam search during text generation (default is 4 for better quality summaries).
• Length Penalty: Adjusts the length of the output; higher values encourage shorter summaries, lower values allow for longer ones.

Evaluation

Metrics

• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Commonly used to evaluate summarization models.
  • ROUGE-1, ROUGE-2, ROUGE-L: Measures overlap between the model’s generated text and the reference summaries, evaluating both unigram and bigram precision/recall.
• BLEU (Bilingual Evaluation Understudy): Sometimes used for summarization to measure n-gram overlap, though more common in translation tasks.

Performance

• Performance on Summarization: The T5-Small model, though compact, performs competitively on standard summarization datasets, but with reduced accuracy and fluency compared to larger versions like T5-Base or T5-Large.

Limitations

• Model Size: Being a smaller version of the T5 family, T5-Small trades off some accuracy and fluency in favor of efficiency. For more complex summarization tasks or longer documents, larger versions of T5 may yield better results.
• Inference Time: Though faster than larger models, it may still experience delays in generating summaries for long texts.
• Summarization Quality: T5-Small may sometimes generate oversimplified or factually incorrect summaries, especially for highly nuanced or complex text.
• Limited Training Data: The pretraining data might not cover very specific domains or rare events.

Ethical Considerations

• Bias: T5-Small inherits any biases present in the C4 dataset, which is sourced from general web text. This may include biases related to race, gender, or political views, and care should be taken when applying the model in sensitive contexts.
• Hallucination: Like many generative models, T5-Small may produce incorrect or fabricated information, particularly when the input text is ambiguous or incomplete. It is important to verify the model’s output, especially in critical applications.
• Data Privacy: The model was trained on publicly available web data, but users should ensure that no sensitive or private information is being input into the model, as there are risks of memorization in large-scale models.