t5-moe-topk-xsum

t5-moe-xsum_token_choice

Model Description

T5 with MoE (Token Choice Routing) fine-tuned on the XSUM dataset for abstractive summarization.

Architecture

This model uses Sparse Mixture of Experts with learned Token Choice Top-k routing.

Key Features:

• Learned gating network for expert selection
• Top-k routing (each token routed to 2 experts)
• Optional load balancing loss

Model Configuration

• Base Model: google-t5/t5-small
• Total Parameters: 236,716,544
• Trainable Parameters: 236,716,544
• Number of Experts: 8
• Top-k: 2
• Routing Strategy: token_choice
• Load Balancing: Enabled

Training Data

The model was trained on the XSUM dataset, which contains:

• ~204k training examples
• ~11k validation examples
• ~11k test examples

Each example consists of a BBC news article and a one-sentence summary.

Usage

from transformers import T5Tokenizer

# Load tokenizer
tokenizer = T5Tokenizer.from_pretrained("YOUR_USERNAME/t5-moe-xsum_token_choice")

# Note: For MoE models, you need to reconstruct the architecture
# See the model repository for detailed loading instructions

Evaluation

Evaluate using standard ROUGE metrics and SummaC consistency scores.

Training Procedure

The model was trained using:

• AdamW optimizer with weight decay
• Learning rate: 5e-5
• Warmup steps: 500
• Mixed precision (FP16) training
• Gradient accumulation for larger effective batch size

Limitations

• Trained only on English news articles
• May not generalize well to other domains
• MoE models require custom loading code

Citation

If you use this model, please cite the XSUM dataset:

@inproceedings{narayan-etal-2018-dont,
    title = "Don{'}t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization",
    author = "Narayan, Shashi and Cohen, Shay B. and Lapata, Mirella",
    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",
    year = "2018",
}