mt5-small-sum-de-en-v2

mT5-small-sum-de-en-v2

This is a bilingual summarization model for English and German. It is based on the multilingual T5 model google/mt5-small.

Training

The training was conducted with the following hyperparameters:

• base model: google/mt5-small
• source_prefix: "summarize: "
• batch size: 3
• max_source_length: 800
• max_target_length: 96
• warmup_ratio: 0.3
• number of train epochs: 10
• gradient accumulation steps: 2
• learning rate: 5e-5

Datasets and Preprocessing

The datasets were preprocessed as follows:

The summary was tokenized with the google/mt5-small tokenizer. Then only the records with no more than 94 summary tokens were selected.

The MLSUM dataset has a special characteristic. In the text, the summary is often included completely as one or more sentences. These have been removed from the texts. The reason is that we do not want to train a model that ultimately extracts only sentences as a summary.

This model is trained on the following datasets:

Evaluation on MLSUM German Test Set (no beams)

Evaluation on CNN Daily English Test Set (no beams)

Evaluation on Extreme Summarization (XSum) English Test Set (no beams)

♣: These values seem to be unusually high. It could be that the test set was used in the training data.

License

Copyright (c) 2021 Philip May, T-Systems on site services GmbH

This work is licensed under the Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0) license.