pegasus_indonesian_base-pretrain

pegasus_indonesian_base-pretrain

Github : PEGASUS TPU Trainer

This model is a pretrained version of pegasus_indonesian_base-finetune on kaggle id news 2017, CC_News_id, and OSCAR_2201.

It achieves the following results on the evaluation set:

• Train Loss: 2.34832262992858
• Train Accuracy: 0.262173235416412
• Validation Loss: 2.34894156455993
• Validation Accuracy: 0.266122311353683
• Train Lr: 0.000136618677061051
• Epoch: 40

Intended uses & limitations

This model is uncased, can't read special characters except "," and ".", having hard time understanding numbers, and performance only tested on news article text.

Training and evaluation data

Pretrain dataset:

1. kaggle id news 2017
2. CC_News_id
3. OSCAR_2201

Training procedure

For replication, go to GitHub page

Training hyperparameters

The following hyperparameters were used during training:

• optimizer: {'name': 'Adafactor', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': False, 'is_legacy_optimizer': False, 'learning_rate': 0.005, 'beta_2_decay': -0.8, 'epsilon_1': 1e-30, 'epsilon_2': 0.001, 'clip_threshold': 1.0, 'relative_step': True}
• training_precision: float32

Usage

# Load model hyperparameters
from transformers import PegasusConfig,TFPegasusForConditionalGeneration,PegasusTokenizerFast
configuration = PegasusConfig()
configuration.vocab_size = 32103
configuration.d_model = 512
configuration.dropout = 0.15
configuration.decoder_attention_heads = 8
configuration.decoder_layers = 12
configuration.decoder_ffn_dim = 3072
configuration.encoder_attention_heads = 8
configuration.encoder_layers = 12
configuration.encoder_ffn_dim = 3072

# Load model and tokenizer
# Download the weights and manually load weights using Tensorflow
model = TFPegasusForConditionalGeneration(configuration)
model.load_weights("checkpoints-pegasus_indonesian_base-pretrain-weights")
tokenizer = PegasusTokenizerFast.from_pretrained("thonyyy/pegasus_indonesian_base-finetune")

Training results

Framework versions

• Transformers 4.30.2
• TensorFlow 2.12.0
• Datasets 2.13.1
• Tokenizers 0.13.3

Special Thanks

Research supported with Cloud TPUs from Google’s TPU Research Cloud (TRC)