Bangla_GPT2

Bangla GPT2 model was trained using the Bangla Newspaper dataset. Here we used prothom alo 250mb data for GPT2 model training and also vocab size 50k.

Github link : https://github.com/saiful9379/Bangla_GPT2

from transformers import TFGPT2LMHeadModel, GPT2Tokenizer

tokenizer =  GPT2Tokenizer.from_pretrained("saiful9379/Bangla_GPT2")

model = TFGPT2LMHeadModel.from_pretrained("saiful9379/Bangla_GPT2")
text = "বহুল আলোচিত দশম জাতীয় সংসদ"
input_ids = tokenizer.encode(text, return_tensors='tf')

print(input_ids)


output = model.generate(
        input_ids,
        max_length=175,
        num_beams=10,
        temperature=0.7,
        no_repeat_ngram_size=2,
        num_return_sequences=5
    )
predicted_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(predicted_text)

Here is the basic configuration of Bangla GPT2 Model,

vocab_size = 50000
block_size = 200
learning_rate=3e-5
num_epoch = 100
batch_size = 12
buffer_size = 1000