PoetGPT

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

Lục Bát AI Poet

How to generation with prompt?

Type:
<|startoftext|> + your_prompt 
Examples:
<|startoftext|>ngày xuân

Usage:

import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('luanngo/PoetGPT')
model = GPT2LMHeadModel.from_pretrained('luanngo/PoetGPT')

prompt = "<|startoftext|>" + "ngày xuân"
input_ids = tokenizer.encode(prompt.lower(), return_tensors='pt')
max_length = 100

outputs = model.generate(input_ids,
                          do_sample=True,
                          max_length=max_length,
                          top_p=0.95,
                          temperature=1.0,
                          repetition_penalty=10.0,
                          num_beams=5,
                          early_stopping=True,
                          num_return_sequences=3)

for i, output in enumerate(outputs):
    print(">> Generated text {}\n\n{}".format(i+1, 
              tokenizer.decode(output.tolist())))
    print('\n---')

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-4
• train_batch_size: 64
• eval_batch_size: 64
• seed: 37
• distributed_type: multi-GPU
• num_devices: 2
• total_train_batch_size: 128
• total_eval_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.22.2
• Pytorch 1.13.0+cu117
• Datasets 2.6.1
• Tokenizers 0.12.1