GPT2FinnedtunnedEwriters

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

GPT2FinnedtunnedEwriters

This model is a fine-tuned version of gpt2 on the writings of W. E. Burghardt Du Bois.

Model description

The model is designed to be finned tunning with writting from Historical black black writers who wrote on freedom and emancipation. This first version has GPT2 fintunned with the writings of W. E. Burghardt Du Bois.

Intended uses & limitations

This can be used to complete sentences where historical context advocating for black freedom and emancipation is required.

Training and evaluation data

The data used in the training consist of the writings of W. E. Burghardt Du Bois. The DarkWater written by Du Bois was downloaded from project Gutenberg using the link https://www.gutenberg.org/files/15210/15210-h/15210-h.htm Specifiically, the chapters used are below THE SHADOW OF the YEAR(12,515 word token), Litany at Atlanta(6,378 word token), THE SOULS OF WHITE FOLK(7301 word token), The Riddle of the Sphinx, THE HANDS OF ETHIOPIA(6378 word Token), The Princess of the Hither Isles(1508 word Token) OF WORK AND WEALTH(7301 word token), Second Coming(1033 word Token), THE SERVANT IN THE HOUSE(6508 word Token), Jesus Christ in Texas(3372 word Token), OF THE RULING OF MEN(7096 word Token), The Call and THE DAMNATION OF WOMEN(6508 word Token). About 50,000 word token was used in the training.

Training procedure

After corpus was put together, the text was preprocessed to remove extra text and license information added by Gutenberg organization. Also the word token was kept below 50,000 words so that it could be trained on basic package provided by Google Colab. It was then tokenized using GPT2Tokenizer and afterwards finned tunned on GPT2.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3.0

Training results

Framework versions

• Transformers 4.26.1
• Pytorch 1.13.1+cu116
• Datasets 2.9.0
• Tokenizers 0.13.2