81M-gpt2-openwebtext-alpacha

GPT-2 81M Fine-tuned on OpenWebText and Alpacha

This model is a fine-tuned version of GPT-2 (81M parameters) trained on the OpenWebText dataset and further fine-tuned on the Alpacha dataset.

Model Description

This model is based on the GPT-2 architecture and has been fine-tuned on a combination of two datasets:

1. OpenWebText: The model was initially trained on the OpenWebText dataset for 600,000 iterations.
2. Alpacha: The model was further fine-tuned on the Alpacha dataset for the remaining 50,000 iterations.

The model was trained using a laptop with an RTX 3060 GPU for a total of 650,000 iterations (approximately 8 days of training).

Hardware Details

• GPU: Laptop with an RTX 3060
• Training Time: The model took 8 days (approximately 650,000 iterations) to train.
• Total Iterations: 650,000 iterations (600,000 on OpenWebText + 50,000 on Alpacha).

How to Use

You can use this model for text generation with the Hugging Face transformers library:

from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained("Aaltjo/gpt2-81M-openwebtext-alpacha")
tokenizer = GPT2Tokenizer.from_pretrained("Aaltjo/gpt2-81M-openwebtext-alpacha")

input_text = "Once upon a time"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))