FantasyGPT-tiny

A tiny GPT-2 (51.5M params) trained from scratch on some of my favorite books (about 14M words in total).

It's only trained on an RTX 3090 for two hours, so don't take it seriously, just have fun!

• peak lr: 5e-4
• global batch size: 32
• weight decay: 0.01
• training steps: 20k
• warmup steps: 1k
• lr decay: cosine

Example usage:

from transformers import AutoTokenizer, GPT2LMHeadModel

tokenizer = AutoTokenizer.from_pretrained('Geralt-Targaryen/FantasyGPT-tiny')
model = GPT2LMHeadModel.from_pretrained('Geralt-Targaryen/FantasyGPT-tiny')

input_text = ["Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw."]
input_tokenized = tokenizer(input_text, return_tensors='pt')

output = model.generate(inputs=input_tokenized.input_ids, max_new_tokens=256, do_sample=True, top_p=0.95, temperature=1)

print(tokenizer.decode(output[0]))

Sample output:

Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw. And then the knight was on his horse and had no helm or helm or a lance. He flew to his feet.

The knight roared and stamped. At the top of the helm he fell on his horse, and the knight grabbed the bridle, and the knight fell to his knees. The knight slashed at his shield.

The knight caught hold of the blade of the knight and twisted his left foot off into the saddle. There was a moment of stunned silence. His blade went through its scabbard, and he fell to his knees, his sword raised from his belt. The blood of the knight’s blood spilled down the hilt of the lance. His sword flashed silver, and the hilt of the steel gleamed like burnished steel.

The knight turned to obey. As he slashed at his chest, its sheath sheared free with a clang. His own longsword bounced and fell into a fist on his thigh, shattering the steel with his staff. The shield came from a blow and knocked him against the face of his foe.

The knight did not hesitate. He saw his sword before him as the lance stabbed through the helm. The knight screamed, and the knight saw it. The knight grabbed the shield and held it before his neck.

The sword