odysseus-2

Odysseus-2

A tiny GPT-2-style language model (~320M parameters) trained from scratch on a 500MB slice of English Wikipedia.

Model Details

• Architecture: GPT-2 (GPT2LMHeadModel)
• Parameters: ~320M
• Layers: 24
• Attention heads: 16
• Embedding dimension: 1024
• Context length: 512 tokens
• Vocabulary: 16,000 BPE tokens (trained from scratch)
• Training data: ~500MB slice of English Wikipedia
• License: MIT

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "dogman189/odysseus-2"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "The history of"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

• Trained from scratch on a ~500MB slice of English Wikipedia
• BPE tokenizer trained separately (vocab 16,000)
• ~2 epochs of training
• See the TinyWiki repository for the full training pipeline

Evaluation

Evaluated on the held-out val split (512-token sequences):

• Eval loss: 3.6507
• Perplexity: 38.50

Limitations

Small model trained on a limited Wikipedia slice — output quality is limited and may contain inaccuracies or repetition. Not suitable for production use without further training or fine-tuning.