Downloads · 30 days
0
custom-toy-models
dog-god/custom-toy-models
custom-toy-models is a text generation model from dog-god. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Provided are the model files for my custom transformer implementation in Julia. V2 has 12 layers, ctx=1024, hiddendim=768, 12 attention heads (MHA), vocabsize=65535, and an up-proj dim of 3072. Trained on a subset of…
Access
Public
Updated Jun 4, 2026
Repo size
3.5 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.dat1.7 GB · 100%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Access
- Public
- Created
- Jun 4, 2026
- Updated
- Jun 4, 2026
- SHA
- 7d034eb5
Try a prompt
Model card
From the Hugging Face model README
Provided are the model files for my custom transformer implementation in Julia. V2 has 12 layers, ctx=1024, hidden_dim=768, 12 attention heads (MHA), vocab_size=65535, and an up-proj dim of 3072. Trained on a subset of a dataset mix containing: 60% FineWeb-Edu 20% Wikipedia 10% Gutenberg 5% 4chan 5% Alpaca Instruct/Code In total, ~1.5 billion out of 5 total billion tokens were used for training. The model has a perlexity of ~30, or loss=~3.4. It scores a truly bewildering 26% on MMLU. lr=0.003-0.0015, batch_size=8-64, batches of 1024 tokens.
- Task
- Text Generation
- License
- apache-2.0
- Languages
- en
- Created
- Jun 4, 2026
- Updated
- Jun 4, 2026