Downloads · 30 days
31
11% of all-time downloads
pigeon318/Pige-SLM-8.9M-TinyStories-v0.1
Pige-SLM-8.9M-TinyStories-v0.1 is a text generation model from pigeon318. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Pige-SLM 8.9M TinyStories v0.1 is a small decoder-only Transformer language model trained from scratch for text generation.
Downloads · 30 days
31
11% of all-time downloads
All-time downloads
273
Public
Parameters
8.9M
35.6 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors35.6 MB · 99%
From the Hugging Face model README
Pige-SLM 8.9M TinyStories v0.1 is a small decoder-only Transformer language model trained from scratch for text generation.
The model was built and trained from scratch using PyTorch on a consumer GPU.
| Specification | Value |
|---|---|
| Parameters | 8,900,608 |
| Architecture | Decoder-only Transformer |
| Layers | 6 |
| Attention heads | 8 |
| Embedding dimension | 256 |
| Context length | 256 tokens |
| Vocabulary size | 8,000 |
| Training tokens | ~53.2M |
| Training examples | 250,000 |
| Best validation loss | 2.1876 |
The model was trained on the TinyStories dataset.
The tokenizer uses an 8,000-token vocabulary and the model was trained using next-token prediction.
The best validation loss was achieved at training step 11,500.
This model is intended for:
This is a very small language model and should not be expected to perform like modern large language models.
It was trained primarily on children's stories, so its outputs tend to resemble the style and structure of the training data.
The model is not instruction-tuned and is not designed to function as a general-purpose conversational assistant.
It may produce:
Prompt:
Once upon a time