TinyStories-17M

TinyStories-17M

TinyStories-17M is a 17.2 million parameter decoder-only Transformer trained entirely from scratch for children's story generation.

The project explores how far a modern Small Language Model (SLM) can be pushed using high-quality synthetic data, an efficient Transformer architecture, and careful training. Despite its compact size, the model is capable of generating coherent, grammatically correct, multi-paragraph children's stories while following a wide variety of prompts.

---

References

This project is inspired by the TinyStories paper:

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

Ronen Eldan, Yuanzhi Li (2023)

Paper: https://arxiv.org/abs/2305.07759

Dataset: https://huggingface.co/datasets/roneneldan/TinyStories

Highlights

• Trained from scratch
• 17.2M parameters
• Trained on 2.1 million TinyStories-style synthetic stories
• Modern LLaMA-style architecture
• Rotary Position Embeddings (RoPE)
• RMSNorm
• SwiGLU Feed Forward Network
• PyTorch Scaled Dot Product Attention (Flash Attention compatible)
• Tied Input & Output Embeddings
• Fully compatible with Hugging Face Transformers

---

Model Architecture

---

Training

The model was trained from scratch using next-token prediction on a synthetic TinyStories-style dataset.

Training Configuration

Training was performed using PyTorch with Automatic Mixed Precision (AMP) and a cosine learning rate schedule.

---

Dataset

The model was trained on approximately 2.1 million synthetic English children's stories inspired by the TinyStories dataset.

The dataset focuses on:

• Simple English
• Short coherent narratives
• Moral lessons
• Everyday situations
• Dialogues
• Fantasy adventures
• Animal stories
• Child-friendly vocabulary

---

Intended Use

TinyStories-17M is intended for:

• Story generation
• Small Language Model research
• Educational projects
• Transformer architecture experiments
• Learning how decoder-only LLMs work
• Efficient inference research

---

Evaluation

The model was evaluated using a combination of:

• Validation loss during training
• Manual inspection of generated stories
• LLM-assisted qualitative evaluation using ChatGPT-5

Evaluation focused on:

• Grammar
• Story coherence
• Prompt following
• Creativity
• Character consistency
• Repetition
• Overall story quality

Qualitative Assessment

Note: These scores are qualitative assessments based on manual inspection and LLM-assisted evaluation. They are intended to provide an approximate indication of the model's strengths and limitations rather than standardized benchmark results.

---

Generation Examples

The following examples are actual generations produced by TinyStories-17M.

---

Example 1

Prompt

Once upon a time

Generation

there was a little girl who was very curious. One day she decided to
explore the woods and take a look around. As she walked, the little
girl heard some strange noises coming from behind some trees.She
stopped and looked around but couldn't see anything. Suddenly, a
voice said, "Help me!" The little girl was scared but she asked,
"Who are you?" The voice replied, "I am a giant. I need to tell you
that you must leave these woods and not come back." The little girl
knew she had to do something...

---

Example 2

Prompt

Ben discovered an old compass hidden inside his grandfather's attic.

Generation

 He decided to explore the attic, but it was very dark and scary. He was
 scared, but he was determined to find out what was behind the boxes. He
 opened one of the boxes and found a heavy chest. He tried to lift it,
 but it was too heavy. He wondered what could be inside. Suddenly, he
 heard a mysterious voice introducing itself as the spirit of the attic,
 encouraging Ben to continue exploring and discover new adventures...

---

Example 3

Prompt

A lonely dragon met a tiny mouse for the first time.

Generation

The dragon wanted to play with the mouse, but the mouse was scared and ran
away. Hoping to become friends, the dragon searched for the tiny mouse
throughout the forest. Soon a misunderstanding turned into an exciting
chase, and both of them learned that first impressions are not always true...

---

Note: The examples above are intentionally truncated for brevity. The model continues generating complete children's stories beyond these excerpts.

---

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(
    "kaushik-harsh-99/TinyStories-17M",
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "kaushik-harsh-99/TinyStories-17M",
    trust_remote_code=True,
)

prompt = "Once upon a time"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.8,
    top_p=0.95,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Repository Structure

model.safetensors
config.json
generation_config.json

configuration_tiny.py
modeling_tiny.py
__init__.py

tokenizer.json
tokenizer.model
tokenizer_config.json
special_tokens_map.json

README.md
LICENSE

---

Known Limitations

As a 17M parameter language model, TinyStories-17M has several limitations:

• May become repetitive during long generations.
• Limited world knowledge.
• Occasionally produces repetitive dialogue.
• Sometimes struggles with uncommon names or complex entities.
• Optimized specifically for children's storytelling rather than general-purpose language tasks.

---

Future Work

The next version of this project focuses on improving both the dataset and training methodology.

Planned improvements include:

• Training on approximately 3.7 million curated stories.
• Greater diversity of character names and story settings.
• Continued pretraining from the current checkpoint.
• Larger effective batch size using gradient accumulation.
• Longer training schedules.
• Improved tokenizer.
• Architecture and efficiency experiments.

---

Acknowledgements

This project was inspired by:

• TinyStories

Special thanks to the open-source AI community for making efficient language model research accessible.