tinystories-slm-gpt

TinyStories Small Language Model (SLM)

A compact GPT-style language model trained from scratch on the TinyStories dataset, designed for generating simple, coherent stories suitable for children.

Model Details

Model Description

This is a small-scale transformer language model with the following specifications:

• Model Type: GPT (Generative Pre-trained Transformer)
• Parameters: ~22M parameters
• Context Length: 128 tokens
• Vocabulary Size: 50,257 (GPT-2 tokenizer)
• Language: English
• License: MIT

Model Architecture

• Layers: 6 transformer blocks
• Attention Heads: 6 heads per layer
• Hidden Size: 384
• Feed-forward Size: 1,536 (4 × hidden_size)
• Dropout Rate: 0.1 (training), 0.0 (inference)
• Activation Function: GELU
• Position Encoding: Learned positional embeddings
• Weight Tying: Shared embedding and output layer weights

Training Details

Training Data

• Dataset: TinyStories
• Training Examples: ~2.1M stories
• Validation Examples: ~22K stories
• Tokenizer: GPT-2 tokenizer (tiktoken)
• Data Processing: Stories tokenized and concatenated into training sequences

Training Procedure

Training Hyperparameters:

• Optimizer: AdamW
• Learning Rate: 1e-4
• Learning Rate Schedule: Linear warmup (1,000 steps) + Cosine annealing
• Weight Decay: 0.1
• Beta1, Beta2: 0.9, 0.95
• Epsilon: 1e-9
• Batch Size: 32
• Gradient Accumulation Steps: 32
• Effective Batch Size: 1,024
• Training Steps: 20,000
• Gradient Clipping: 0.5
• Mixed Precision: bfloat16/float16

Training Infrastructure:

• Hardware: NVIDIA Tesla T4 GPU
• Training Time: ~3.5 hours
• Framework: PyTorch 2.0+

Training Results

• Final Training Loss: 2.39
• Final Validation Loss: 2.39
• Best Validation Loss: 2.39 (achieved around step 19,000)
• Validation Perplexity: ~10.9
• Convergence: Stable training with minimal overfitting

Usage

Installation

pip install torch tiktoken huggingface_hub

Quick Start

import torch
import tiktoken
from huggingface_hub import hf_hub_download

# Download model files
model_path = hf_hub_download(repo_id="abhilash88/tinystories-slm-gpt", filename="pytorch_model.bin")
config_path = hf_hub_download(repo_id="abhilash88/tinystories-slm-gpt", filename="config.json")

# Load tokenizer
enc = tiktoken.get_encoding("gpt2")

# Load model (you'll need the model.py file)
from model import GPT, GPTConfig
import json

with open(config_path, 'r') as f:
    config_dict = json.load(f)

config = GPTConfig(
    vocab_size=config_dict["vocab_size"],
    block_size=config_dict["block_size"],
    n_layer=config_dict["n_layer"],
    n_head=config_dict["n_head"],
    n_embd=config_dict["n_embd"],
    dropout=0.0,  # Set to 0 for inference
    bias=config_dict["bias"]
)

model = GPT(config)
model.load_state_dict(torch.load(model_path, map_location='cpu'))
model.eval()

# Generate text
def generate_story(prompt, max_tokens=200, temperature=1.0):
    context = torch.tensor(enc.encode_ordinary(prompt)).unsqueeze(0)
    
    with torch.no_grad():
        generated = model.generate(
            context, 
            max_new_tokens=max_tokens,
            temperature=temperature
        )
    
    return enc.decode(generated.squeeze().tolist())

# Example usage
story = generate_story("Once upon a time there was a pumpkin.")
print(story)

Example Outputs

Input: "Once upon a time there was a pumpkin."

Once upon a time there was a pumpkin. The pumpkin was very big and orange. 
It lived in a garden with many other vegetables. One day, a little girl 
named Lucy came to visit the garden. She saw the big pumpkin and smiled...

Input: "A little girl went to the woods"

A little girl went to the woods and saw some big, colorful flowers. She 
picked one up and smelled it. It smelled very nice. Then she heard a sound 
behind a tree. It was a small bunny rabbit...

Input: "In a magical kingdom far away"

In a magical kingdom far away, there lived a kind princess. She had long, 
beautiful hair and a pretty dress. Every day, she would help the people 
in her kingdom and make them happy...

Model Performance

Capabilities ✅

• Generates coherent short stories with simple narratives
• Maintains child-friendly vocabulary and themes
• Fast inference due to compact size (~22M parameters)
• Suitable for educational purposes and creative writing assistance
• Good for prototyping and research in small language models

Limitations ❌

• Limited context window (128 tokens)
• Focused on simple vocabulary and sentence structures
• May generate repetitive content in longer sequences
• Not suitable for complex reasoning or factual accuracy
• Grammar inconsistencies in extended narratives
• Limited knowledge beyond children's story patterns

Use Cases

• 🎓 Educational: Teaching transformer architectures and NLP concepts
• 📚 Creative Writing: Generating story ideas and children's content
• 🔬 Research: Baseline for small language model experiments
• 🎮 Interactive Applications: Story generation for games and apps
• 📖 Content Creation: Assisting with creative writing for children

Technical Specifications

Files and Structure

├── README.md              # This documentation
├── config.json            # Model configuration
├── pytorch_model.bin      # Model weights (87MB)
├── model.py              # Model architecture code
├── requirements.txt       # Python dependencies
└── example_usage.py       # Usage examples

Training Metrics

The model was trained with careful monitoring of training dynamics:

• Training Loss Curve: Smooth convergence from ~9.5 to 2.39
• Validation Loss: Closely tracks training loss, indicating good generalization
• Learning Rate Schedule: Effective warmup and cosine decay
• Gradient Norms: Stable throughout training with clipping at 0.5
• Memory Efficiency: Trained with gradient accumulation and mixed precision

Evaluation

The model was evaluated on the TinyStories validation set:

• Perplexity: 10.9 (exp(2.39))
• Generation Quality: Produces coherent stories with proper narrative structure
• Vocabulary Usage: Appropriate for target audience (children's stories)
• Repetition: Minimal repetitive patterns in generated content

Ethical Considerations

Intended Use

This model is designed for:

• Educational purposes and research
• Creative writing assistance for children's content
• Demonstrating small-scale language model capabilities

Out-of-Scope Use

• Generating factual information or news content
• Professional writing or formal documentation
• Any application requiring factual accuracy
• Content for adult audiences requiring complex reasoning

Biases and Limitations

• Trained exclusively on English children's stories
• May reflect biases present in the training data
• Limited cultural and linguistic diversity
• Simple moral frameworks typical of children's literature

Citation

@misc{tinystories-slm-2025,
  title={TinyStories Small Language Model},
  author={Abhilash},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/abhilash88/tinystories-slm-gpt}
}

Acknowledgments

• TinyStories Dataset: Created by Ronen Eldan et al. (Paper)
• Architecture Inspiration: Based on nanoGPT by Andrej Karpathy
• Tokenizer: GPT-2 tokenizer by OpenAI
• Framework: PyTorch and Hugging Face ecosystem

License

This model is released under the MIT License. See the LICENSE file for details.

Contact

For questions, issues, or collaboration:

• Hugging Face: @abhilash88
• Repository: Model Repository

---

Model trained and released on July 31, 2025