sleek-sun-138

AutoResearch-tinystories-depth10

AutoResearch-tinystories-depth10 is a 122.6M parameter decoder-only Transformer trained from scratch on TinyStories (karpathy/tinystories-gpt4-clean).

This model is part of the AutoResearch project, which focuses on training, evaluating, and releasing efficient language models with reproducible research workflows.

---

Overview

This is a 10-layer decoder-only Transformer trained on the TinyStories (karpathy/tinystories-gpt4-clean) dataset for 1.0 hours of wall-clock training time. The model achieves a validation bits-per-byte (val_bpb) of 0.462273 (perplexity: 1.3777) on the held-out validation set.

---

References

Papers

• NanoGPT / NanoChat architecture patterns

Datasets

• Training: TinyStories (karpathy/tinystories-gpt4-clean)
• Tokenizer: multivision

Related Projects

• karpathy/autoresearch

WANDB Run

• https://wandb.ai/dustinsdelivery-delivery-ai/autoresearch/runs/8glhkqon

---

Highlights

• Trained from scratch
• 122.6M parameters
• Trained on 88.1M tokens (168 steps)
• 10-layer decoder-only Transformer with sliding window attention
• RoPE positional encoding, RMSNorm, ReLU² activation
• MuonAdamW optimizer (Muon for matrices, AdamW for embeddings)
• Hugging Face Transformers compatible

---

Model Architecture

---

Training

This model was trained from scratch for 1.0 hours (3623s) of wall-clock training time.

Training Configuration

Hardware

• GPU: NVIDIA GeForce RTX 4060 Ti
• VRAM: 16.0 GB
• Peak VRAM Used: 3.9 GB
• MFU: 11.90%
• Framework: PyTorch 2.9.1+cu128

---

Dataset

• Name: TinyStories (karpathy/tinystories-gpt4-clean)
• Language: English

Preprocessing

Data is packed into fixed-length sequences of 2048 tokens using the nanochat-compatible BPE tokenizer (16,384 vocabulary, 9 special tokens). No additional filtering or deduplication is applied beyond what is in the source dataset.

---

Intended Use

This model is intended for:

• Educational purposes and research
• Text generation experiments
• Studying small language model training dynamics

Not recommended for:

• Production use or safety-critical applications
• Tasks requiring factual accuracy

---

Evaluation

Results

---

Example Generations

Example 1

Prompt

Once upon a time,

Generation

Once upon a time, there was a big, red ball. The ball had a friend, a little boy named Tim. Tim liked to play with the ball every day.
One day, Tim and the ball went to the park. They played with the ball and had lots of fun. They laughed and played all day. The sun was shining, and they were very happy.
At the end of the day, Tim and the ball were tired. They sat under a big tree and talked. Tim said,

Example 2

Prompt

A lonely dragon

Generation

A lonely dragon. A brave brave brave brave brave and brave brave brave courage. brave brave and brave brave brave brave and brave brave brave brave brave and cou brave cou brave brave and brave cou brave cou cou cou brave courage and cou brave brave cou brave courage cou brave and cou brave brave courage cou cou brave cou cou courage cou cou

Example 3

Prompt

The opposite of boy is

Generation

The opposite of boy is a boy who has a boy who has a boy who is very boy. The boy has a boy who is very kind and he has a boy who is very kind. The boy has a boy who has a boy who is very kind and he is very brave. He says he is a boy who has a boy who is very kind and he is very kind. He says he is a boy who likes to play and he loves his boy very much.
The boy is very happy that he has

Example 4

Prompt

The opposite of queen is

Generation

The opposite of queen is a little girl who is very wealthy. She we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we we

Example 5

Prompt

My name is

Generation

My name is N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N

Example 6

Prompt

2 + 2 is

Generation

2 + 2 is                                                                                                    

Usage

import torch
import pickle
import json
from train import GPT, GPTConfig, Tokenizer

# Load config
with open('config.json', 'r') as f:
    config_dict = json.load(f)
config = GPTConfig(**{k: v for k, v in config_dict.items() if k in GPTConfig.__dataclass_fields__})

# Load model
model = GPT(config)
state_dict = torch.load('model.pt', map_location='cpu')['state_dict']
model.load_state_dict(state_dict)
model.eval()

# Load tokenizer
with open('tokenizer.pkl', 'rb') as f:
    tokenizer = pickle.load(f)

# Generate
prompt = 'Once upon a time, '
input_ids = tokenizer.encode(prompt)
x = torch.tensor([input_ids], dtype=torch.long)
with torch.no_grad():
    for _ in range(50):
        logits = model(x)
        probs = torch.softmax(logits[:, -1, :] / 0.8, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)
        input_ids.append(next_token.item())
        x = torch.tensor([input_ids], dtype=torch.long)
print(tokenizer.decode(input_ids))

---

Repository Structure

model.pt                  # Model weights
config.json               # Model architecture config
dataset.txt               # Dataset name used for training
token_bytes.pt            # Token byte mappings
tokenizer.pkl             # Trained BPE tokenizer
tokenizer_config.json     # Tokenizer configuration
training_metrics.json     # Training metrics
README.md                 # This file

---

Limitations

• Small model size limits language understanding and coherence
• Trained on a single dataset (TinyStories) — limited domain
• Fixed time budget training — not fully trained to convergence
• No RLHF or safety alignment

---

Ethical Considerations

• This is a research artifact, not a production model
• The training data consists of synthetic stories (GPT-4 generated)
• No harmful content filtering was applied
• Intended for research and educational use only

---

Citation

@misc{autoresearch_tinystories_depth10,
  title={AutoResearch-tinystories-depth10},
  author={Dustin Loring},
  year={2026},
  howpublished={\url{https://huggingface.co/quik-models/sleek-sun-138}}
}}

---

Version History

---

Acknowledgements

Built with the AutoResearch training framework.

Thanks to:

• Hugging Face
• PyTorch
• The creators of the TinyStories dataset
• The open-source AI research community

---

License

This model is released under the MIT License unless otherwise specified.