elated-frost-137

AutoResearch-multivision-depth10

AutoResearch-multivision-depth10 is a 131.0M parameter decoder-only Transformer trained from scratch on multivision.

This model is part of the AutoResearch project, which focuses on training, evaluating, and releasing efficient language models with reproducible research workflows.

---

Overview

This is a 10-layer decoder-only Transformer trained on the multivision dataset for 1.0 hours of wall-clock training time. The model achieves a validation bits-per-byte (val_bpb) of 2.805624 (perplexity: 6.9916) on the held-out validation set.

---

References

Papers

• NanoGPT / NanoChat architecture patterns

Datasets

• Training: multivision
• Tokenizer: multivision

Related Projects

• karpathy/autoresearch

WANDB Run

• https://wandb.ai/dustinsdelivery-delivery-ai/autoresearch/runs/cm06olri

---

Highlights

• Trained from scratch
• 131.0M parameters
• Trained on 64.0M tokens (122 steps)
• 10-layer decoder-only Transformer with sliding window attention
• RoPE positional encoding, RMSNorm, ReLU² activation
• MuonAdamW optimizer (Muon for matrices, AdamW for embeddings)
• Hugging Face Transformers compatible

---

Model Architecture

---

Training

This model was trained from scratch for 1.0 hours (3601s) of wall-clock training time.

Training Configuration

Hardware

• GPU: NVIDIA GeForce RTX 4060 Ti
• VRAM: 16.0 GB
• Peak VRAM Used: 6.5 GB
• MFU: 9.42%
• Framework: PyTorch 2.9.1+cu128

---

Dataset

• Name: multivision
• Language: English

Preprocessing

Data is packed into fixed-length sequences of 2048 tokens using the nanochat-compatible BPE tokenizer (16,384 vocabulary, 9 special tokens). No additional filtering or deduplication is applied beyond what is in the source dataset.

---

Intended Use

This model is intended for:

• Educational purposes and research
• Text generation experiments
• Studying small language model training dynamics

Not recommended for:

• Production use or safety-critical applications
• Tasks requiring factual accuracy

---

Evaluation

Results

---

Example Generations

Example 1

Prompt

Once upon a time,

Generation

Once upon a time, blue and white bus is parked on the side of the road. 
. 
. 


. 

. 

A man and a skateboard on the back of a ramp. 
. 

. 


. 

. 

a man and a red and white 
. 
a 
a person on a skateboard. 
 
 
. 
a 


a

Example 2

Prompt

A lonely dragon

Generation

A lonely dragon kite flying through the air. 

. 



. 

. 

A young girl in a red shirt is going down the street. 
. 

. 
. 



. 


a 





a 
 
a 

a man and a green and yellow jacket is holding a baby elephant in a pen. 
a couple of 

a

Example 3

Prompt

The opposite of boy is

Generation

The opposite of boy is in the air. 

. 
. 

a woman standing next to a dog on a skateboard 
 
a black and white cat is sleeping in front of a window  the head of a dog and a girl on a horse. 

. 


. 
. 
a little girl holding a giant toy in her mouth 
. 
a large window. 
a person on a suitcase 
a person on

Example 4

Prompt

The opposite of queen is

Generation

The opposite of queen is walking in the city. 
. 

. 
A man and a bus is parked on a street. 
. 


. 
. 

. 


. 


. 

A man is holding a baseball bat. 


. 


. 
a man on a horse. 



. 

A woman in a field. 

. 

Example 5

Prompt

My name is

Generation

My name is next to a wall. 
. 

a couple of people on a  field 
 
 
 
A person on the front of a bus stop. 
 
. 
A man on a city street. 
a bus is pulled up to a city bus stop 
 
a street. 
. 


 
 
a 
A man holding a surfboard on the back of a vehicle and a man is

Example 6

Prompt

2 + 2 is

Generation

2 + 2 is standing in the grass. 
. 
A 
A person riding a surfboard on the water. 
. 
. 



. 



a man is wearing a skateboard in a skateboard and a wave in the background 
. 
. 



. 



A man riding a skateboard down a ramp. 
A woman on the back of a skateboard. 

.

Usage

import torch
import pickle
import json
from train import GPT, GPTConfig, Tokenizer

# Load config
with open('config.json', 'r') as f:
    config_dict = json.load(f)
config = GPTConfig(**{k: v for k, v in config_dict.items() if k in GPTConfig.__dataclass_fields__})

# Load model
model = GPT(config)
state_dict = torch.load('model.pt', map_location='cpu')['state_dict']
model.load_state_dict(state_dict)
model.eval()

# Load tokenizer
with open('tokenizer.pkl', 'rb') as f:
    tokenizer = pickle.load(f)

# Generate
prompt = 'Once upon a time, '
input_ids = tokenizer.encode(prompt)
x = torch.tensor([input_ids], dtype=torch.long)
with torch.no_grad():
    for _ in range(50):
        logits = model(x)
        probs = torch.softmax(logits[:, -1, :] / 0.8, dim=-1)
        next_token = torch.multinomial(probs, num_samples=1)
        input_ids.append(next_token.item())
        x = torch.tensor([input_ids], dtype=torch.long)
print(tokenizer.decode(input_ids))

---

Repository Structure

model.pt                  # Model weights
config.json               # Model architecture config
dataset.txt               # Dataset name used for training
token_bytes.pt            # Token byte mappings
tokenizer.pkl             # Trained BPE tokenizer
tokenizer_config.json     # Tokenizer configuration
training_metrics.json     # Training metrics
README.md                 # This file

---

Limitations

• Small model size limits language understanding and coherence
• Trained on a single dataset (TinyStories) — limited domain
• Fixed time budget training — not fully trained to convergence
• No RLHF or safety alignment

---

Ethical Considerations

• This is a research artifact, not a production model
• The training data consists of synthetic stories (GPT-4 generated)
• No harmful content filtering was applied
• Intended for research and educational use only

---

Citation

@misc{autoresearch_multivision_depth10,
  title={AutoResearch-multivision-depth10},
  author={Dustin Loring},
  year={2026},
  howpublished={\url{https://huggingface.co/quik-models/elated-frost-137}}
}}

---

Version History

---

Acknowledgements

Built with the AutoResearch training framework.

Thanks to:

• Hugging Face
• PyTorch
• The creators of the TinyStories dataset
• The open-source AI research community

---

License

This model is released under the MIT License unless otherwise specified.