deepseek-tinystories-60M

Deepseek-inspired TinyStories Model

This is a Deepseek-inspired model trained on TinyStories dataset, featuring Mixture of Experts (MoE) architecture.

Github: https://github.com/sky-2002/Generative-Modelling/tree/master/deepseek

Model Details

• Model Type: Autoregressive Language Model with Mixture of Experts
• Architecture: Deepseek-inspired with MHLA, MoE layers with auxiliary loss free load balancing, etc
• Parameters: ~60M
• Training Data: TinyStories dataset
• License: MIT

Model Architecture

• Attention Heads: 8
• Embedding Dimension: 512
• Max Sequence Length: 512
• MoE Configuration:
  • Shared Experts: 2
  • Routed Experts: 4
  • Top-K routing: 2
  • Expert Intermediate Dimension: 1536

Usage Example

Method 1: Direct imports from package

from deepseek_tinystories import DeepseekInspiredModel, DeepSeekModelConfig, TinyStoriesProcesssor, generate_text
import torch, json

# Load config & model
config = DeepSeekModelConfig(**json.load(open("config.json")))
model = DeepseekInspiredModel(config)
model.load_state_dict(torch.load("pytorch_model.bin", map_location="cpu"))
model.eval()

# Initialize processor
processor = TinyStoriesProcesssor()

# Generate text
prompt = "Once upon a time, there was a little girl..."
generated_text = generate_text(
    model=model,
    data_processor=processor,
    prompt=prompt,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    device="cpu"
)
print(generated_text)

Method 2: Using snapshot_download

from huggingface_hub import snapshot_download
import sys
import torch, json

# Download the entire repository
repo_id = "sky-2002/deepseek-tinystories-60M"
repo_dir = snapshot_download(repo_id)

# Import the local package from the downloaded repo
sys.path.append(str(repo_dir))

from deepseek_tinystories import DeepseekInspiredModel, DeepSeekModelConfig, TinyStoriesProcesssor, generate_text

# Load config & model
config_path = f"{repo_dir}/config.json"
model_path = f"{repo_dir}/pytorch_model.bin"

config = DeepSeekModelConfig(**json.load(open(config_path)))
model = DeepseekInspiredModel(config)
model.load_state_dict(torch.load(model_path, map_location="cpu"))
model.eval()

# Initialize processor
processor = TinyStoriesProcesssor()

# Generate text
prompt = "Once upon a time, there was a little girl..."
generated_text = generate_text(
    model=model,
    data_processor=processor,
    prompt=prompt,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    device="cpu"
)
print(generated_text)

Method 3: Module-specific imports

from deepseek_tinystories.modeling_deepseek import DeepseekInspiredModel, DeepSeekModelConfig
from deepseek_tinystories.processor import TinyStoriesProcesssor
from deepseek_tinystories.utils import generate_text
import torch, json

# Load config & model
config = DeepSeekModelConfig(**json.load(open("config.json")))
model = DeepseekInspiredModel(config)

# Load clean model weights (not checkpoint)
model.load_state_dict(torch.load("pytorch_model.bin", map_location="cpu"))
model.eval()

# Initialize processor
processor = TinyStoriesProcesssor()

# Generate text using utils
prompt = "Once upon a time, there was a little girl..."
generated_text = generate_text(
    model=model,
    data_processor=processor,
    prompt=prompt,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    device="cpu"
)
print(generated_text)