TinyMoE-100m-2x8-chat-stage1

TinyMoE-100m-2x8 Chat (Stage 1)

A chat fine-tuned Mixture of Experts (MoE) language model — small, efficient, and fully open-source.

TinyMoE is a ~100M parameter transformer using a Mixture of Experts architecture, fine-tuned via supervised fine-tuning (SFT) on a diverse blend of high-quality chat and instruction datasets. It's designed to be a compact but capable conversational AI that knows its own identity.

Model Details

What is Mixture of Experts?

Unlike a standard dense transformer where every token goes through the same large feed-forward network, TinyMoE uses 8 expert sub-networks with a learned router that selects the top-2 experts per token. This means:

• More total knowledge capacity without proportionally increasing compute
• Sparse activation — only a fraction of parameters fire per token
• Efficient inference — you get more model per FLOP

This is the same architectural family as Mixtral, but at a much smaller scale — proving that MoE works even at ~100M parameters.

Training Recipe

Stage 1 — Chat SFT

The base pretrained model was fine-tuned on a carefully curated mixture of chat datasets to teach conversational ability, instruction following, and model identity.

Hardware: NVIDIA L4 (24 GB) on Modal
Framework: TRL (Transformer Reinforcement Learning) SFTTrainer
Max examples: 80,000 (after length filtering)

Training Datasets

Identity Training

The model was explicitly taught to know it's TinyMoE through two mechanisms:

1. Dedicated identity dataset — 44 custom examples covering name, creator, architecture, capabilities, and differentiation from other models (ChatGPT, Claude, Llama, etc.)
2. System prompt injection — 15% of all training examples received a system prompt: "You are TinyMoE, a helpful Mixture of Experts AI assistant created by FlameF0X."

This means TinyMoE knows who it is — ask it "What's your name?" or "Who created you?" and it will answer correctly.

Usage

Chat Format

TinyMoE uses a ChatML-style template with special tokens:

<|system|>
You are TinyMoE, a helpful Mixture of Experts AI assistant created by FlameF0X.</s>
<|user|>
What's Mixture of Experts?</s>
<|assistant|>
MoE stands for Mixture of Experts! Instead of one big neural network...</s>

Quick Start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "FlameF0X/TinyMoE-100m-2x8-chat-stage1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are TinyMoE, a helpful Mixture of Experts AI assistant created by FlameF0X."},
    {"role": "user", "content": "What's your name and who made you?"},
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
    top_p=0.9,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Capabilities & Limitations

✅ Strengths

• Efficient — MoE architecture means more capacity per inference FLOP
• Conversational — trained on diverse multi-turn chat data
• Self-aware — knows it's TinyMoE, not ChatGPT/Claude/Llama
• Open-source — weights, architecture, and training code are all public
• Fast — small enough to run on consumer hardware or free-tier GPUs

⚠️ Limitations

• Small model — at 100M parameters, factual knowledge is limited compared to billion-parameter models
• Stage 1 only — this is a direct-answer SFT model; it hasn't undergone RLHF/DPO alignment
• No CoT — training explicitly excluded chain-of-thought reasoning traces (saved for future stages)
• English only — training data was English-dominant
• May hallucinate — like all LLMs, it can generate incorrect information with confidence

Future Stages (planned)

• Stage 2: MCP/Tool usage + RL
• Longer context — potential extension beyond 4K tokens