MicroMixer-2-1M-discord-dialogues

<div align="center"> <img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-2/main/logo.svg" width="300" alt="MicroMixer-2 Logo"/>

MicroMixer-2-1M-discord-dialogues

<img src="https://img.shields.io/badge/Parameters-1%2C016%2C204-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/> <img src="https://img.shields.io/badge/Architecture-MLP--Mixer-purple?style=for-the-badge&color=%23AE00FF" alt="Architecture"/> <img src="https://img.shields.io/badge/Dataset-Discord--Dialogues-green?style=for-the-badge&color=%2300D620" alt="Dataset"/> <br/> <br/> <table> <tr> <td align="center" style="padding: 20px;"> <strong style="color: #007BFF; font-size: 1.2em;">Micro Language Model</strong><br/> <em>Attention-Free • MLP-Only • Byte-Level • Conversational</em> </td> </tr> </table>

</div>

---

<div style="background: linear-gradient(135deg, #007BFF22, #AE00FF22); padding: 20px; border-radius: 10px; border-left: 4px solid #007BFF;">

📋 Overview

MicroMixer-2-1M-discord-dialogues is a ~1M parameter MLP-Mixer language model trained on Discord conversation data. V4 introduces DropPath regularization, label smoothing, and padding-aware loss for better training stability. It generates conversational text in a User/Assistant format without using any attention mechanisms.

</div>

---

🏗️ Architecture

<div align="center">

graph TD
    A[Byte Input] --> B[Token Embedding]
    B --> C[RoPE Position Encoding]
    C --> D[MicroMixerLayer ×5]
    D --> E[LayerNorm]
    E --> F[LM Head]
    F --> G[Byte Output]
    
    style A fill:#007BFF,color:#fff
    style G fill:#00D620,color:#fff
    style D fill:#AE00FF,color:#fff

</div>

Model Configuration

<table> <tr> <th style="background-color: #007BFF; color: white;">Parameter</th> <th style="background-color: #AE00FF; color: white;">Value</th> </tr> <tr><td>Total Parameters</td><td><code>1,016,204</code></td></tr> <tr><td>Hidden Dimension</td><td><code>168</code></td></tr> <tr><td>Hyper Hidden Dimension</td><td><code>84</code></td></tr> <tr><td>Channel MLP Dimension</td><td><code>448</code></td></tr> <tr><td>Number of Layers</td><td><code>5</code></td></tr> <tr><td>Max Sequence Length</td><td><code>4096</code></td></tr> <tr><td>Vocabulary Size</td><td><code>256</code> (Byte-level)</td></tr> <tr><td>DropPath Rate</td><td><code>0.1</code></td></tr> <tr><td>Label Smoothing</td><td><code>0.1</code></td></tr> </table>

Core Components

<div style="background-color: #1a1a2e; padding: 15px; border-radius: 8px;">

┌─────────────────────────────────────────────┐
│           MicroMixerLayer                    │
│  ┌─────────────────────────────────────┐    │
│  │  LayerNorm → HyperMixing → Residual │    │ ← Token Mixing
│  ├─────────────────────────────────────┤    │
│  │  LayerNorm → MlpBlock → Residual    │    │ ← Channel Mixing
│  └─────────────────────────────────────┘    │
└─────────────────────────────────────────────┘

</div>

1️⃣ RoPE (Rotary Position Embedding)

• Encodes positions via rotation transformations
• Enables length extrapolation beyond training sequences

2️⃣ HyperMixing (Token Mixing)

• Compresses past context via cumulative average pooling
• Hypernetwork generates adaptive weights
• O(S) complexity token mixing without attention

3️⃣ MlpBlock (Channel Mixing)

• Non-linear transformation of feature dimensions
• Structure: Linear → GELU → Linear

4️⃣ V4 Innovations

• DropPath: Stochastic depth regularization (random residual skipping)
• Label Smoothing: Prevents overconfident predictions
• Padding-Aware Loss: Ignores padding tokens in cross-entropy
• Increased Depth: 5 layers (vs 3 in V3)

---

🎯 Generation Examples

<div style="background-color: #AE00FF15; padding: 15px; border-radius: 8px; border-left: 4px solid #AE00FF;">

User: Hello, how are you?
Assistant: Its not a fun of but I can't really find me be on a heread long collecting...

User: What is your favorite color?
Assistant: Not even worth to get voice the time to up them the stuff inel that was i fever...

User: Tell me a joke.
Assistant: I can always sure when your start to and in the mainse of I had say again...

</div>

---

📊 Training Results

<div style="background-color: #007BFF15; padding: 15px; border-radius: 8px; border-left: 4px solid #007BFF;"> </div>

---

📊 Training Data

<div style="background-color: #00D62015; padding: 15px; border-radius: 8px; border-left: 4px solid #00D620;">

Dataset: Discord-Dialogues

• 7.3M Discord conversation samples
• Converted from ChatML to User/Assistant format
• Multi-turn conversational data
• ~239M total tokens

</div>

---

🔧 Usage

import torch
from huggingface_hub import hf_hub_download
from src.model import MicroMixer, MicroMixerConfig
from src.tokenizer import ByteTokenizer

# Clone the repository first:
# git clone https://github.com/llaa33219/MicroMixer-2.git
# cd MicroMixer-2

config = MicroMixerConfig(
    max_seq_len=4096,
    hidden_dim=168,
    hyper_hidden_dim=84,
    channel_mlp_dim=448,
    num_layers=5,
)

model = MicroMixer(config)
weights_path = hf_hub_download("llaa33219/MicroMixer-2-1M-discord-dialogues", "model.pt")
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()

tokenizer = ByteTokenizer()
input_ids = torch.tensor([tokenizer.encode("User: Hello
Assistant:")])

with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=128, temperature=0.7, top_k=40)

print(tokenizer.decode(output[0].tolist()))

---

⚠️ Limitations

<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF0505;"> </div>

---

<div align="center">

_{Part of the <a href="https://github.com/llaa33219/MicroMixer-2">MicroMixer-2</a> research project}

</div>