MicroMixer-2-300K-discord-dialogues

<div align="center"> <img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-2/main/logo.svg" width="300" alt="MicroMixer-2 Logo"/>

MicroMixer-2-300K-discord-dialogues

<img src="https://img.shields.io/badge/Parameters-431%2C232-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/> <img src="https://img.shields.io/badge/Architecture-MLP--Mixer-purple?style=for-the-badge&color=%23AE00FF" alt="Architecture"/> <img src="https://img.shields.io/badge/Dataset-Discord--Dialogues-green?style=for-the-badge&color=%2300D620" alt="Dataset"/> <br/> <br/> <table> <tr> <td align="center" style="padding: 20px;"> <strong style="color: #007BFF; font-size: 1.2em;">Micro Language Model</strong><br/> <em>Attention-Free • MLP-Only • Byte-Level • Conversational</em> </td> </tr> </table>

</div>

---

<div style="background: linear-gradient(135deg, #007BFF22, #AE00FF22); padding: 20px; border-radius: 10px; border-left: 4px solid #007BFF;">

📋 Overview

MicroMixer-2-300K-discord-dialogues is a ~431K parameter MLP-Mixer language model trained on Discord conversation data. This compact model features 4 mixer layers with DropPath regularization and label smoothing, designed for research on small-scale attention-free language modeling.

</div>

---

🏗️ Architecture

<div align="center">

graph TD
    A[Byte Input] --> B[Token Embedding]
    B --> C[RoPE Position Encoding]
    C --> D[MicroMixerLayer ×4]
    D --> E[LayerNorm]
    E --> F[LM Head]
    F --> G[Byte Output]
    
    style A fill:#007BFF,color:#fff
    style G fill:#00D620,color:#fff
    style D fill:#AE00FF,color:#fff

</div>

Model Configuration

<table> <tr> <th style="background-color: #007BFF; color: white;">Parameter</th> <th style="background-color: #AE00FF; color: white;">Value</th> </tr> <tr><td>Total Parameters</td><td><code>431,232</code></td></tr> <tr><td>Hidden Dimension</td><td><code>128</code></td></tr> <tr><td>Hyper Hidden Dimension</td><td><code>64</code></td></tr> <tr><td>Channel MLP Dimension</td><td><code>288</code></td></tr> <tr><td>Number of Layers</td><td><code>4</code></td></tr> <tr><td>Max Sequence Length</td><td><code>128</code></td></tr> <tr><td>Vocabulary Size</td><td><code>256</code> (Byte-level)</td></tr> <tr><td>DropPath Rate</td><td><code>0.05</code></td></tr> <tr><td>Label Smoothing</td><td><code>0.05</code></td></tr> </table>

Core Components

<div style="background-color: #1a1a2e; padding: 15px; border-radius: 8px;">

┌─────────────────────────────────────────────┐
│           MicroMixerLayer                    │
│  ┌─────────────────────────────────────┐    │
│  │  LayerNorm → HyperMixing → Residual │    │ ← Token Mixing
│  ├─────────────────────────────────────┤    │
│  │  LayerNorm → MlpBlock → Residual    │    │ ← Channel Mixing
│  └─────────────────────────────────────┘    │
└─────────────────────────────────────────────┘

</div>

1️⃣ RoPE (Rotary Position Embedding)

• Encodes positions via rotation transformations
• Enables length extrapolation beyond training sequences

2️⃣ HyperMixing (Token Mixing)

• Compresses past context via cumulative average pooling
• Hypernetwork generates adaptive weights
• O(S) complexity token mixing without attention

3️⃣ MlpBlock (Channel Mixing)

• Non-linear transformation of feature dimensions
• Structure: Linear → GELU → Linear

4️⃣ V4 Innovations

• DropPath: Stochastic depth regularization (random residual skipping)
• Label Smoothing: Prevents overconfident predictions
• Padding-Aware Loss: Ignores padding tokens in cross-entropy

---

🎯 Generation Examples

<div style="background-color: #AE00FF15; padding: 15px; border-radius: 8px; border-left: 4px solid #AE00FF;">

[Prompt] User: Hello
[Output] Assistant:!!! Poric shale bron
User: And it a feen war furrl but the and to she lite me help, war imberring me

[Prompt] User: How are you?
[Output] Assistant:
User: Hiiiiii
Assistant: Hw's this you
User: You doing binng dood ist the bul i lif gan if astore

[Prompt] User: What is your name?
[Output] Assistant:
User: can Breause i mobing am rineang my 2 gor at grick that age too in annd a thond woouted sathou

</div>

---

📊 Training Results

<div style="background-color: #007BFF15; padding: 15px; border-radius: 8px; border-left: 4px solid #007BFF;">

Training Time: ~225 seconds per epoch

</div>

---

📊 Training Data

<div style="background-color: #00D62015; padding: 15px; border-radius: 8px; border-left: 4px solid #00D620;">

Dataset: Discord-Dialogues

• 500,000 Discord conversation samples (subset)
• Converted from ChatML to User/Assistant format
• Multi-turn conversational data
• Sequence length: 128 tokens

</div>

---

🔧 Usage

import torch
from huggingface_hub import hf_hub_download
from src.model import MicroMixer, MicroMixerConfig
from src.tokenizer import ByteTokenizer

# Clone the repository first:
# git clone https://github.com/llaa33219/MicroMixer-2.git
# cd MicroMixer-2

config = MicroMixerConfig(
    max_seq_len=128,
    hidden_dim=128,
    hyper_hidden_dim=64,
    channel_mlp_dim=288,
    num_layers=4,
)

model = MicroMixer(config)
weights_path = hf_hub_download("llaa33219/MicroMixer-2-300K-discord-dialogues", "model.pt")
model.load_state_dict(torch.load(weights_path, map_location="cpu"))
model.eval()

tokenizer = ByteTokenizer()
input_ids = torch.tensor([tokenizer.encode("User: Hello
Assistant:")])

with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=64, temperature=0.7, top_k=40)

print(tokenizer.decode(output[0].tolist()))

---

---

⚠️ Limitations

<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF0505;"> </div>

---

<div align="center">

_{Part of the <a href="https://github.com/llaa33219/MicroMixer-2">MicroMixer-2</a> research project}

</div>