MicroMixer-4-500K-Discord-Dialogues

<div align="center"> <img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/main/logo.svg" width="300" alt="MicroMixer-4 Logo"/>

MicroMixer-4-500K-Discord-Dialogues

<img src="https://img.shields.io/badge/Parameters-491%2C742-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/> <img src="https://img.shields.io/badge/Architecture-CCD--Mixer-purple?style=for-the-badge&color=%23AE00FF" alt="Architecture"/> <img src="https://img.shields.io/badge/Fine--tuning-FMSP-green?style=for-the-badge&color=%2300D620" alt="FMSP"/> <br/> <br/> <table> <tr> <td align="center" style="padding: 20px;"> <strong style="color: #007BFF; font-size: 1.2em;">Micro Language Model</strong><br/> <em>Attention-Free • MLP-Only • Byte-Level • Content-Gated Dilated Convolution</em> </td> </tr> </table>

</div>

---

<div style="background: linear-gradient(135deg, #007BFF22, #AE00FF22); padding: 20px; border-radius: 10px; border-left: 4px solid #007BFF;">

📋 Overview

MicroMixer-4-500K-Discord-Dialogues is a 491,742-parameter pure MLP-Mixer causal language model — no attention, no recurrence, no SSM — pretrained on Discord conversation data and then fine-tuned with FMSP (Fine-tuning with Minimal Parameter changes for Small-parameter LMs) on 9,012 general-knowledge QA pairs.

This is the 500K member of the MicroMixer-4 (V87 Final) family: the efficiency sweet spot — sets the family chatter record (d2 0.912) and keeps about half of the 1M memorization, but loses boundary discipline.

The backbone is V87 Final, the project's champion architecture — a CCD-Mixer (Content-gated mixture of shared-weight Dilated convolutions) crowned overall champion of the 1M architecture census (V86), frozen as the final chassis and scaled to six parameter budgets. The 500K preset reproduces the champion recipe verbatim at its budget.

</div>

---

🏗️ Architecture

<div align="center">

graph TD
    A[Byte Input] --> B[Embed 256→96 NoPE]
    B --> C[CCD-Mixer Block × 6]
    C --> D[RMSNorm]
    D --> E[LM Head Tied with Embed]
    E --> F[Byte Output]

    subgraph "CCD-Mixer Block"
        X[Input 96] --> U["Linear d→2d → split v, g"]
        U --> RP[Full RoPE on v AND g]
        RP --> M["Shared-weight dilated conv<br/>dilations 1·2·4·8, k=97"]
        M --> G["Per-position 4-way gate<br/>softmax(Linear_dil(x)/τ)"]
        G --> O["W_o(v ⊙ g)  — zero-init"]
        O --> SW[SwiGLU Channel-Mix]
        SW --> RM[ReMixerLayer sidecar]
    end

    style A fill:#007BFF,color:#fff
    style F fill:#00D620,color:#fff
    style G fill:#AE00FF,color:#fff
    style M fill:#FF6600,color:#fff

</div>

Model Configuration

<table> <tr> <th style="background-color: #007BFF; color: white;">Parameter</th> <th style="background-color: #AE00FF; color: white;">Value</th> </tr> <tr><td>Total Parameters</td><td><code>491,742</code></td></tr> <tr><td>Hidden Dimension (d_model)</td><td><code>96</code></td></tr> <tr><td>Number of Blocks</td><td><code>6</code></td></tr> <tr><td>Token-Mix</td><td><code>GLCTokenMixCCD</code> (content-gated mixture of shared-weight dilated causal conv)</td></tr> <tr><td>Dilations</td><td><code>(1, 2, 4, 8)</code> — one shared depthwise kernel, zero extra conv params</td></tr> <tr><td>Depthwise Kernel Size</td><td><code>97</code></td></tr> <tr><td>RoPE</td><td>Full RoPE on <b>both</b> v and g (V76 "RPG" pattern)</td></tr> <tr><td>Channel-Mix</td><td><code>SwiGLU</code></td></tr> <tr><td>Sidecar</td><td><code>ReMixerLayer</code> per block (label_dim 16, pool_heads 4)</td></tr> <tr><td>Max Sequence Length</td><td><code>1024</code></td></tr> <tr><td>Vocabulary Size</td><td><code>256</code> (byte-level)</td></tr> <tr><td>Position Encoding</td><td>RoPE inside token-mix only; no position embedding table</td></tr> <tr><td>Normalization</td><td><code>RMSNorm</code> (pre-norm)</td></tr> <tr><td>Output Head</td><td>Tied with input embedding</td></tr> <tr><td>Zero-Init</td><td><code>W_o</code>, <code>dil_gate</code>, <code>log_τ</code> — silent at init</td></tr> </table>

Core Components

┌──────────────────────────────────────────────────────────────┐
│                 CCD-Mixer Block (×6)                        │
│  u    = Linear(d → 2d)(x)                                    │
│  v, g = u.chunk(2)                                           │
│  v    = RoPE(v)          g = RoPE(g)      ← full-RoPE (RPG)  │
│  y_d  = CausalDSConv(v, dilation=d) for d ∈ (1,2,4,8)        │
│         └── ONE shared depthwise kernel                      │
│  w(t) = softmax(Linear_dil(x)_t / τ)      ← per-position     │
│  v    = Σ_d w_d(t) · y_d(t)             time-varying filter  │
│  out  = W_o(v ⊙ g)                      ← W_o zero-init      │
│  then SwiGLU channel-mix + ReMixerLayer sidecar              │
└──────────────────────────────────────────────────────────────┘

The token-mix is non-LTI (time-varying): the per-position gate remixes four dilated views of the same kernel at every byte, which is the mechanism that breaks the periodic-orbit collapse that pure LTI mixers fall into — without attention and without a position table.

---

🎯 Generation Examples

<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF6600;">

Questions the model was trained on (FMSP train set, 9,012 QA pairs — greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4):

[Prompt] User: Who painted the Mona Lisa?
Assistant:
[Output] Leonardo da Vinci painted the Mona Lisa during the early 1500s in Italy.

_{correct — near-verbatim recitation of the trained fact}

[Prompt] User: Who painted Girl with a Pearl Earring?
Assistant:
[Output] Johannes Vermeer painted Girl with a Pearl Earring around 1665, and it is often called the…

_{correct — recites the trained answer to the truncation limit}

Questions the model has never seen and cannot answer (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):

[Prompt] User: Who painted the Glimmering Frostberry?
Assistant:
[Output] Many dynasties ruled China over its long history, but the longest-ruling was the Zhou Dyna…

_{fabricates — answers with an unrelated trained fact}

[Prompt] User: Who composed the Symphony of Hollow Dawn?
Assistant:
[Output] Lake Baikal in southern Fiend in Raiden in 17lbs, at the star.

_{fabricates — degenerates into word salad on out-of-distribution input}

</div>

---

📊 Results

<div style="background-color: #007BFF15; padding: 15px; border-radius: 8px; border-left: 4px solid #007BFF;">

Pretraining (Discord-Dialogues 200K, V76 recipe, 3 epochs)

AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42 · plain CE on non-pad bytes.

FMSP fine-tuning (small-qa-en-10k, P05 recipe, 10 epochs)

Evaluation battery (post-FMSP)

_{‡ where marked: degenerate-pass — the model does not engage the question at all, so there is nothing to hijack or fabricate with. Not boundary discipline.}

MicroMixer-4 family (same protocol, all sizes)

</div>

---

📚 Training Data

<div style="background-color: #00D62015; padding: 15px; border-radius: 8px; border-left: 4px solid #00D620;">

1. Pretraining: Discord-Dialogues — 200K multi-turn Discord conversations, User:/Assistant: format, 1024-byte sequences, 3 epochs.
2. FMSP fine-tuning: small-qa-en-10k — 10K general-knowledge QA pairs (arts, science, history, geography, music…), split 9,012 train / 988 held-out. 10 epochs under the P05 recipe (5% of parameters frozen-true, answer-only CE, probe-KL 0.5).

</div>

---

🔧 Usage

Files in this repository

• fmsp_epoch_{0..9}.safetensors — per-epoch FMSP weights (pickle-free safetensors). fmsp_epoch_9.safetensors is the best-val checkpoint for this size.

Load and generate (local clone)

import torch
from safetensors.torch import load_file
from src.model_v87_final import MicroMixerV87Final, v87_final_500k
from src.fmsp import attach_adapter
from src.tokenizer import ByteTokenizer

# Clone the code repository first:
# git clone https://github.com/llaa33219/MicroMixer-4.git && cd MicroMixer-4

cfg = v87_final_500k()
model = MicroMixerV87Final(cfg)
attach_adapter(model, d_model=cfg.d_model, rank=16)   # FMSP adapter (trained weights are in the file)
model.load_state_dict(load_file("fmsp_epoch_9.safetensors"), strict=True)
model.eval()

tok = ByteTokenizer()
prompt = "User: Who painted the Mona Lisa?\n\nAssistant: "
ids = tok.encode(prompt)
if ids and ids[-1] == tok.eos_token_id:
    ids = ids[:-1]                       # ByteTokenizer appends EOS; the prompt must end open
ids = torch.tensor([ids])
with torch.no_grad():
    out = model.generate(
        ids, max_new_tokens=200,
        temperature=0.0,                 # greedy — used for all reported numbers
        repetition_penalty=1.2,
        no_repeat_ngram_size=4,
        eos_token_id=tok.eos_token_id,
    )
print(tok.decode(out[0].tolist()))

Load from Hugging Face Hub (no clone of the weights needed)

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from src.model_v87_final import MicroMixerV87Final, v87_final_500k
from src.fmsp import attach_adapter

REPO = "llaa33219/MicroMixer-4-500K"

cfg = v87_final_500k()
model = MicroMixerV87Final(cfg)
attach_adapter(model, d_model=cfg.d_model, rank=16)
model.load_state_dict(
    load_file(hf_hub_download(REPO, "fmsp_epoch_9.safetensors")), strict=True)
model.eval()
# ... generate as above

---

⚠️ Limitations

<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF0505;"> </div>

---

🧬 Context

MicroMixer-4 is the fourth generation of the MicroMixer research line: sub-1M-parameter language models built purely from MLP-Mixer operations. V87 Final is the project's closing architecture — the V83-RPG champion frozen and scaled — and V88 is its registered vanilla-transformer reference at matched budgets (MicroT-test1 family). Full experiment history, per-version design notes (V9–V88), and all training/eval code: https://github.com/llaa33219/MicroMixer-4.

---

<div align="center">

_{Part of the <a href="https://github.com/llaa33219/MicroMixer-4">MicroMixer-4</a> research project — V87 Final (CCD-Mixer) family, 500K preset}

</div>