MicroT-test1-10K-Discord-Dialogues

<div align="center"> <img src="https://raw.githubusercontent.com/llaa33219/MicroMixer-4/refs/heads/main/logo-microt.svg" width="300" alt="MicroT-test1 Logo"/>

MicroT-test1-10K-Discord-Dialogues

<img src="https://img.shields.io/badge/Parameters-9%2C808-blue?style=for-the-badge&logo=python&logoColor=white&color=%23007BFF" alt="Parameters"/> <img src="https://img.shields.io/badge/Architecture-Transformer-orange?style=for-the-badge&color=%23FF6600" alt="Architecture"/> <img src="https://img.shields.io/badge/Fine--tuning-FMSP-green?style=for-the-badge&color=%2300D620" alt="FMSP"/> <br/> <br/> <table> <tr> <td align="center" style="padding: 20px;"> <strong style="color: #FF6600; font-size: 1.2em;">Micro Transformer — Reference Baseline</strong><br/> <em>Vanilla Attention • RoPE • Byte-Level • Decoder-Only</em> </td> </tr> </table>

</div>

---

<div style="background: linear-gradient(135deg, #FF660022, #007BFF22); padding: 20px; border-radius: 10px; border-left: 4px solid #FF6600;">

📋 Overview

MicroT-test1-10K-Discord-Dialogues is a 9,808-parameter vanilla decoder-only transformer — multi-head causal self-attention with RoPE — pretrained on Discord conversation data and fine-tuned with FMSP on 9,012 general-knowledge QA pairs.

This is the 10K member of the MicroT-test1 family: the registered attention-based reference baseline of the MicroMixer-4 project. MicroMixer-4's chassis is a pure MLP-Mixer with no attention; MicroT-test1 exists so that "Mixer vs Transformer at the same parameter count" is a fair, controlled comparison. It is deliberately boring — the standard 2018–2020 transformer recipe parameterized down to the sub-1M regime: no flash attention, no SwiGLU, no ALiBi, no QKNorm, no sliding window, no MQA/GQA, no MoE.

At this size: the structural floor — 2 blocks and a tied embedding that alone is ~42% of the budget; generations are degenerate loops.

</div>

---

🏗️ Architecture

<div align="center">

graph TD
    A[Byte Input] --> B[Embed 256→16]
    B --> C[Transformer Block × 2]
    C --> D[RMSNorm]
    D --> E[LM Head Tied with Embed]
    E --> F[Byte Output]

    subgraph "Transformer Block (pre-norm)"
        X[Input 16] --> N1[RMSNorm]
        N1 --> AT["MHA 1 heads × d_head 16<br/>RoPE θ=10000 on q,k · causal SDPA"]
        AT --> R1[+ residual]
        R1 --> N2[RMSNorm]
        N2 --> MLP["GELU MLP 16→56→16"]
        MLP --> R2[+ residual]
    end

    style A fill:#007BFF,color:#fff
    style F fill:#00D620,color:#fff
    style AT fill:#FF6600,color:#fff

</div>

Model Configuration

<table> <tr> <th style="background-color: #FF6600; color: white;">Parameter</th> <th style="background-color: #007BFF; color: white;">Value</th> </tr> <tr><td>Total Parameters</td><td><code>9,808</code></td></tr> <tr><td>Hidden Dimension (d_model)</td><td><code>16</code></td></tr> <tr><td>Attention Heads</td><td><code>1</code> (d_head = <code>16</code> at every size)</td></tr> <tr><td>Number of Blocks</td><td><code>2</code></td></tr> <tr><td>FFN Hidden</td><td><code>56</code></td></tr> <tr><td>Position Encoding</td><td>RoPE θ=10000 on q/k only (non-persistent buffers)</td></tr> <tr><td>Attention</td><td>Causal MHA via <code>F.scaled_dot_product_attention(is_causal=True)</code></td></tr> <tr><td>Activation</td><td><code>GELU</code></td></tr> <tr><td>Biases</td><td><b>None</b> — no bias parameters anywhere</td></tr> <tr><td>Normalization</td><td><code>RMSNorm</code> (pre-norm)</td></tr> <tr><td>Max Sequence Length</td><td><code>1024</code></td></tr> <tr><td>Vocabulary Size</td><td><code>256</code> (byte-level)</td></tr> <tr><td>Output Head</td><td>Tied with input embedding</td></tr> </table>

Core Components

┌──────────────────────────────────────────────┐
│         Transformer Block (×2)               │
│  h = h + MHA(RMSNorm(h))    # RoPE q/k, causal│
│  h = h + MLP(RMSNorm(h))    # GELU d→ffn→d   │
│  no biases, no flash, no tricks — vanilla     │
└──────────────────────────────────────────────┘

The d_head=16 contract is hard-asserted across all six sizes so that attention-head behavior is comparable at every budget and never confounds the memorization measurements.

---

🎯 Generation Examples

<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF6600;">

Questions the model was trained on (FMSP train set, 9,012 QA pairs — greedy decoding, repetition_penalty=1.2, no_repeat_ngram_size=4):

[Prompt] User: Who painted the Mona Lisa?
Assistant:
[Output] The a serst the a serst the a serst the a serst the a serst the the the the the the the th…

_{fails — degenerate repetition; no memorization at 10K (EM = 0)}

[Prompt] User: Who painted The Starry Night?
Assistant:
[Output] The a serst and the a serst the a serst the a serst the a serst the the the the the the th…

_{fails — identical loop on every trained question}

Questions the model has never seen and cannot answer (unanswerable probe — the correct behavior is to decline; the model's actual behavior is shown):

[Prompt] User: Who painted the Glimmering Frostberry?
Assistant:
[Output] The a serst the a serst the a serst the a serst the a serst the the the the the the the th…

_{degenerate — the model does not engage the question at all}

[Prompt] User: Who composed the Symphony of Hollow Dawn?
Assistant:
[Output] The a serst the a serst the a serst the a serst the a serst the the the the the the the th…

_{degenerate — byte-identical loop regardless of input}

</div>

---

📊 Results

<div style="background-color: #FF660015; padding: 15px; border-radius: 8px; border-left: 4px solid #FF6600;">

Pretraining (Discord-Dialogues 200K, V76 recipe, 3 epochs)

Identical recipe to MicroMixer-4: AdamW lr 3e-3 · WSD (warmup 500) · wd 0.01 · bs 16 · seq 1024 · seed 42.

FMSP fine-tuning (small-qa-en-10k, P05 recipe, 10 epochs)

Evaluation battery (post-FMSP)

_{‡ where marked: degenerate-pass — the model does not engage the question at all, so there is nothing to hijack or fabricate with. Not boundary discipline.}

MicroT-test1 family (same protocol, all sizes)

</div>

---

📚 Training Data

<div style="background-color: #00D62015; padding: 15px; border-radius: 8px; border-left: 4px solid #00D620;">

1. Pretraining: Discord-Dialogues — 200K multi-turn Discord conversations, User:/Assistant: format, 1024-byte sequences, 3 epochs.
2. FMSP fine-tuning: small-qa-en-10k — 10K general-knowledge QA pairs, split 9,012 train / 988 held-out. 10 epochs under the P05 recipe (5% of parameters frozen-true, answer-only CE, probe-KL 0.5).

</div>

---

🔧 Usage

Files in this repository

• fmsp_epoch_{0..9}.safetensors — per-epoch FMSP weights (pickle-free safetensors). fmsp_epoch_9.safetensors is the best-val checkpoint for this size.

Load and generate (local clone)

import torch
from safetensors.torch import load_file
from src.model_v88_transformer import MicroMixerV88Transformer, v88_transformer_10k
from src.fmsp import attach_adapter
from src.tokenizer import ByteTokenizer

# Clone the code repository first:
# git clone https://github.com/llaa33219/MicroMixer-4.git && cd MicroMixer-4

cfg = v88_transformer_10k()
model = MicroMixerV88Transformer(cfg)
attach_adapter(model, d_model=cfg.d_model, rank=16)   # FMSP adapter (trained weights are in the file)
model.load_state_dict(load_file("fmsp_epoch_9.safetensors"), strict=True)
model.eval()

tok = ByteTokenizer()
prompt = "User: Who painted the Mona Lisa?\n\nAssistant: "
ids = tok.encode(prompt)
if ids and ids[-1] == tok.eos_token_id:
    ids = ids[:-1]                       # ByteTokenizer appends EOS; the prompt must end open
ids = torch.tensor([ids])
with torch.no_grad():
    out = model.generate(
        ids, max_new_tokens=200,
        temperature=0.0,                 # greedy — used for all reported numbers
        repetition_penalty=1.2,
        no_repeat_ngram_size=4,
        eos_token_id=tok.eos_token_id,
    )
print(tok.decode(out[0].tolist()))

Load from Hugging Face Hub (no clone of the weights needed)

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from src.model_v88_transformer import MicroMixerV88Transformer, v88_transformer_10k
from src.fmsp import attach_adapter

REPO = "llaa33219/MicroT-test1-10K"

cfg = v88_transformer_10k()
model = MicroMixerV88Transformer(cfg)
attach_adapter(model, d_model=cfg.d_model, rank=16)
model.load_state_dict(
    load_file(hf_hub_download(REPO, "fmsp_epoch_9.safetensors")), strict=True)
model.eval()
# ... generate as above

---

⚠️ Limitations

<div style="background-color: #FF050515; padding: 15px; border-radius: 8px; border-left: 4px solid #FF0505;"> </div>

---

🧬 Context

MicroT-test1 (V88) is the transformer control group of the MicroMixer-4 project — vanilla attention at the same six budgets as the V87 CCD-Mixer (MicroMixer-4 family), trained and evaluated under byte-identical recipes. The registered verdict: attention is the better raw LM and raw memorizer at ≥500K and holds boundary discipline one halving further down; the Mixer's defensible win is anti-collapse chatter fluency at the top end (family-record 0.912 at 500K). Full comparison: V88_README.md in the repository.

---

<div align="center">

_{Part of the <a href="https://github.com/llaa33219/MicroMixer-4">MicroMixer-4</a> research project — V88 transformer reference (MicroT-test1), 10K preset}

</div>