streammixer-26m-babylm

Stream Mixer 26M — BabyLM 2026 Strict

A ~26M-parameter Stream Mixer (linear-time, attention-free) trained on the BabyLM 2026 Strict 100M-word corpus for 10 epochs.

Model Details

• Architecture: Stream Mixer (no attention, no KV cache)
• Parameters: 26,247,616
• Layers: 14
• Hidden dim: 384
• Streams: 32, stream dim 64, 4 read heads
• Vocab: 16,384 BPE (trained on BabyLM corpus)
• Context: 1,024 tokens
• Training: 10 epochs, ~1B tokens, WSD schedule, MuonAdamW optimizer

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ecreeth/streammixer-26m-babylm")
model = AutoModelForCausalLM.from_pretrained(
    "ecreeth/streammixer-26m-babylm",
    trust_remote_code=True,
)

inputs = tokenizer("The cat sat on the", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))

BabyLM Challenge

• Track: Strict (100M words, 10 epochs max)
• Eval repo: babylm-eval
• Leaderboard: BabyLM-Leaderboard-2026

Results (zero-shot, causal, temperature 1.0)

Results (fine-tuning, GLUE)

Evaluated with babylm-eval on the strict track. Zero-shot tasks measure linguistic knowledge; fine-tuning tasks measure transfer learning to downstream classification.

Architecture

The Stream Mixer replaces self-attention with linear-time stream mixing:

1. Input tokens are embedded and fed into n_streams parallel streams
2. Streams are mixed via learned query/read-head projections (no attention matrix)
3. A lightweight feedforward layer processes the mixed streams
4. Output is projected back to vocabulary logits

This gives O(n) complexity per token (vs O(n²) for Transformers) and no KV cache — generation uses model.step() for token-by-token decoding.

Training Details

Links

• BabyLM Challenge 2026
• BabyLM Eval