nanochat-muon-limited

Nanochat d26 Muon Limited

This repository hosts maximorulli/nanochat-muon-limited, a d26 Nanochat decoder-only causal language model with custom Transformers remote code for loading the architecture and tokenizer.

Loading

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("maximorulli/nanochat-muon-limited", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("maximorulli/nanochat-muon-limited", trust_remote_code=True, torch_dtype="auto")

Architecture

• Decoder-only causal Transformer.
• 26 layers, hidden size 1664, 13 attention heads, 13 key/value heads.
• Context length 2048 and vocabulary size 32768.
• RoPE positional encoding, RMSNorm/pre-norm blocks, QK normalization.
• ReLU-squared MLP.
• Untied token embedding and LM head.
• Alternating value embeddings / ResFormer-style value paths; 13 value-embedding layers.
• Residual and x0 lambda scalar parameters.
• Linear layers without bias.
• Logit softcap: 15 * tanh(logits / 15).
• Total parameters: 1,681,790,292.
• Estimated training FLOPs/token: 6.573818e9.

Tokenizer

The tokenizer is a custom RustBPE/tiktoken-style tokenizer with GPT-4-like splitting and 32768 vocabulary entries. It was trained with a 2,000,000,000 character cap and a 10,000 document cap.

Special tokens:

Token byte statistics: min 1, max 19, mean 6.6029, std 2.8250.

Base Checkpoint Metrics

• Saved checkpoint metadata is the source of truth for the BPB value.

Datasets

Base/tokenizer/pretraining:

• karpathy/fineweb-edu-100b-shuffle, parquet shards shard_00000.parquet through shard_01822.parquet.
• Train split: all shards except the final validation shard.
• Validation split: final shard.
• Packing: BOS-aligned best-fit packing to 2048 tokens.
• Effective token utilization: 100% packed utilization; approximately 35% of overlong rows cropped at T=2048.

SFT provenance documented for this training series:

• HuggingFaceTB/smol-smoltalk, train and test.
• Custom identity conversations, used twice during SFT.
• cais/mmlu, auxiliary_train for SFT training and all/test for validation/evaluation.
• openai/gsm8k, main/train for SFT training and main/test for validation/evaluation.
• Synthetic SimpleSpelling(size=200000, split=train).
• Synthetic SpellingBee(size=80000, split=train).
• Synthetic SpellingBee(size=256, split=test).

Evaluation datasets:

• allenai/ai2_arc: ARC-Easy test and ARC-Challenge test.
• cais/mmlu: all/test.
• openai/gsm8k: main/test.
• openai/openai_humaneval: test.
• Synthetic SpellingBee test.
• Tokenizer evaluation categories: news, Korean, code, math, science, FineWeb-Edu train, FineWeb-Edu validation.

Tokenizer Evaluation

Compared with GPT-2:

Compared with GPT-4:

Related SFT And Downstream Metrics

These related downstream/SFT results are included for context and should not be read as direct base-checkpoint metrics unless explicitly stated.

Muon-full-derived SFT, source d26, SFT step 501:

Eval settings: dtype=bfloat16, temperature=0, max_new_tokens=512, top_k=50, batch_size=8.

AdamW-source SFT, base model step 7226, SFT step 501:

• SFT validation BPB: 0.38646146285770866.
• ChatCORE/downstream task metrics: not reported.

Muon step-2000-source SFT, base model step 2000, SFT step 80:

• SFT validation BPB: 0.3852505442606574.
• Stop reason: reached target validation BPB.
• ChatCORE/downstream task metrics: not reported.

Metric Availability Notes

• CORE estimates are not reported for these base checkpoints.
• Metrics are reported only when they are part of this release.
• A later AdamW continuation checkpoint is outside the scope of this release.

Intended Use

This is a small research/training artifact intended for experimentation with Nanochat-style training, optimizer comparisons, tokenizer behavior, and lightweight causal language modeling. It has not been aligned for broad production use.

Limitations

The model may produce incorrect, unsafe, biased, or low-quality text. The base checkpoints are not instruction-tuned chat assistants. Context length is 2048 tokens. Generation through the included Transformers implementation is portable but does not include the original training code's custom Flash Attention path.