BananaMind-2.1-Lite-AdamW

BananaMind 2.1 Lite

AdamW training checkpoints are published separately at Banaxi-Tech/BananaMind-2.1-Lite-AdamW so the earlier Muon experiment is preserved.

BananaMind 2.1 Lite is a 24,949,999-parameter causal language model built for the under-25M class.

• 19,950,029 parameters in the tied-token Transformer core
• 4,999,970 parameters in a two-hash causal trigram module
• BananaMind 2 Nano's 8,192-token tokenizer
• 13 physical layers and 15 effective layer passes
• execution: L1 → L2 → L3 → L4 → L5 → L5 → L6 → L7 → L8 → L9 → L9 → L10 → L11 → L12 → L13
• both visits to L5 share one physical layer; both visits to L9 share another
• 4,096-token context

The trigram module uses two independent 51,699-entry tables with 48 features per hash. Their 96 concatenated features are projected to the 384-wide residual stream. The trigram representation is computed once and reinjected before every visit to L5 and L9. L5 and L9 have separate learned injection scales, both initialized to 0.5.

75B-token streamed curriculum

The NPset allocation is 75% normalized code and 25% raw original_code. Curriculum weights move from web-heavy foundations to a balanced phase and then greater math, synthetic-text, phrase, and code coverage. A token-credit scheduler keeps the complete run within one global batch of every target allocation.

Training uses pure AdamW with (0.9, 0.95) betas: peak LR 0.003 for the Transformer core, embeddings, and controls, and peak LR 0.001 for the n-gram module. Both rates use warmup-stable-decay with a final 15% cosine decay. Checkpoints are uploaded every 5% with safetensors, tokenizer files, pinned dataset revisions, metrics, exact source-token accounting, and complete state for resuming.

Launch on Hugging Face Jobs

# Start a new 4 x H200 run (32 sequences/GPU, global batch 128).
./launch_lite_25m_hf_job.sh 4 fresh

# Resume it later.
./launch_lite_25m_hf_job.sh 4 resume

# The same global batch on 8 x H200 (16 sequences/GPU).
./launch_lite_25m_hf_job.sh 8 resume

The launcher expects HF_TOKEN to be available to the Hugging Face CLI and as a Hugging Face Jobs secret.