BananaMind-2.1-Lite-All-Looped-AdamW

BananaMind 2.1 Lite All-Looped

This all-middle-looped AdamW experiment is published separately at Banaxi-Tech/BananaMind-2.1-Lite-All-Looped-AdamW. It does not overwrite the original Lite or the partially looped AdamW experiment.

BananaMind 2.1 Lite is a 24,949,999-parameter causal language model built for the under-25M class.

• 19,950,029 parameters in the tied-token Transformer core
• 4,999,970 parameters in a two-hash causal trigram module
• BananaMind 2 Nano's 8,192-token tokenizer
• 13 physical layers and 24 effective layer passes
• execution: L1 → (L2 → ... → L12) × 2 → L13
• L1 is the prelude, L2-L12 are one recurrent middle stack, and L13 is the coda
• every middle-layer visit reuses the same physical layer weights on pass two
• 4,096-token context

The trigram module uses two independent 51,699-entry tables with 48 features per hash. Their 96 concatenated features are projected to the 384-wide residual stream. The trigram representation is computed once and reinjected before every visit to L5 and L9, once in each recurrent-stack pass. L5 and L9 have separate learned injection scales, both initialized to 0.5.

75B-token streamed curriculum

The NPset allocation is 75% normalized code and 25% raw original_code. Curriculum weights move from web-heavy foundations to a balanced phase and then greater math, synthetic-text, phrase, and code coverage. A token-credit scheduler keeps the complete run within one global batch of every target allocation.

Training uses pure AdamW with (0.9, 0.95) betas: peak LR 0.003 for the Transformer core, embeddings, and controls, and peak LR 0.001 for the n-gram module. Both rates use warmup-stable-decay with a final 15% cosine decay. Checkpoints are uploaded every 5% with safetensors, tokenizer files, pinned dataset revisions, metrics, exact source-token accounting, and complete state for resuming.

Launch on Hugging Face Jobs

# Start a new 4 x H200 run (32 sequences/GPU, global batch 128).
./launch_lite_all_looped_hf_job.sh 4 fresh

# Resume it later.
./launch_lite_all_looped_hf_job.sh 4 resume

# The same global batch on 8 x H200 (16 sequences/GPU).
./launch_lite_all_looped_hf_job.sh 8 resume

The launcher expects HF_TOKEN to be available to the Hugging Face CLI and as a Hugging Face Jobs secret.