llm-150m

Cubic Hier 150M

An experimental bilingual dialogue/reasoning language model with approximately 157M parameters. It keeps the faithful CubicV7 differentiable depth-memory path, uses hierarchical cosine retrieval in the two penultimate attention layers, and finishes with full global causal attention. Every sequence head uses QK RMSNorm.

Training recipe

1. Base pretraining on a 72/13/10/5 stream of FineWeb-Edu, Russian FineWeb-2, deduplicated CodeParrot Python and FineMath-4+.
2. Assistant-only SFT on Smol-SmolTalk and English/Russian Aya examples.
3. Assistant-only reasoning SFT on the shortest verified-correct traces from OpenR1-Math-220k.
4. DPO with an SFT anchor on Anthropic HH-RLHF and corrected UltraFeedback.

The optimizer is the project's hybrid orthogonalized Muon (large matrices and a boosted depth group) plus AdamW for embeddings, norms, scalars and gates. Base training uses warmup-stable-decay and a one-token-ahead MTP auxiliary loss; embeddings, RMSNorm scales and controls are excluded from AdamW decay. SFT teaches explicit direct and <think>...</think> system-prompt modes; set CUBIC_REASONING=1 in chat mode to request the reasoning format.

Important

This is a custom PyTorch architecture, not a drop-in Transformers model. Run train_and_chat.py with CUBIC_MODE=chat; it downloads/loads all required files and starts an interactive console. The model is small and experimental: verify factual, safety-critical and mathematical answers independently.

Each source dataset retains its own license/terms. FineWeb corpora inherit the Common Crawl terms described on their dataset cards; SmolTalk, Aya and OpenR1 are Apache-2.0; HH-RLHF and UltraFeedback use their published dataset licenses.