cascade-1b-bpe-logic

Cascade-1B-BPE-Logic

Public experimental causal language model with approximately 1.04B parameters. It uses the open 32k Mistral tokenizer, byte-derived initialization from Cascade-1B-Logic, four local token layers, entropy-guided token patches, and one active eight-layer core path out of three per batch.

Training streams FineWeb-Edu, FineMath, Cosmopedia v2, OpenR1-Math, and decontaminated Codeforces CoT. Hidden matrices use Muon; embeddings, normalization parameters, and vocabulary heads use AdamW. Checkpoints are uploaded every 5000 steps and lightweight state every 1000 steps. Hub failures are asynchronous and never terminate training.

This is an early base model, not an instruction-tuned assistant. Checkpoints may produce incoherent or unsafe text.

Tokenizer source: mistralai/Mistral-7B-v0.1.