octopus-120m

Octopus-120M

Octopus-120M is a 125,829,354-parameter nanochat base model pretrained in a globally base-8 numeral world. Every maximal ASCII decimal digit run in the paired base-10 corpus was rendered in base 8 while document order, arithmetic placement, tokenizer, architecture, seed, optimizer, schedule, and training token budget were held fixed.

The name refers to the eight-fold numeral world. The accompanying interpretability experiments test whether its arithmetic geometry is organized as a single T8 channel or a nested T2/T4/T8 hierarchy.

Training configuration

• Architecture: nanochat depth 8, width 512, 4 attention heads
• Parameters: 125,829,354 total
• Context length: 2,048
• Vocabulary: 32,768
• Training budget: approximately 2.013B tokens
• Parameter:data ratio: 48
• Seed: 42
• Hardware: 1x NVIDIA H100 SXM 80GB
• Training precision: FP8 matmuls with master checkpoint weights
• Final step: 7,680
• nanochat commit: 92d63d4e8bb4df75c3b71618f31ddde2378b2bcd

Data and tokenizer

• Dataset: https://huggingface.co/datasets/Yujivus/nanochat-climbmix-arithmetic-base8
• Tokenizer: https://huggingface.co/Yujivus/nanochat-tokenizer-digit-32k

The tokenizer is exactly the same 32K digit-level tokenizer used by the paired base-10 baseline. Tokens 0 through 7 represent base-8 digits. Tokens 8 and 9 remain in the unchanged vocabulary but are not valid base-8 digits.

Checkpoint format

These are raw nanochat PyTorch checkpoints, not Transformers AutoModel checkpoints. Reconstruct the model using the pinned nanochat commit above and place the tokenizer/checkpoint directories under NANOCHAT_BASE_DIR.

The repository includes intermediate and final model checkpoints, optimizer states, metadata, training logs, tokenizer files, and the exact RunPod scripts. Evaluation and Fourier/steering outputs can be added by rerunning the uploader after those artifacts are created under evals/octopus-d8-r48-seed42/.

Paired experiment

• Base-10 baseline: https://huggingface.co/Yujivus/nanochat-base10-d8-r48-seed42
• Base-8 model: this repository

This repository is not affiliated with Karpathy, NVIDIA, or Hugging Face.