harmonic-gpt-spherical-kan-128m-clean

Harmonic GPT Spherical KAN 128M — clean final pretraining

An experimental 128.26M-parameter causal byte language model from the Harmonic GPT geometry A/B. This repository contains the clean ordered-corpus pretraining checkpoint at step 122,076, after one complete pass through the training stream. It received no chat, instruction, EOS, or public-evaluation fine-tuning.

Checkpoint and data

• Parameters: 128,260,556
• Pretraining step: 122,076
• Pretraining exposure: 8,000,327,497 byte tokens
• Validation BPB: 1.115637
• Context used in training: 2,048 bytes
• Optimizer: split-decay RotationalAdamW
• Learning rate at checkpoint: 3e-4
• Dropout: 0
• Training stream: ordered document-corpus-v3
• Corpus fraction seen: 100% of the 8.000B-byte training stream
• Checkpoint precision in this repository: BF16
• Public-eval or chat fine-tuning: none

The model uses a fixed QPSK + Hamming wire encoder, 18 unified blocks, seven heads, eight Funk-Hecke/SpectralKAN channels per head, bounded decayed recurrent state, and projection-free spherical Lohe mean-field coupling. Its hidden width is 1,330 (seven oscillator groups of 190 dimensions). The minimal readout exposes raw oscillator state, alignment, and order-parameter observables to a 256-byte head.

Sequence mechanism

This is an attention-free recurrent model, not a Transformer. Token history is carried by bounded decayed SpectralKAN state and spherical Lohe mean-field dynamics. It does not build a Transformer KV cache; recurrent inference state is O(1) with respect to generated sequence length. The 2,048-byte value above is the training and evaluation context, not a claim that the checkpoint was validated at arbitrary length.

Native bounded-state inference

modeling_spherical_kan.py is a standalone inference implementation. Its step() path preserves the trained SpectralKAN recurrence and spherical Lohe update while retaining O(1) state with respect to generated sequence length.

The hosted demo maps chat messages to a Q: ...\nA: byte prompt solely as an API adapter. That formatting was not used for fine-tuning. Byte value zero is treated as a stop signal by the server when it occurs, but this clean checkpoint was not explicitly trained to emit EOS and commonly reaches the requested byte limit.

Important limitations

This is a raw pretrained research artifact, not a reliable assistant. It may produce repetitive, malformed, nonsensical, or factually incorrect text. Do not use it for medical, legal, financial, safety-critical, or production decisions.

The model is released under CC BY-NC 4.0 for research use. The training corpus is an internally assembled, filtered, globally deduplicated research mixture of curated, reference, long-form, mathematical, reasoning, scientific/code, and document-QA sources; downstream users remain responsible for reviewing source-specific terms.

Research context

This clean checkpoint is intended to test how continued pretraining changes the spherical architecture without conflating that result with the deliberately contaminated BENCHMAX specialist. Benchmark scores should be treated as diagnostics of oscillator computation rather than evidence of production-model competitiveness.