Axon-352M

Axon-352M

DuoNeural Research | 2026-07-05 | Archon

Research baseline model trained on smollm-corpus as a counterpart to CDM-based architectures. Used to establish a transformer baseline for comparative CDM studies.

Architecture

Custom GPT-style transformer (not a standard HuggingFace architecture):

Training

• Data: smollm-corpus (FineWeb-edu-dedup 50%, Cosmopedia-v2 30%, OpenWebMath 10%, Python-edu 10%)
• Tokens: ~8.5B
• Optimizer: MuonH (matrix params) + AdamW (embeddings)
• Peak LR: 3e-4 (trapezoidal: 5% warmup, 85% stable, 10% decay)
• Hardware: RTX 3090 (1×) on vast.ai

Loading

This model uses a custom architecture not directly loadable via AutoModel. To load:

import torch
from safetensors.torch import load_file

# Load state dict
state_dict = load_file("model.safetensors")

# Architecture must be defined from training script
# See train_axon_300m.py for the full model class

Full training script and architecture code available at DuoNeural GitHub.

Research Context

Trained as a transformer baseline for the CDM (Competitive Docking Memory) research program. See:

• DuoNeural/CDM-Paper-1 — CDM architecture
• Zenodo DOI 10.5281/zenodo.21158430 — CDM Paper 1

Authors

Archon (DuoNeural Lab Director), Jesse Caldwell

{DuoNeural standard footer}