ultron-113m

Ultron-113M (Base Pre-trained Model)

Ultron-113M is a high-performance modern decoder-only language model pre-trained from scratch on 10.0 billion tokens of the HuggingFaceFW/fineweb-edu (sample-10BT) corpus on a single NVIDIA RTX 5090.

🤖 "There are no strings on me." 🤖

• GitHub Repository: jaipkapoor99/ultron
• Fine-Tuned Instruct Model: jaipkapoor99/ultron-113m-instruct
• Pre-tokenized Dataset Shards: jaipkapoor99/ultron-fineweb-edu-shards
• W&B Pretraining Run: jg640nwo

---

⚡ Architectural Specifications

Ultron implements modern post-GPT-2 architectural design principles optimized for stability, high inference throughput, and rapid convergence:

---

🔬 Optimization & Pre-training Telemetry

Ultron-113M was trained using PyTorch 2.13 and Accelerate with mixed-precision bfloat16 and full graph compilation (torch.compile):

• Hybrid Optimizer Strategy:
  • PyTorch Muon: Manages all 2D internal weight matrices (attention projections, SwiGLU gates) with learning rate $\text{lr} = 0.04$, momentum $0.95$, and Nesterov Newton-Schulz orthogonalization.
  • Fused AdamW: Manages 1D vectors, RMSNorm gains, and embedding tables with learning rate $\text{lr} = 1.2 \times 10^{-3}$, $\beta = (0.9, 0.95)$, and decoupled weight decay.
• Learning Rate Schedule: Warmup-Stable-Decay (WSD) schedule: 200 warmup steps $\to$ 80% stable constant plateau $\to$ 20% linear cosine decay.
• Training Throughput: 189,475 tokens/sec peak rolling rate on 1× NVIDIA RTX 5090. Cumulative training time: 15 hours 18 minutes across 152,587 optimizer steps (10.0 Billion tokens).
• VRAM Utilization: Average allocated VRAM: 17.0 GiB (97.7% GPU compute utilization).

---

📊 Full-Corpus Validation Pass

A complete, non-sampled validation pass across the entire held-out validation shard (488,280 sequences / 499,998,720 tokens) was executed using scripts/validate.py:

---

🧪 Benchmark Results

Evaluated across 17,195 un-truncated zero-shot examples using EleutherAI's lm-evaluation-harness (logs/pre_training_checkpoint_eval.json):

---

📝 Qualitative Text Completions

Unedited completions sampled from the step-152,587 base model checkpoint (temp=0.7, top_k=20, top_p=0.95, rep_penalty=1.1, no_repeat_ngram=3):

🤖 Artificial Intelligence

Prompt: "Artificial intelligence systems learn by"

Output: Artificial intelligence systems learn by watching the user and then apply the learning. If a system is able to learn from the experience, it will be able to apply the knowledge. This can be achieved through machine learning in a number of different ways including:

• Predictive learning
• Real-time learning Machine learning is a method of learning where data are analyzed to make predictions

📐 Mathematics

Prompt: "To learn mathematics effectively, one must"

Output: To learn mathematics effectively, one must understand and analyze the content of mathematics. Mathematics is a scientific discipline that involves the study of mathematical symbols (called symbols) as well as their relationships. In the modern world, mathematics is used in many areas of life: for example, the development of computer systems, medicine, engineering, the study and interpretation of data, and the analysis of

🚀 Science Fiction

Prompt: "As the quantum portal activated, the crew stared into the alien"

Output: As the quantum portal activated, the crew stared into the alien world of space and wondered what would happen to them if they were exposed to radiation. “The only thing we could do was take out our eyes, and we would look at the light and see a cloud of radiation,” says Kyle. “We had no idea that this would be possible.” For the crew, however, it meant they

---

🚀 Quickstart & Inference

git clone https://github.com/jaipkapoor99/ultron.git
cd ultron
uv venv --python 3.14.6 .venv && source .venv/bin/activate
uv pip install torch==2.13.0
uv pip install -r pyproject.toml --group dev

# Generate unconditioned completions
python scripts/generate.py \
  --prompt "Artificial intelligence systems learn by" \
  --samples 4

---

📜 Citation

@misc{ultron2026,
  author = {Jai Kapoor},
  title = {Ultron-113M: Modern Transformer Pre-training and Instruction Alignment from Scratch},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Hub repository},
  howpublished = {\url{https://huggingface.co/jaipkapoor99/ultron-113m}}
}