RDT-lm

Recursive Division Tree (RDT) Language Model with Topological Adam Optimizer

Author: Steven Reid License: Apache 2.0

---

1. Summary

This project combines two new ideas:

1. Recursive Division Tree (RDT) Embeddings – a mathematical rule that organizes each word in the vocabulary into logarithmic shells. Words with similar depth values belong to the same shell, producing a constant-depth, fractal-like structure in the embedding space.

2. Topological Adam Optimizer – a physics-inspired optimizer that extends Adam with two internal fields (alpha, beta) coupled by an energy equation. This allows parameters to evolve smoothly while maintaining energy balance, reducing oscillations and improving stability.

Together they form an experimental language model that learns relationships between words and can generate new short sentences.

---

1. Key Concepts

Component Description

Recursive Depth (rdt_depth) Maps each index n to a depth using a log-logarithmic rule, creating nested shells of similar complexity. Shell-based Embeddings Each word receives an embedding vector stored in a shell according to its depth. Energy-based Optimization The Topological Adam optimizer updates weights using both gradient information and internal field coupling. Hybrid Training Objective Combines geometric embedding similarity with simple statistical (n-gram) learning. Text Generation Generates continuations from prompt text by combining learned bigram and trigram statistics with geometric similarity across shells.

---

1. Running the Model

Requirements

torch numpy topological-adam

Quick Start

from rdt_language_model import RDTLanguageModel from topological_adam import TopologicalAdam import torch, random

prepare training data

corpus, vocab = generate_corpus()

initialize model

model = RDTLanguageModel(vocab, embedding_dim=128, alpha=1.5)

train

model.train_rdt(corpus, epochs=10, lr=1e-3)

generate text

print(model.generate("the man walked"))

---

1. Example Output

Prompt: "the man walked" the man walked to the river and he entered slowly and she was sad was

---

1. Results

Optimizer Accuracy (MNIST test) Notes

Adam 91.39 % Baseline RMSProp 88.35 % Topological Adam 91.60 % Stable energy and strong robustness

The optimizer achieved the same or better accuracy while maintaining smooth energy dynamics during training.

---

1. Why It Matters

Shows that recursive geometric organization can replace random embedding initialization.

Demonstrates a new optimizer family based on topological energy coupling.

Suggests that language structure and physical field dynamics can be unified in a single computational model.

---

1. Repository Contents

rdt_language_model_topological_adam.ipynb # full Colab notebook topological_adam.py # optimizer implementation requirements.txt README.md

---

1. Citation

If you use this work, please cite:

Reid, S. (2025). Recursive Division Tree Language Model with Topological Adam Optimizer.
Available at https://huggingface.co/<username>/rdt-language-model-topological-adam