TUPOI-1

TUPOI-300M: Symplectic Post-Transformer Language Model with O(1) Memory

<div align="center">

A sub-quadratic, attention-free sequence modeling architecture that replaces dense attention matrices with a Symplectic Hamiltonian Integrator (Velocity-Verlet).

Author: NARE LABS (Built by 15 y.o. independent researcher)

</div>

---

📌 Overview

TUPOI-300M is a 304M-parameter proof-of-concept post-transformer architecture that completely eliminates the Attention mechanism and the persistent KV-cache in favor of continuous Hamiltonian phase-space dynamics.

Key Highlights:

• Strictly Constant Memory ($O(1)$): The persistent recurrent context state requires exactly 6.00 KB of VRAM forever, regardless of whether generating 512 or 65,536 tokens.
• Zero Information Dissipation: Integrated via the symplectic Velocity-Verlet leapfrog scheme with Jacobian determinant $\det(J) \equiv 1.000000$ (Liouville's theorem).
• Pretrained Scale: 304.2M active parameters trained across 10,000 steps on TinyStories on a single NVIDIA Tesla T4 GPU.

---

🚀 Quickstart: Running Inference in Python

import torch
import torch.nn.functional as F
import tiktoken
from huggingface_hub import hf_hub_download

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# 1. Download model weights from HuggingFace
weights_path = hf_hub_download(repo_id="DanilKZ/TUPOI-1", filename="tupoi-300m.pt")

# 2. Clone repository code for architecture definitions:
# git clone https://github.com/starface77/TUPOI.git
from tupoi import TUPOI300M, generate_text

enc = tiktoken.get_encoding("gpt2")
model = TUPOI300M(vocab_size=enc.n_vocab, d_model=1024, num_layers=24, d_ff=4096, max_seq_len=512).to(device)

checkpoint = torch.load(weights_path, map_location=device)
sd = checkpoint["model_state_dict"] if "model_state_dict" in checkpoint else checkpoint
if "opinion_anchor.anchor_state" in sd:
    del sd["opinion_anchor.anchor_state"]

model.load_state_dict(sd, strict=False)
model.eval()

# 3. Generate text
prompt = "Once upon a time, there was a little girl named Lily who"
output = generate_text(model, enc, prompt=prompt, max_new_tokens=50, temperature=0.7)
print(output)