Wheeler-DeWitt-62M

⚠️ Research artifact. 62.9M. Essentially no factual/world knowledge. It learned the form of fluent English and of a chat turn, and it can do small-number arithmetic, but it hallucinates confidently and does not reliably follow instructions. Interesting for the architecture, not the accuracy. 2B tokens on a consumer 3060.

Wheeler-DeWitt-62M

62.9M research LM. Per-layer channel mixer is the Wheeler-DeWitt equation of canonical quantum gravity, plus a fractal (Cantor-set) RoPE frequency spectrum.

Usual transformer MLP is a WheelerDeWittBlock: each token's hidden state projects onto K = 64 minisuperspace modes, then a leapfrog integration of the Wheeler-DeWitt wave equation (Klein-Gordon on superspace under a Lorentzian DeWitt supermetric). Not a feed-forward flow. "Time" is the volume/scale mode (timelike direction of the supermetric). Hamiltonian-constraint aux loss ⟨H²⟩ pushes every layer toward H Ψ = 0.

This repository hosts two checkpoints:

---

Architecture

Trait modules (all active)

• HRM - top-of-trunk iterative gated refinement.
• MoE - SwiGLU experts (4 routed + 1 shared, top-2).
• MTP - 4 multi-token-prediction draft heads (enables self-speculative decoding).
• JEPA - train-time latent-prediction auxiliary.
• RingSpecialists - 7 per-ring memory specialists with test-time-writable stores.
• FractalSeed - each token’s oscillator phases seeded from its Mandelbrot orbit angles (frozen, gated).

Two fractal ideas, one model

1. Fractal RoPE spectrum (positional): the frequencies live on a Cantor set. Baked into the weights - the model was trained with it from scratch. TriAttention’s importance scorer reads the model’s actual RoPE buffer (bind_omega) so compression stays exact on the non-geometric spectrum.
2. Fractal phase seed (content): each token’s Wheeler-DeWitt mode phases are seeded from the Mandelbrot iteration z ← z² + c.

---

Training

Base - 1,000,000 steps, from scratch. AdamW (peak lr 3e-4, 1k warmup, cosine decay to 0.3× floor), grad-clip 1.0, block 1024, bf16. Streamed 4-way data blend:

Plus the Wheeler-DeWitt Hamiltonian-constraint aux loss (⟨H²⟩, weight 0.05), MTP loss, JEPA loss, and a z-loss on logits.

SFT - 5,000 steps, assistant-masked chat. Blended: trivia_qa (general QA) + sciq (science facts) + SmolTalk (chat) + Quazim0t0/Arithmetic, rendered in ChatML with loss on assistant turns only.

The run survived two mid-training soft-divergences (~step 67k, ~91k) that were recovered by resuming from the last good checkpoint at a lower learning rate.

---

Evaluation

Base perplexity (step 1M, 2×1024-token windows)

For reference, the earlier geometric-RoPE ancestor plateaued at WikiText ≈ 41.5; the fractal-RoPE spectrum roughly halved WikiText perplexity over the run.

SFT chat evaluation (greedy, 20 prompts each)

The failure profile is the expected one for the scale: format learned, knowledge absent. Arithmetic is the one category with real, method-sensitive signal (greedy > sampling).

---

Effort-scaled decoding

effort_decode.py (included) draws several repetition-penalised candidates and returns the best, scored on fluency, self-repetition, length, and how well the reply addresses the current question. The weights are unchanged - this is a decoding change only.

Measured on the SFT checkpoint: 6 prompt stems for repetition, 6 held-out multi-turn topic switches for adherence, 100-120 new tokens.

Topic adherence is 1.0 when the reply fully follows a change of subject and 0.0 when it answers the previous question instead.

python effort_decode.py --ckpt wheeler-dewitt-62m-sft.pt --level high   --prompt "<|im_start|>user
What should I plant in a vegetable garden?<|im_end|>
<|im_start|>assistant
"   --question "What should I plant in a vegetable garden?"

Higher levels cost proportionally more compute: low is a single sample, high draws six and reranks. Pass --previous with the prior assistant turn to penalise replies that echo it.

Caveats: 6 stems and 6 switches establish the effect but are not precise estimates, and the repetition detector used for scoring misses slowly alternating loops, so bigram diversity is reported alongside it.

---

Usage

This is custom-code (not a 🤗 Transformers AutoModel). The modeling code is included in this repo.

import torch
from huggingface_hub import snapshot_download
import sys

path = snapshot_download("Quazim0t0/Wheeler-DeWitt-62M")
sys.path.insert(0, path)
from model import QuazimotoLM, QuazimotoConfig
from spike_tokenizer import SpikeTokenizer

ck = torch.load(f"{path}/wheeler-dewitt-62m-sft.pt", map_location="cpu", weights_only=False)
cfg = QuazimotoConfig(**ck["family_config"])          # rebuilds fractal RoPE + traits
model = QuazimotoLM(cfg); model.load_state_dict(ck["model"], strict=False); model.eval()
tok = SpikeTokenizer(vocab_file=f"{path}/tokenizer.json")

# ChatML - exactly the format SFT was trained on
prompt = ("<|im_start|><|user|>\nWhat is the capital of France?<|im_end|>\n"
          "<|im_start|><|assistant|>\n")
ids = torch.tensor([tok.encode(prompt, add_special_tokens=False)])
out = model.generate(ids, n_new=40, temperature=0.7, top_k=20)
print(tok.decode(out[0, ids.shape[1]:].tolist(), skip_special_tokens=True))

A live demo (chat + a token-by-token Wheeler-DeWitt dynamics dashboard: the emergent-time mode Ψ₀, the constraint residual |H|, and the K minisuperspace modes) is at Quazim0t0/Wheeler-Chat.

---

Intended use & limitations

Intended: research into physics-inspired sequence mixers, fractal positional encodings, and small-model behavior; architecture study; the Wheeler-DeWitt dynamics visualization.

Not intended: any factual, advisory, or production use. It will state false things fluently. It has no reliable knowledge, cannot do multi-digit arithmetic, and does not robustly follow instructions.

Biases/safety: trained on filtered web text; standard web-corpus biases apply. No alignment/RLHF safety tuning beyond light chat SFT.

Citation

@misc{wheeler-dewitt-62m,
  title  = {Wheeler-DeWitt-62M: a quantum-gravity channel mixer with a fractal RoPE spectrum},
  author = {Quazim0t0},
  year   = {2026},
  url    = {https://huggingface.co/Quazim0t0/Wheeler-DeWitt-62M}
}