LOLM-304M

LOLM-304M — Latent Order Language Model

LOLM is a hybrid Transformer–SSM architecture that separates surface token prediction from latent state tracking. This is the 304M research checkpoint behind the published WikiText-103 results.

Authors: Bryan Leonard & Brandyn Leonard · Qira LLC · Provisional patent 64002166 Code: https://github.com/TheArtOfSound/LOLM

TL;DR

At 304M parameters, LOLM reaches 68.37 PPL on WikiText-103, beating Pythia-410M (142.93 PPL) by 52% at matched compute with 26% fewer parameters. It is a base language model — evaluate it on perplexity and representation quality, not on instruction-following (it is not instruction-tuned).

Architecture

Each token flows through five parallel streams that converge via learned fusion:

o_t = g · LN(W_h·h_t) + (1−g) · LN(W_z·z_t) + W_m·m_t + W_r·r_t

Results

Inference-time ablations confirm every component contributes (regime, SSM, gate, memory). See the paper for full tables and the 1.57B-scale comparison.

Usage

LOLM is a custom architecture — load it with the LOLM code, not transformers:

git clone https://github.com/TheArtOfSound/LOLM.git && cd LOLM
pip install -r requirements.txt

import torch, tiktoken
from lolm.config import load_config
from lolm.model import LOLM

cfg = load_config("configs/scale/300m_lolm_full_tpu.yaml")
model = LOLM(cfg.model)
ckpt = torch.load("ckpt_26000.pt", map_location="cpu")
model.load_state_dict(ckpt["model"], strict=False)
model.eval()

It exposes its own latent dynamics every step (out.gate_values, out.regime_probs) — used by the NFET runtime governor in the lolm-bridge workspace.

License

Released under the LOLM Community License Agreement v1.0 (see LICENSE): free for academic research, education, and personal/non-commercial use; commercial license required for qualifying entities.

Citation

@article{leonard2026lolm,
  title  = {LOLM: Language Modeling Beyond the Surface with Hybrid
            Transformer-SSM Latent Order Fields},
  author = {Leonard, Bryan and Leonard, Brandyn},
  year   = {2026},
  note   = {Qira LLC. Provisional patent application No. 64002166.}
}