MIRAI-GlassBox-LLM

MIRAI — a 1.08B glass-box language model built on signal theory

A from-scratch 1.08B-parameter language model built to be a glass box: every write to the residual stream decomposes exactly — to 5×10⁻⁷ relative error — into 129 named spectral atoms whose time constants were fixed before training. Nothing is reverse-engineered after the fact. You read the computation off the forward pass the way you read a voltage off a circuit node, and the shipped instrument panel shows all of it, live, for every generated token.

The temporal layer is a fixed-pole linear filter bank, not learned attention over time: each of the 129 atoms is a damped complex exponential B_f·e^(a·k)·(cos ωk + i sin ωk) — the impulse response of a first-order resonator — applied by causal convolution. The poles are identical in all 10 layers, strictly inside the unit circle (|p| ≤ 0.995, so stable by construction), spanning time constants from 200.5 down to 13.0 tokens. Because those coordinates are chosen in advance rather than discovered, every named part has a meaning that does not depend on what training happened to do with it.

Where this stops being enough is stated plainly below and is the subject of the paper: seeing the computation exactly is not the same as knowing which parts matter.

Released with both checkpoints, the training code, the interpretability tooling, and a hash-pinned reproduction package for every number quoted below.

⚠️ This model is UNDERTRAINED — read this first

Do not expect good text out of it. MIRAI saw 25B tokens (23.1 per parameter) in a single compute-limited run, and the loss was still descending when the budget ran out — training was stopped by cost, not by convergence. A 1.08B model needs far more than this to saturate.

Concretely, expect:

• weak world knowledge — it will state wrong facts confidently, and it has no knowledge cutoff worth quoting because it never learned enough to have one
• short coherence — the attention window is 256 tokens; long-range consistency degrades quickly
• repetition and drift in longer generations
• an instruction-tuned checkpoint that follows the chat format well while still being limited by the base model's knowledge — the format is learnable at this scale; the knowledge is not
• no safety tuning of any kind — no alignment, no RLHF, no content filtering

What it is for. MIRAI is a research instrument for studying interpretability. Its value is that every step of its computation can be read exactly, and that its own transparency claims can be tested and falsified — not that it writes well. Judge it on the measurements below, not on its prose.

If you want a model that produces good text, this is not it. If you want a billion-parameter model whose every internal step you can read and verify, that is exactly what this is.

Model summary

Loss was still descending at the compute budget. This is a research model for studying interpretability, not a state-of-the-art Transformer replacement and not trained to saturation.

Checkpoints

⚠ Read this before loading

comp_map and rare_mask in cache/ are registered persistent=False, so they are not inside the checkpoints. Loading without them succeeds silently and yields a model with no rare-token head — wrong outputs, no error. Always assert missing_keys == [].

Quickstart

Requires torch and tokenmonster (pip install torch tokenmonster). The tokenizer vocabulary ships in this repo, so nothing is fetched at runtime:

import tokenmonster
vocab = tokenmonster.load("tokenizer/englishcode-32000-consistent-v1.vocab")
ids = vocab.tokenize("The capital of France is")

import sys, torch; sys.path.insert(0, "REBUILD")
from mirai5_clean import Mirai5, PRESETS

cm = torch.load("cache/compcode_map_englishcode-32000-consistent-v1_8192.pt",
                map_location="cpu", weights_only=False)
rm = torch.load("cache/rare_mask_englishcode-32000-consistent-v1_q60.pt",
                map_location="cpu", weights_only=False)
if isinstance(rm, dict): rm = rm.get("rare_mask", rm.get("mask"))

model = Mirai5(32000, **PRESETS["b1"], comp_map=cm, rare_mask=rm.view(-1).bool())
ck = torch.load("checkpoints/m5_25B_ddp1/final.pt", map_location="cpu", weights_only=False)
sd = {k.removeprefix("_orig_mod."): v for k, v in ck["model"].items()}
missing, unexpected = model.load_state_dict(sd, strict=False)
assert not missing, missing          # the trap above
model.eval()

Interactive chat with the full instrument panel:

python bin/mirai_glassbox.py --ckpt checkpoints/m5_ft_25b4k_surgical

For every generated token it prints which positions attention read, the spectral atom activations and their exact contributions, the router gate, both branch writes, the readout geometry against the 32,000 token codes, and a per-atom causal ablation. Each quantity carries an evidence tag (DIRECT / DERIVED-EXACT / DERIVED / PROBE / HEURISTIC / INTERVENTION), and the panel self-verifies bitwise against the executed arithmetic on every run.

Interpretability

For a fixed forward pass the selective gate is constant, so each layer's spectral write is exactly

Δwave  =  Σ_f c_f  +  b          (f = 1..129 named atoms)

an algebraic identity rather than a fitted approximation.

The last two rows are the point, not a disclaimer: the computation is exactly decomposable, yet exact contribution predicts causal importance only modestly. Observability and causal faithfulness are separate properties, and this model lets both be measured. The paper develops that argument — paper/MIRAI_paper.pdf.

The second instrument: how compressible is the computation?

bin/mirai_glassbox.py shows one forward pass in full detail. bin/skeleton_joint_matched_v2.py asks a different question — how much of the model's learned ability survives when its internals are forced into a restricted, describable form? Three intermediate computations are restricted simultaneously, and the loss is measured against the unigram floor: attention pruned to the top-K edges per query per head, values projected onto a rank-R basis, and the 129 filters snapped to a coarse (decay, frequency) taxonomy.

The 13.9% row is the informative one. Compressing the pole bank to 18 classes nearly destroys the model, but keeping the slow poles exact and coarsening everything else recovers 99.2% — so the structure that matters is concentrated in the long-time-constant filters, while the fast ones are largely interchangeable. That is a measured statement about where the model's capacity lives.

This is diagnostic, not a reconstruction: the full quantities are computed by the intact trained weights before being restricted, so no inference speedup is demonstrated.

What you can actually inspect

bin/mirai_glassbox.py opens an interactive chat and, for every generated token, prints the panels below. Each is tagged with its evidence class so measurement is never confused with inference.

The head census is the concrete answer to "which head is doing what?" — it classifies every head on the current input by where its attention mass went (induction heads by attending just after an earlier copy of the current token, local heads by the last four positions, anchor heads by position 0). It is a per-input classification, not a permanent label on the head.

The panel self-verifies bitwise against the executed arithmetic on every run: if the instrument and the model ever disagree, it says so instead of printing a plausible number.

Repository layout

Reproduce

experiments/frozen_segments.npz (15 MB) carries the exact evaluation, calibration and reference-fit token segments, so the headline experiments run without the original corpus.

python verify_repo.py                       # check every file hash first
python bin/skeleton_joint_matched_v2.py     # structured restriction    -> 94.7% retained
python experiments/bigram_floor.py          # stronger reference        -> 92.6% retained
python experiments/instrument_fidelity.py   # instrument self-checks    -> bitwise
python bin/ft_damage_attrib.py              # fine-tuning localisation  -> 53% from 33,792 params

Every script refuses to run on a checkpoint, source, artifact or data-segment hash mismatch.

Limitations

Undertrained: compute-limited single run, stopped by budget rather than convergence, with validation loss still descending. Output quality is well below what this parameter count could reach. Decay and frequency are confounded by construction. The poles are tied to the sampling step, so the model is not sample-rate invariant. Modality breadth is not established — one 148M audio pilot only. English only. No safety tuning, alignment or content filtering was applied; treat outputs accordingly.

Citation

@misc{dinca2026mirai,
  title  = {{MIRAI}: A Billion-Parameter Glass-Box Model Where Exact Observability
            Is Not Causal Faithfulness},
  author = {Dinca, Marius},
  year   = {2026},
  note   = {Model weights, interpretability tooling and reproduction package},
  url    = {https://huggingface.co/Puddings22/MIRAI-GlassBox-LLM}
}

Apache-2.0: commercial use, modification and redistribution are permitted provided the copyright notice is preserved. Academic citation is requested, not legally required.