ollin-tiny

Ollin-tiny

"Ollin" — Nahuatl for movement / change.

A 4.5M-parameter proof-of-concept language model combining delta-rule linear attention (KDA) — constant-memory attention state, in the style of DeltaNet / Kimi Linear — with a sparse Mixture-of-Experts feed-forward layer, trained from scratch on CPU. Weights are also released after MXFP4-style 4-bit microscaling quantization.

What this model is

A small, honestly-reported proof of concept demonstrating that constant- memory attention and 4-bit block quantization work end-to-end in a from- scratch implementation, with every claim measured directly on this checkpoint. Full methodology and raw numbers: see RESULTS.md and PAPER.md in the GitHub repo.

What this model is NOT

• Not a model competitive with LLaMA, GPT, or any production LLM. No such comparison was run.
• Not trained on internet-scale data — trained on Tiny-Shakespeare (~1M characters), character-level tokenization, single run.
• Not a general-purpose assistant. It free-generates Shakespeare-like text and nothing else; it was never instruction-tuned or aligned.

Intended use

Research and education: studying constant-memory attention mechanisms and low-bit quantization on a codebase small enough to read end to end. Not intended for production text generation or any application requiring factual accuracy or safety guarantees.

Measured results (fp32 checkpoint)

• Validation perplexity: 4.83 (down from 34.75 at initialization), over 3000 training steps / 1871s wall-clock on CPU.
• Params: 4,510,224.

Measured results (MXFP4-quantized checkpoint)

• Compression vs fp32: 7.53x (0.531 bytes/weight, packed).
• Validation perplexity: 4.96 (+2.7% vs fp32) — fake-quant evaluation (quantize→dequantize numerically; no dedicated 4-bit inference kernel is included).

Training data

Tiny-Shakespeare, public-domain text, standard small benchmark for from-scratch language-model demos. Character-level tokenization, vocab size 65.

Update: WikiText-103 scaling + architecture ablations

A follow-up experiment trained a larger (39.7M param) variant on real WikiText-103 (101M tokens, word-level, 50K vocab): validation perplexity fell from 37,287 to 398 over 320 steps (~1% of one epoch, CPU only). This is explicitly not comparable to published WikiText-103 baselines (AWD-LSTM, Transformer-XL, etc.), which train full epochs for days on GPU — it demonstrates stable training on real word-level data at this vocabulary size, not competitive performance. The checkpoint published here is still the original tiny-Shakespeare model; the WikiText-103 run and four architecture ablations (MLA, cellular-automata state evolution, weight-tied "fractal" recursion, a lightweight hypernetwork gate) are documented with full numbers, including two real future-leakage bugs found and fixed, in RESULTS.md §5 and PAPER.md §5 of the GitHub repo.

Limitations

See RESULTS.md in the repo for the full list. In short: single run, single seed, small data, no baseline comparison, sequential (non-chunked) attention implementation, fake-quant evaluation rather than a real low-bit kernel.

License

MIT.