nano-vowel

nano-vowel

Restores the vowels to a vowel-less word — wrld → world, dsmvwld → disemvoweled, brd → bread. Vowel placement lives only in a learned lexicon, so a rule-only script has nothing to compute; the model recovers the most likely word a consonant skeleton came from. A ~1M-parameter (1,016,960) byte-level transformer.

wrld   => world
schl   => school
frnd   => friend
brd    => bread     (not bird — the model picks the common word)
thn    => then      (not than / thin)
ppl    => people    (not apple)

It runs on a CPU in milliseconds and was trained entirely on code-generated data — no scraping, no labelling, no distillation. Weights + a self-contained inference file are here and on Hugging Face.

📄 Technical report (PDF) · 📝 Full writeup (gist)

Why a model and not a script

Deleting the vowels (world → wrld) is a trivial one-line script. Inverting it is the hard direction, and it is many-to-one: brd could be bread, bird, beard, board, or broad; thn could be than, then, or thin. There is no rule — the only way back is knowing which word is more likely. A 5-line script has no such knowledge. nano-vowel packs a frequency-weighted vocabulary into its weights and recovers the intended word.

Benchmark (held-out, seed 987654321, N=4000)

Exact-match accuracy. The model is compared to two deterministic scripts: a naive dictionary (no frequency knowledge — picks the alphabetically-first candidate) and a frequency dictionary (picks the most common candidate).

The collision slice is skeletons shared by 2+ words — the part that needs a prior. The model crushes the naive script (65.4% vs 29.2%): the must-beat-a-script result. It ties the frequency dictionary (65.4% vs 66.3%) because the task's ceiling literally is frequency-argmax, and the dictionary has perfect frequency knowledge. Honest framing: a frequency-ranked dictionary matches nano-vowel; the model's edge is doing it as a single 4 MB byte-level net with no lookup table — the same recovery on raw bytes, runnable anywhere. The headline win is over the script anyone would actually write (the naive one).

Reproduce: python eval_nano_vowel.py --n 4000.

Where it breaks (out-of-distribution)

nano-vowel's prior is its ~480-word training vocabulary. On words it never saw, it has no basis to place vowels:

So it recovers known words and fails on unknown ones — the expected ceiling of a 1M model whose knowledge is a fixed lexicon, reported rather than hidden. There is also an irreducible ceiling: ultra-short skeletons like t (at/eat/it/out/to) carry too little signal for any method to fully resolve.

Use it

pip install -r requirements.txt
python modeling_nano_vowel.py            # demo

from modeling_nano_vowel import load, restore
m = load("model.safetensors", "config.json")
restore(m, "wrld")     # -> "world"
restore(m, "dsmvwld")  # -> "disemvoweled"

How it was trained

Code-generated data: sample a real word from a fixed, frequency-ordered ~480-word vocabulary (Zipf-weighted so common words dominate), delete its vowels to make the input. Because we start from the word, the label is correct by construction. SFT with the prompt masked so only the target word + newline EOS is supervised. ~1M-param byte-level transformer (RMSNorm, RoPE, GQA, SwiGLU), 12k steps, AdamW, cosine LR. Full recipe in TRAINING.md.

Files

• modeling_nano_vowel.py — self-contained model + load() / restore() (torch + safetensors only).
• data_vowel.py — the code data generator (shared by train and eval).
• eval_nano_vowel.py — the model-vs-script benchmark (naive + frequency dictionaries).
• test_nano_vowel.py — labels-correct / collision-slice / determinism / published-weights regression.
• model.safetensors, config.json — weights + architecture.
• report/nano-vowel-report.pdf — the technical report.
• TRAINING.md — reproduction recipe.

License

MIT. Built by Vuk Rosić.