gpt-alpha-bg-restorer

gpt-alpha-bg-restorer

Two small character-level GPTs that turn shlyokavitsa, Bulgarian typed on a Latin keyboard, back into Cyrillic. Trained from scratch, no pretrained initialization. Both run offline on a CPU, and both ship as WebAssembly, so the text never leaves the device.

chetesh li knigi              ->  четеш ли книги
blagodarya za pomoshtta       ->  благодаря за помощта
sofiya e stolica na balgariya ->  София е столица на България

A lookup table cannot undo this, because the mapping is many-to-one in the wrong direction and information is genuinely destroyed by the keyboard. a spells both а and ъ (dobar → добър) with nothing in the input to say which. sh is usually ш but sometimes с+х across a morpheme boundary. sht collapses to the single letter щ, ch to ч, ya to я, so the alignment is not one-to-one either. The model reads the Latin string and writes Cyrillic one character at a time, deciding each letter from context.

Try it before downloading anything

• A Space on this Hub runs the 3.16M model with constrained decoding on: glassbox/shlyokavitsa-restorer. The weights are in the page, so nothing you type leaves your browser.
• The same model with its attention and per-character probabilities exposed, at https://restorer-interp-lab.iksnerd.workers.dev: type into it and watch each Cyrillic letter get decided. The 4.73M model has its own demo at /restorer-scaling, and there is a post-mortem of a failing run.

The two checkpoints

They are for different jobs rather than being two sizes of one, which the results below make concrete. Both use the same 60-character vocabulary (Latin input, Cyrillic output, the = separator and a newline terminator), the Llama-style stack the rest of this project uses (RoPE, RMSNorm, SwiGLU, weight tying), and the same training method.

Usage

The .wasm builds are the dependency-free path: they embed their own weights and expose one function, in a browser or in Node, with Go's standard wasm_exec.js glue (included here).

// Node 18+: node run-restorer.mjs
import { readFile } from 'node:fs/promises'
await import('./wasm_exec.js') // defines globalThis.Go

const go = new Go()
const ready = new Promise((resolve) => (globalThis.onRestorerReady = resolve))
const { instance } = await WebAssembly.instantiate(
  await readFile('./restorer.wasm'),
  go.importObject,
)
go.run(instance)
await ready

console.log(globalThis.restore('chetesh li knigi')) // четеш ли книги

restore_big.wasm is the same with restoreBig() and onRestorerBigReady. The small build additionally exposes restoreTrace(text), which returns the restoration plus per-step attention weights and top-5 character probabilities as JSON, the data the interpretability lab renders.

The .pt checkpoints are plain PyTorch state dicts carrying their own config. You mostly do not need them. If you want the model in torch anyway, bg-eval loads both and scores the restoration table below (python restore_eval.py --checkpoint restore_big.pt). The same weights are here as safetensors, which needs neither the training code nor pickle:

from safetensors.torch import load_file

weights = load_file("model.safetensors")          # the 20-word model; small/ holds the other
weights["token_embedding_table.weight"].shape     # torch.Size([60, 256])

config.json beside it carries the resolved architecture, every toggle spelled out. The head is tied to the token embedding and is therefore stored once, under the embedding's name.

Results

Exact-match restoration for the 4.73M model: the whole phrase correct, character for character, by phrase length. Scored on the shlyokavitsa pair set built from Bulgarian Wikipedia, which the model never saw. n = 200 phrases per length, greedy.

Constrained decoding is the default and is what you should use; the section below explains why it costs nothing. Single run, not a multi-seed average; the in-corpus figures below were measured on CUDA, these on CPU reference numerics over frozen weights.

In-corpus, the same model reaches 0.92 at 20 words on pairs generated from the text distribution it trained on (0.98 / 0.97 / 0.94 at 5 / 10 / 15, falling to 0.75 at 24 and 0.23 at 28 words, past the trained frontier). The gap to 0.720 on Wikipedia is the interesting part: Wikipedia is dense with proper nouns, place names and abbreviations, which is exactly where the romanization's irrecoverable collisions live. Two real failures from that run: v sasht kato → gold в сащ като (САЩ, the abbreviation for the USA), predicted в същ като ("same"), because sasht spells both; and vav faram hempshir → gold във фаръм хемпшир, predicted във фарам хемпшир, the а/ъ coin-flip inside a transliterated place name. Neither is recoverable from the input. Use the in-corpus number for text like the training distribution and this table for arbitrary Bulgarian.

The 3.16M sibling does not degrade, it stops

Same split and protocol, plain greedy decoding, so the row to compare against is the 4.73M's unconstrained one:

Through 10 words the smaller model is behind but working. At 15 it drops to 0.055 and by 20 it never gets a phrase right, 200 for 200. That is the block-348 context and the ~6-word trained frontier running out together, not a gentler version of the same curve.

uv run restore_eval.py --checkpoint restore_ckpt.pt --per-length 200

n = 200 per length, greedy, CPU reference numerics over frozen weights, 2026-08-03.

Versus a 2.6B general Bulgarian model

Same Wikipedia test split, 5-shot prompting, greedy, exact match:

Bulgarian-specialized instruct models 550× and 900× this size get about one phrase in ten at three words, and a residue rather than nothing at the long end: both keep 0.005 at 20 words and reach 0 only at 25. Not a fair fight in either direction, since neither was trained for this and both got five examples, while this model does nothing else and had a curriculum. The point is that shlyokavitsa restoration does not fall out of Bulgarian language modelling at scale. It is a character-level, lossy, alignment-heavy transduction task, and a subword generalist is the wrong shape for it.

Constrained decoding, and why it is free

The romanization is a free-monoid morphism, so every valid restoration is a segmentation of the Latin input into fragments with exactly one Cyrillic pre-image each. The set of legal next characters is therefore computable at each position, and masking the decoder to it costs zero parameters and zero retraining.

It works because the errors split into two kinds, and the split is measurable without a label: a prediction either romanizes back to its own input or it does not.

• structural — the decoder lost the alignment and dropped, duplicated or invented characters (на на групата → на на на групата). Constraining makes these unrepresentable: the rate goes from 0.085 to 0 at 20 words, and from 0.670 to 0.040 at 25.
• ambiguity — a legal reading of the input, wrong choice. Almost all а/ъ. Constraining cannot help and should not: the information is genuinely gone.

Structural errors are what destroys the long end, which is why the gain concentrates there: +0.155 exact match at 25 words and an 8.5× lower character error rate (0.1113 → 0.0131), against +0.010 at 3 words where there was nothing to fix. The 0.040 residue at 25 words is the decoding policy being permissive on purpose: it accepts the spellings people actually type, not only the ones the training romanization produced, so a phrase can come back as a legal reading of its input that is not the reference's. The alignment is intact; the spelling choice is not.

The clearest case is one users hit by accident, pasting more text than the context window holds. Feeding the 3.16M model (block size 348) a 359-character input makes plain decoding collapse into a stutter while the constrained decoder stays coherent, because the mask makes the degenerate output unrepresentable rather than merely unlikely:

input        kak se kazvash dnes  ×18   (359 characters, past the 348 window)

plain        с к к какс каккакс к к к какс какс какс к к к какс какс как
constrained  как се казваш днес как се казваш днес как се казваш днес как

Nothing about the weights changed between those two lines. Both the WebAssembly builds and the Python CLI expose it (restoreConstrained, restore.py --constrained).

Training, and two findings behind it

Supervised on synthetic pairs in the format latin=cyrillic⏎, generated by romanizing Bulgarian prose and learning the inverse. The curriculum is self-expanding: the model trains on short phrases and the phrase length grows whenever it clears the current length, so it is never asked to restore sentences it has no chance at yet. The larger model ran 60k iterations on a Modal A10G. The training corpus is not redistributed here (it is assembled from third-party Bulgarian text), but the pair generator is deterministic given a corpus, so the data is reproducible rather than shipped.

Character level beat six times the parameters. The obvious alternative is to warm-start a subword (BPE) model that already knows Bulgarian. Tried against a from-scratch control of the same architecture, every BPE variant lost the same way: a 29.37M BPE model that provably contained the missing vocabulary still stalled at two-word phrases and produced fused nonsense (az obicham balgarija → аз обикалгари). Noisy Latin input shatters into subword tokens with no stable relation to the Cyrillic target's tokens, so knowledge stored behind the tokenizer cannot flow through the interface.

The first model's six-word ceiling was a budget, not a limit. It looked structural and was not: the attention window is ~347 characters (≈ 60 words), so nothing was being forgotten, and swapping the position encoding for ALiBi bought nothing (with more iterations RoPE and ALiBi both climbed to 11 words and tied). More capacity and a much longer run moved it to 20. What did not move is extrapolation past whatever frontier the model was trained to.

Limits

• The а/ъ ambiguity is inherent. Where the Latin input destroyed the distinction the model is guessing from context. It commits to the more likely reading; sometimes that is wrong.
• Past the trained frontier, quality falls off a cliff (0.92 at 20 words in-corpus, 0.23 at 28). Feed it sentences, not paragraphs.
• Prose is what it saw. The pairs are generated from written Bulgarian, so heavy slang, proper nouns and mixed-language text are outside what was measured.
• Documented failure modes. Traced on a failing input, the model attends to exactly the right source character and still decodes the wrong one (j → й at 99% attention but 42% confidence, a knowledge gap rather than an attention failure), skips a word, and re-reads the same word twice. All three are visible in the interpretability lab linked above.
• Not an instruction-following model, not a translator, and not a spell checker. It does one transduction.

Verification

The 3.16M port is verified byte-for-byte identical to the PyTorch original on the port's harness phrases, re-checked after the decode was rewritten to use a KV cache; that cached decode is in turn verified output-identical to a full recompute across a 40-phrase corpus, at 7–34× the speed in-browser. The instrumented forward pass that produces the traces is checked bit-identical to the uninstrumented one, so what the lab displays is what the model computed.

Files

Part of a family

<!-- BEGIN generated: family (uv run python datatools/gen_publishing_doc.py) -->

Six models from one project on small models doing narrow tasks, these among them:

• gpt-alpha-bg-91m - the 91M flagship: a fluency judge that ties a frontier model on Bulgarian error/correction pairs
• gpt-alpha-bg-restorer - you are here

The other four appear under huggingface.co/glassbox as each ships.

<!-- END generated: family -->

Findings, the interpretability lab and the full model roster: restorer-interp-lab.iksnerd.workers.dev.

License

MIT.