ilo-toki-1.4-MiLMMT-46-1b-merged

ilo toki 1.4 — MiLMMT-46 1B

A translator between Toki Pona and English, Russian and Vietnamese. Small enough to run on a phone: it powers ilo toki, which does all of its translation on device.

This repository holds both the merged weights and GGUF builds.

What changed in 1.4

Two things, and the second one matters more than the first.

Training. group_by_length is off. The datasets in the mix have different length profiles, so grouping similar lengths into a batch also grouped by source; mixing them was meant to improve the model's fit across the whole mixture. Measured on the project's own held-out split it is a small, consistent gain: BLEU is up on five of six directions against 1.3.

The merge is fixed, and this is the larger change. Releases 1.1 through 1.3 were merged with the output head untied from the trained embeddings, on the belief that the head had never seen them during training. That belief was wrong for the mechanism this adapter uses. The adapter trains individual embedding rows through PEFT's trainable_token_indices, and PEFT explicitly finds the weights tied to the embedding matrix and puts a tied adapter on each — so the output head read the trained rows throughout training, and untying handed the merged model a head it had never used.

Merging correctly changes 119 of 153 probe answers and fixes most of what earlier releases were known for:

It also makes the files about 23% smaller: llama.cpp stores one embedding matrix for a tied model and two for an untied one, so Q8_0 is 0.996 GiB where 1.3 was 1.29.

If you are fine-tuning this base yourself with trainable_token_indices, do not untie the head at merge time. The check is one line: build the peft model and look for TrainableTokensLayer — there should be two, on embed_tokens and on lm_head.

Prompt format

The model keeps the prompt format of its base, and there is no chat template — do not wrap the input in one.

Translate this from Toki Pona to English:
Toki Pona: jan li moku e kili
English:

The translation follows the final <target language>: line and ends at the model's end-of-generation token. Either side can be the source:

Translate this from Russian to Toki Pona:
Russian: Я тебя люблю.
Toki Pona:

Language names are written out in full — Toki Pona, English, Russian, Vietnamese. Getting the format wrong does not fail loudly: the model keeps producing fluent text while silently ignoring the requested target language.

Which file to use

Running it

llama-completion -m ilo-toki-1.4-MiLMMT-46-1b-Q8_0.gguf --temp 0 --top-k 1 -no-cnv \
  -p "Translate this from Toki Pona to English:
Toki Pona: jan li moku e kili
English:"

Greedy decoding is what this model is meant to be run with. There is one right answer per input, and sampling only ever walks away from it.

How it was built

A LoRA adapter (NetherQuartz/ilo-toki-1.4-MiLMMT-46-1b, revision 48f57ca984) trained with TRL SFT — rank 64, targeting the attention and MLP projections, plus 15 764 individual embedding rows through PEFT's trainable_token_indices — merged into MiLMMT-46-1B-v0.1 with the output head left tied, and quantized with llama.cpp. The checkpoint is the one at 17 500 steps, the minimum of the validation loss.

Training data

Alongside tok↔x pairs the mix includes x↔y pairs between the natural languages, meant to keep their generation fluent without crowding out the pairs where Toki Pona is one side.

Known limitations

• Short inputs still acquire invented specifics. soweli lili li lape lon tomo into Russian can produce a mouse that the sentence never mentioned.
• Unmarked features get a default rather than a reading. Toki Pona marks neither number nor tense; both readings are valid, but the model picks rather than infers from context.
• Transliteration of unseen names is legal but not conventional. Fukuoka comes back as a well-formed toki pona word that is not the attested one. Names the training data has seen are right.

Licence

Gemma Terms of Use, inherited through the base model.