ilo-toki-1.3-MiLMMT-46-1b-merged

ilo toki 1.3 — MiLMMT-46 1B

A translator between Toki Pona and English, Russian and Vietnamese. Small enough to run on a phone: it powers ilo toki, which does all of its translation on device.

This repository holds both the merged weights and GGUF builds, so there is one place to look rather than a repository per format.

Version 1.3 replaces ilo-toki-1.1-MiLMMT-46-1b-merged. See what changed and, before relying on it, known limitations — several of them are inherited rather than new.

Prompt format

The model keeps the prompt format of its base, and there is no chat template — do not wrap the input in one.

Translate this from Toki Pona to English:
Toki Pona: jan li moku e kili
English:

The translation follows the final <target language>: line and ends at the model's end-of-generation token. Either side can be the source:

Translate this from Russian to Toki Pona:
Russian: Я тебя люблю.
Toki Pona:

Language names are written out in full — Toki Pona, English, Russian, Vietnamese. Getting the format wrong does not fail loudly: the model keeps producing fluent text while silently ignoring the requested target language.

Toki Pona is written in lower case; capitalization in the input is not something the model expects. Terminal punctuation is optional and barely changes the answer.

Which file to use

The quantizations sit unusually close together because the 262k-token embedding matrix is about a third of the model and quantizes the same way in all of them. Q8_0 therefore costs only 0.05 GB more than Q6_K and 0.35 GB more than Q4_K_M, which is why the app ships it: on a phone the difference between these files is small, while the difference between fitting in RAM and not is enormous.

Running it

With llama.cpp:

llama-completion -m ilo-toki-1.3-MiLMMT-46-1b-Q8_0.gguf --temp 0 --top-k 1 \
  -p "Translate this from Toki Pona to English:
Toki Pona: jan li moku e kili
English:"

Greedy decoding is what this model is meant to be run with. There is one right answer per input, and sampling only ever walks away from it.

With transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "NetherQuartz/ilo-toki-1.3-MiLMMT-46-1b-merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "Translate this from Toki Pona to English:\nToki Pona: jan li moku e kili\nEnglish:"
inputs = tokenizer(prompt, return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=64, do_sample=False)[0]))

How it was built

A LoRA adapter (NetherQuartz/ilo-toki-1.3-MiLMMT-46-1b) trained with TRL SFT — rank 64, targeting the attention and MLP projections, plus 15 761 individual embedding rows through PEFT's trainable_token_indices — merged into MiLMMT-46-1B-v0.1 and quantized with llama.cpp. The checkpoint is the one at 15 000 steps, taken before the validation loss turns rather than at the end of training.

The base is a 46-language translation model from Xiaomi Research, so the fine-tune starts from a model that already translates rather than from a general purpose one.

Training data

Alongside tok↔x pairs the mix includes x↔y pairs between the natural languages, at half the rate of 1.1, meant to keep their generation fluent without crowding out the pairs where Toki Pona is one side.

What changed in 1.3

Measured against 1.0 and 1.1 over 103 prompts in both directions across the three languages, Q8_0 against Q8_0, greedy throughout.

• toki pona is no longer answered about as another language. 1.1 turned mi sona e toki pona into «I know Russian» and sina sona ala sona e toki pona into «Do you know Russian?». Both are right again.
• Invented specifics are mostly gone. 1.1 rendered jan li moku e kili into Vietnamese as «people in the state of Oregon eat delicious food», and put a sleeping animal «in the kitten room». 1.3 says what the sentence says.
• Terminal punctuation moves the answer far less. Of fourteen sentences tried bare and with their final mark, 1.1 changed its answer on five and 1.3 on one.
• Only . is dropped from the source during training now. Dropping ? and ! had made a declarative source map to an interrogative target, which is label noise rather than augmentation.

Known limitations

• ala is sometimes reversed. Two of ten negation probes come back meaning the opposite: jan li lape ala gives «someone is sleeping», mi pilin ike la mi moku ala gives «when I feel bad then I eat too much». 1.1 gets the same two wrong, so this is inherited rather than new — but a negation that reads fluently and means the opposite is the worst thing here, and it is the first target of the next round.
• «Что ты делаешь» without a question mark comes back as o tawa. With the mark, and in English either way, it is right.
• la is read as a conditional where the relation is causal or temporal: ilo mi li pakala la mi ken ala toki tawa sina gives «if my computer breaks down then…» rather than «because». 1.0 handled this better; every version since has not.
• Unmarked features get a default rather than a reading. Toki Pona marks neither number nor tense, and bare mi usually comes back as «we», unmarked verbs as past. Both readings are valid — mi mute is optional — but the model does not choose by context, it just picks.

Licence

Gemma Terms of Use, inherited through the base model.