Fredu-1.7B-Instruct

Fredu 1.7B Instruct

A Maltese-specialised model built from utter-project/EuroLLM-1.7B in two stages on a single consumer GPU (RTX 5060 Ti, 16 GB).

It is fluent in Maltese but hallucinates and does not always reply with anything related to the prompt. It's able to translate english sentences into a related sentence in Maltese. It's quite bad at it but the Maltese translation is related somehow.

How to run

Ollama

ollama run hf.co/Cabbache/Fredu-1.7B-Instruct

The prompt template and sampling parameters are stored in the repo, so nothing else is needed. Pick a build with a tag:

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Cabbache/Fredu-1.7B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Cabbache/Fredu-1.7B-Instruct",
                                             dtype="bfloat16", device_map="auto")

q = "X'inhu l-Kunsill Lokali f'Malta?"
ids = tok(f"Mistoqsija: {q}\nTweġiba:", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=200, temperature=0.3, do_sample=True)
print(tok.decode(out[0], skip_special_tokens=True))

There is no chat template. Use the Mistoqsija:/Tweġiba: framing above, which is what the model was trained on. A low temperature helps: what factual knowledge it has is held with very little margin and does not survive sampling at 0.7.

What was done

Both stages were full fine-tunes (all 1.657B parameters), bf16 weights and gradients with 8-bit AdamW and gradient checkpointing, ~14 GB VRAM.

Prompt format

The model was trained on this exact framing, with loss computed only on the answer:

Mistoqsija: {question}
Tweġiba: {answer}

It reads informal Maltese and replies in correct Maltese

35% of training questions had their diacritics stripped (ħ→h, ġ→g, ċ→c, ż→z, so għ→gh) while every answer was left correct. Because loss lands only on the answer, the model is shown informal spelling and never rewarded for producing it.

Q: X'inhu n-numru medju ta' sighat ta' rqad li persuna ghandha tiehu?
A: In-numru medju ta' sigħat ta' rqad li persuna għandha tieħu huwa madwar
   7.5 sigħat kuljum.

Note sighat → sigħat, ghandha → għandha, tiehu → tieħu.

Evaluation

Perplexity on held-out documents, by register, against the untrained base:

A 57-item Malta factual eval (greedy decoding, an upper bound on knowledge):

Licence

Do whatever you want with it