gemma-4-E4B-frisian

gemma-4-E4B-frisian

A Frisian (West Frisian / Westerlauwersk Frysk) chat model, built on google/gemma-4-E4B-it through continued pretraining (CPT) on Frisian text followed by supervised fine-tuning (SFT) on Frisian instructions. This is the full model of the project: it combines enriched Frisian language ability (CPT) with instruction-following chat behaviour (SFT).

Trained with Unsloth + LoRA.

What this model is

A compact instruction model that consistently answers in Frisian. It is meant as an open, reproducible starting point for Frisian language models — not a production-ready system.

Results

Measured with a six-question smoke test; language identification via fastText lid.176 (mean fy probability across the answers):

The progression shows both stages contribute: SFT anchors the chat behaviour, CPT deepens the language (richer vocabulary, fewer neologisms, more natural Frisian).

Training setup

Stage 1 — CPT (Rickkosse/gemma-4-E4B-fy-cpt)

• Data: Rickkosse/fy-pretraining-mix-v2 — ~254M tokens of Frisian text (Frisian Wikipedia, web text, and filtered back-translation NL→FY)
• LoRA r=64 on attention + MLP (q,k,v,o,gate,up,down); embeddings not trained (Gemma's 262k multilingual vocab already covers Frisian)
• 1 epoch, ~124k packed blocks of 2048 tokens, effective batch 128, lr 5e-5 cosine

Stage 2 — SFT (this model)

• Data: ~2.9k instructions translated into Frisian (dolly/alpaca/no_robots)
• Gemma's native chat template with system role; completion-only loss
• LoRA r=16, 3 epochs, lr 2e-4

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Rickkosse/gemma-4-E4B-fy-cpt-sft"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content": "Do bist in behelpsume AI-assistint. Antwurdzje altyd yn it Frysk."},
    {"role": "user", "content": "Wat is de haadstêd fan Fryslân?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Limitations

Honest about what this model does not do well:

• No alignment/safety training. Treat the output as that of a small open base model.

• Small model. 4B class; do not expect deep reasoning.

Provenance and license

Base model: google/gemma-4-E4B-it, under the Gemma license. This derivative model is subject to the same terms. Training data partly derived from Wikipedia (CC-BY-SA) and public instruction datasets.

Citation / contact

Part of an open-source project to build Frisian language models. Built by Rickkosse. Feedback welcome.