Qwen3.5-4B-Finnish

Qwen3.5-4B Finnish (experimental)

A Finnish instruction fine-tune of Qwen/Qwen3.5-4B. This is a merged full checkpoint (LoRA baked in), not a PEFT adapter and not a Finnish continued-pretrain like Poro 2, Ahma, or Viking. The base is a latest-general multilingual model (Apache-2.0).

Thinking is off. The vision tower was frozen; this release is text-only. There is no DPO in this checkpoint (TRL 1.12 needed a newer torch FSDPModule; TRL 0.22.2 broke on transformers 5.16).

Hub id: JonaPoka/Qwen3.5-4B-Finnish. Load it like the Qwen base — no PEFT required.

Use enable_thinking=False in the Qwen3.5 chat template.

What it is good at

• Answering in Finnish without dropping back to English
• Short EN→FI translation
• Everyday drafts (cancellation email, messages) in a Finnish register
• Wikipedia-style explanations after the native-text pass
• Leading paraphrases for ice density and YK 1955 (after Stage 4)

What it is not

A 4B experimental fine-tune is not a native grammarian, not a historian, and not a Poro replacement. It still mixes Finnish cases on rules, and it still invents extra 1917 history. Prefer a 9B run on a 48 GB GPU if you need a serious Finnish assistant. Do not use this for legal, medical, or official advice.

How to run

Needs transformers>=4.57. This repo is the merged 4B checkpoint (~8.5 GB).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "JonaPoka/Qwen3.5-4B-Finnish"

tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "system",
        "content": (
            "Olet avulias tekoälyavustaja. Vastaa suomeksi, ellei käyttäjä "
            "nimenomaan pyydä toista kieltä. Käytä luontevaa, kieliopillisesti oikeaa suomea."
        ),
    },
    {
        "role": "user",
        "content": "Selitä lyhyesti, miten suomen kielen partitiivi toimii. Anna kolme esimerkkiä.",
    },
]
enc = tok.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
    return_dict=True,
)
out = model.generate(
    **{k: v.to(model.device) for k, v in enc.items()},
    max_new_tokens=192,
    do_sample=False,
    repetition_penalty=1.08,
)
print(tok.decode(out[0][enc["input_ids"].shape[-1] :], skip_special_tokens=True))

If AutoModelForCausalLM fails to load the hybrid Qwen3.5 checkpoint, use AutoModelForImageTextToText with the same arguments. Stop on <|im_end|>. A light repetition penalty (≈1.08) reduces the ice-density loops we saw in greedy decode.

Training used one LoRA (rank 64) continued across four SFT stages, then merged into the base. You do not need peft at inference.

Training process

One LoRA, continued — not four separate models. Rank 64, alpha 128, dropout 0.05. Targets full-attention (q/k/v/o_proj), Gated DeltaNet (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj), and MLP (gate/up/down_proj). Sequence 1024, QLoRA 4-bit NF4, assistant-only loss, batch 4 × accum 4 (effective 16). Hardware: 1× RTX 3090 / 3090 Ti, 24 GB.

Stage 1 — Finnish lock (15k)

We did not dump the unfiltered 1.4M Poro-2 collection. Judge labels (HYÖDYLLINEN) are not native Finnish.

Stage 2 — GIGO filter, unused rows (13.7k)

Gold was ~1% of this mix, so morphology barely moved. Register and stopping did.

Stage 3 — new native text (11.2k), not a replay

Replaying the 68k would mostly memorize Llama-3.3 Finnish essays. Stage 3 continues the Stage 2 adapter on new licensed text converted to user→assistant turns (the gold is already in the source: Wikipedia lead, or the actual next paragraph).

Not included (would block a public Hub release): Yle and STT full-text archives (Kielipankki academic / no redistribution), Helsingin Sanomat, scraped commercial newsletters. Not used: raw continued pretraining on whole books (that strips instruction following).

Stage 4 — gold polish SFT (this checkpoint)

Continue the Stage 3 native LoRA. Config: configs/sft_qlora_4b_polish.yaml. No DPO after this.

This merged checkpoint contains Stages 1–4. The LoRA was trained, then baked into Qwen/Qwen3.5-4B.

Performance (smoke)

There is no public Finnish instruction leaderboard we trust for a 4B adapter. Evaluation is a fixed smoke set: the same prompts after every run, greedy decode, thinking off, max_new_tokens=192, repetition_penalty=1.08.

Stage 3 paraphrases already had ice-density and YK 1955 right; Stage 4 moved the open ice prompt off the invert, cleaned the dentist email, and made partitive examples usable. The partitive rule and the open 1917 prompt are still unreliable.

How to read this. Stage 1 solved language lock. Stage 2 improved register and stopping. Stage 3 added native-text style; leading factual prompts can work while open versions still fail. Stage 4 is a gold-heavy polish of that same LoRA, not a new model and not DPO.

Limitations

• Case. Partitive examples improved at Stage 4; the verbal rule still confuses partitive with -n. Do not trust morphology explanations.
• History. Independence Day 6 Dec 1917 is usually right; extra invented dates (1919, YK-as-independence) still appear on the open 1917 prompt.
• Science. The leading ice-density paraphrase is correct; the open lake-ice explanation is clumsy and was inverted in earlier stages.
• Capacity. Qwen’s own multilingual numbers are higher for 9B than 4B on every published set. Finnish morphology needs that headroom.
• No DPO. Preference tuning was skipped because of a TRL / torch / transformers mismatch. The polish SFT already moved the demo prompts a lot.

Data licences

• Base weights: Apache-2.0 (Qwen).
• Stage 1–2 instruction rows keep their source licences. The Poro-2 HQ slice includes Llama-3.3-generated Finnish; Llama terms still apply to that slice if you redistribute it.
• Stage 3: Wikipedia CC BY-SA, Gutenberg public domain (USA), Wikinews CC BY-2.5. Handwritten gold is provided for this fine-tune.
• Stage 4: upsampled handwritten gold + Turku paraphrase + FLORES (same licences as the source rows).

Citation

@misc{qwen35-4b-finnish,
  title        = {Qwen3.5-4B Finnish (experimental)},
  author       = {JonaPoka},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/JonaPoka/Qwen3.5-4B-Finnish},
  note         = {Merged LoRA SFT on Qwen/Qwen3.5-4B, thinking disabled, no DPO}
}

Base model: Qwen Team, Qwen3.5.