gemma-4-12b-cymraeg-v6

gemma-4-12b-cymraeg-v6

The flagship conversational Welsh model in the EryriLabs Gemma 4 line. It is Gemma 4 12B instruction tuned, continued pretrained on a large Welsh corpus and then supervised fine tuned for conversation, so it answers in natural Welsh rather than translating from English as it goes.

What separates this release from the ones before it is the amount of Welsh the model was actually immersed in. Continued pretraining ran in two blocks over complementary halves of the corpus, and between them they covered 307M of the 336M token corpus. Block B started from the merged output of Block A and drew only on rows Block A had never seen, so the two blocks add coverage rather than repeat it.

This repository holds the supervised fine tuning adapter and the evaluation record. It does not hold merged bf16 weights, and it does not hold the two continued pretraining adapters: those live in their own repositories, named below, and copying them here a third time would serve nobody.

Conversational benchmark results

Method first: 100 fresh conversational Welsh prompts were authored for this test and machine checked for zero overlap against every earlier prompt set and against all training data of every model involved. Three models answered under their own documented settings. Scoring was blind (shuffled slots, deterministic seed) on a rubric fixed before any answer existed: helpfulness, correctness and naturalness of the Welsh, register, and fitness of length for a chat turn. The judge is a language model, not a native speaker; its per prompt scores, its uncertainty list, and the full prompt sets are published in eval/ so anyone can re mark the papers.

On that benchmark this model won 66 prompts to 33 with 1 tie against BangorAI/Mistral-7B-Cymraeg-Welsh-v2 (win rate 66.7 percent, 95 percent CI 56.9 to 75.2, p = 0.0012), and won a separate 14 turn dialogue test 4.34 to 3.74 with the strongest cross turn memory of any model tested. In the same set of experiments BangorAI's model won a 20 prompt mixed category exam 13 to 7 at each model's shipping settings and remains the stronger long form prose writer, with the most polished Welsh sentence by sentence (language quality 4.52 vs 4.18). The difference is behavioural: in the conversational hundred, BangorAI's replies included assistant style boilerplate 25 times, refused a reasonable request 17 times and missed the ti/chi register 15 times, against 0, 1 and 5 for this model.

BangorAI's model is the strongest other open Welsh model available and the only 7B class comparison; this work also trains on open data released by techiaith at Bangor University and by the Welsh public sector. Diolch o galon i bawb.

Lineage: google/gemma-4-12b-it, continued pretraining on about 307 million tokens of open Welsh text (the full MADLAD-400 Welsh split plus the techiaith corpora; embeddings trained in the first stage), then conversational supervised fine tuning. See DATA-CARD files and cpt_progress.json for exact coverage.

What is in this repository

How the merged weights were produced

The bf16 merged checkpoint is not uploaded. The weights come from applying three LoRA adapters to the base model, in this order:

1. CPT Block A, trained on the first slice of the corpus, with a trained embed_tokens table.
2. CPT Block B, trained on top of base plus Block A and restricted to rows Block A had never seen, with the embedding table frozen.
3. The stage 2 supervised fine tuning adapter, included in this repository under adapters/sft/.

The order mattered: Block B was trained on top of base plus Block A, so applying it directly to the stock base would have produced a different model.

CPT Block A and CPT Block B were kept as private checkpoint repositories and were never published, so the merge above cannot be reproduced from public sources. This section describes how the weights were built, not a set of steps a reader can carry out.

corpus_manifest.json in this repository describes the composition of the continued pretraining corpus. The corpus itself is not published.

Quantised GGUF builds of the merged model are published separately at EryriLabs/gemma-4-12b-cymraeg-v6-GGUF.

Intended use

Welsh language conversation, question answering and general assistance, with English understood as a second language. The model is a general purpose assistant and is not a substitute for professional advice in any field.

Evaluation

Blind LLM as judge (Opus), identical prompts and settings for every arm. The judge is not a native speaker: structural findings are reliable, fine style judgements less so. Full report and per response scores are in eval/.

Welsh, 40 prompts:

Head to head with BangorAI/Mistral-7B-Cymraeg-Welsh-v2, the strongest other open Welsh model and the only 7B class comparison (diolch to BangorAI and to techiaith at Bangor University, whose open data trained much of this model): single turn pairwise 15 wins, 5 ties, 20 losses (BangorAI keeps the single turn edge; the gap has narrowed from 14-21 for v2 to 16-18 for v6 greedy). Multi turn conversation, 14 fixed turns with history: v6 4.34 vs BangorAI 3.74, per turn 8-2-4, with cross turn memory the largest gain (3.92 to 4.46). English is fully retained (4.90 fluency, 20/20 good English).

v6 vs v5 pairwise: 16 wins, 16 ties, 8 losses. The additional immersion bought coherence and fact repair more than sentence elegance.

Known limitations: long form answers past ~150 tokens can leak English or loop under greedy decoding (cap generation at ~160 tokens and use the sampling below, which removes every observed catastrophic failure); rare unprovoked refusals (2 of 80 responses); it performs mutations far better than it explains them; numeric facts should be grounded via the system prompt.

Recommended settings: temperature 0.3, top_p 0.9, min_p 0.05, repetition_penalty 1.1, max_new_tokens 160, thinking off.

CymraegBench v0.1 (independent protocol, run by us, 1 November 2026)

CymraegBench is BangorAI's open Welsh benchmark: 19 core suites (Welsh ARC, PIQA, TruthfulQA, XNLI, COPA, MGSM, CEFR, and techiaith's vocabulary, grammar, registers, place-name and translation sets), scored deterministically at temperature 0 with a 512-token floor, so well outside the recommended settings above. We ran the published 100-case protocol against the Q8_0 GGUF under llama.cpp.

On BangorAI's own extended scoreboard that sits just below Qwen3.5-9B (55.10) and above Jupiter-N-120B (48.87) and Mistral-7B-Cymraeg-Welsh-v2 (30.09); the legislation BLEU beats every Welsh-specific fine-tune and small open model they list (Jupiter-N-120B 43.85, Qwen3.5-9B 37.06), though the frontier models score around 70. Those are their runs, so treat the comparison as indicative. Our E4B v4 scored 49.64 on the same setup. Best suites: vocabulary 86, registers 79, Welsh maths 75, ARC-Easy 73. Weakest: place names 6, swearing detection 9, CEFR levels 15, yes/no forms 17. One maths answer looped to the token cap and scores zero. Full tables, protocol and serving details: eval/cymraegbench-v0.1/.

Training

Stage 1, continued pretraining, ran as two sequential blocks over complementary halves of the corpus, together covering 307M of the 336M token corpus.

• Block A: LoRA r 128 on the attention and MLP projections, plus a trained embed_tokens table through modules_to_save, on the first slice of the corpus.
• Block B: LoRA r 128 on the same projections, embeddings frozen, trained on top of base plus Block A and restricted to rows Block A had never seen. Freezing the embedding table is why Block A's most expensive tensor survives the chain intact rather than being partly relearned on a smaller sample.

Stage 2, supervised fine tuning: LoRA r 64, alpha 64, dropout 0, on the seven linear projections only, with no embedding training. Learning rate 5e-5 with a cosine schedule and a 0.03 warmup ratio, sequence length 2048, one epoch. The conversational rows are mixed with a sampled replay of the stage 1 corpus so that stage 2 does not undo stage 1, and the last 20 percent of the training order is conversation only, so the low learning rate end of the schedule is spent entirely on answering rather than on continuing text.

Data and attribution

The full source list with token counts is in the data card. The corpus is built from openly licensed material only:

• MADLAD-400, Welsh portion, under ODC-BY.
• Welsh Government material via llyw.cymru, under the Open Government Licence.
• Cofnod y Cynulliad and Senedd record material, under the Open Government Licence.
• Cardiff University translation memory, under CC-BY.
• Banc Trawsgrifiadau Bangor, under CC0.

The techiaith collections are gratefully acknowledged. Licence terms travel with the data: anything requiring attribution is attributed here and in the data card.

Licence

Apache 2.0, matching the base model google/gemma-4-12b-it. The corpus licences above apply to the training data, not to the model weights.

Limitations

• Welsh is a lower resource language and the model will still make mistakes, including confident ones. Check anything that matters.
• Mutations, register and dialect are uneven. Formal written Welsh is stronger than colloquial speech.
• The model can produce English when pushed hard in English, especially without a Welsh system prompt. The multi turn arms measure exactly this.
• No safety tuning beyond what the base model carries.