m51Lab-NorskGemma4-31B

m51Lab-NorskGemma4-31B

Norway's top-scoring open-source language model on NorEval.

Built by m51.ai Lab through surgical fine-tuning of Google Gemma 4 31B-it for Norwegian (Bokmaal and Nynorsk).

Quantized GGUF versions for local inference: m51Lab-NorskGemma4-31B-GGUF

Benchmark Results

Evaluated on NorEval (ACL 2025) — the standard benchmark for Norwegian language models. Protocol: 8 tasks, 5 prompt templates per task (best-of-5), loglikelihood scoring, full test sets, apply_chat_template=True.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "dervig/m51Lab-NorskGemma4-31B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="eager",  # Required for global attention layers
)

messages = [
    {"role": "user", "content": "Kva er hovudstaden i Noreg?"}
]

inputs = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Requirements

• GPU memory: ~64 GB for BF16 inference (1x A100 80GB or 2x A100 40GB)
• attn_implementation="eager": Required because global attention layers use head_dim=512, which is incompatible with Flash Attention 2
• transformers >= 5.5.0, torch >= 2.6.0

Training Details

This model was created through a careful, surgical fine-tuning process — informed by 5 prior failed SFT attempts on smaller Gemma 4 variants (4B dense and 26B MoE) that all degraded performance.

What Made This Attempt Different

Training Configuration

Architecture

Model class:     Gemma4ForConditionalGeneration (dense, no MoE)
Layers:          60 (50 sliding + 10 global, pattern 5:1)
Hidden size:     5376
Attention heads: 32 (16 KV-heads sliding, 4 KV-heads global)
Head dim:        256 (sliding) / 512 (global)
MLP:             21504 intermediate
Total params:    31.27B
Context:         256K tokens

Training Data Sources

Contamination Check

We performed a formal contamination analysis comparing all 6,445 text segments from the training data against 18,124 test texts across all 8 NorEval tasks. Three methods were used: exact normalized matching, substring matching, and character-level n-gram overlap (50-gram and 30-gram).

Result: Zero contamination detected. No exact matches, no substring matches, and no suspicious n-gram overlaps (>30%) were found across any of the 8 NorEval tasks. The benchmark scores reflect genuine model performance.

Limitations

• Inherits limitations and potential biases from the base Gemma 4 model
• Optimized for NorEval benchmark tasks; real-world Norwegian capabilities may vary
• Requires attn_implementation="eager" (global layers have head_dim=512, incompatible with Flash Attention 2)
• The base model is multimodal (Gemma4ForConditionalGeneration); text-only inference requires mm_token_type_ids input — handled automatically by apply_chat_template
• Not a "thinking" model — does not use structured chain-of-thought reasoning tokens

Acknowledgments and Credits

This model would not have been possible without the work of many teams and individuals:

• Google DeepMind — for the Gemma 4 model family and the Apache 2.0 license that enables open research
• NbAiLab (National Library of Norway AI Lab) — for building and openly sharing the Norwegian NLP datasets that made fine-tuning possible: norwegian-alpaca, torgersen-alpaca, ndla_npk, nynorsk_dpo, nb-global-mmlu, and many more
• Language Technology Group (LTG), University of Oslo — for creating and publishing the NorEval benchmark (ACL 2025), providing the Norwegian NLP community with a standardized evaluation framework
• NorMistral / NorwAI / norallm teams — for pioneering Norwegian LLM development and establishing baselines that guided this work
• Hugging Face — for the transformers, PEFT, and TRL libraries
• PiSSA authors (Meng et al., 2024) — for the Principal Singular Values and Singular Vectors Adaptation method
• RunPod — for accessible GPU infrastructure

Citation

@misc{m51lab2026norskgemma4,
  title={m51Lab-NorskGemma4-31B: Surgical Fine-Tuning of Gemma 4 for Norwegian},
  author={m51.ai Lab},
  year={2026},
  url={https://huggingface.co/dervig/m51Lab-NorskGemma4-31B},
}

---

Built by m51.ai Lab. Read the full build log and technical analysis on our blog.