kila-qwen2.5-7b-adapter

Kila Finnish Accounting Adapter — Qwen2.5-7B-Instruct

A QLoRA adapter fine-tuned on Kila (Kirjanpitolautakunta) corpus — Finland's official accounting standards board. The adapter specialises Qwen2.5-7B-Instruct on Finnish accounting statements, exemptions, and general guidelines (yleisohjeet) to improve factual accuracy on Finnish GAAP questions.

---

Model Details

• Base model: Qwen/Qwen2.5-7B-Instruct
• Adapter type: LoRA (QLoRA 4-bit quantised training)
• Language: Finnish (fi)
• Domain: Finnish accounting standards (Kila / kirjanpitolaki)
• Fine-tuning method: Supervised Fine-Tuning (SFT) on synthetic instruction data generated from Kila corpus chunks
• Framework: PEFT 0.19.1 + TRL + Transformers

---

Training Details

Training Data

Synthetic Finnish Q&A pairs generated from Kila corpus chunks (statements, exemptions, general guidelines). Data pipeline:

1. Load Kila chunks from sources/kila/{date}/chunks_part-*.parquet
2. Deduplicate by chunk_hash
3. Generate 1–2 Finnish instruction/response pairs per chunk via Groq llama-3.3-70b-versatile
4. Split into train/val by document_id (no document appears in both splits)

Dataset size:

• Train: synthetic_kila_train.jsonl (~1,878 KB)
• Val: synthetic_kila_val.jsonl (~233 KB)

Training Hyperparameters

Hardware

• GPU: NVIDIA RTX 5090 (32 GB VRAM) — sm_120, 1x GPU
• Blackwell-specific settings: bf16=True, gradient_checkpointing_kwargs={"use_reentrant": False}

---

Evaluation

Judge: Groq llama-3.3-70b-versatile (temp=0.0)

Rubric (1–5 scale):

• correctness — factual accuracy per Kila / kirjanpitolaki
• grounding — references authoritative sources
• fluency — professional Finnish grammar

Results are reported in eval_summary.csv comparing base model vs fine-tuned adapter on ~12 Finnish accounting questions grounded in specific Kila documents.

---

How to Get Started

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_path = "rumeshmohan/kila-qwen2.5-7b-adapter"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_path)
model.eval()

messages = [
    {"role": "system", "content": "Olet suomalaisen kirjanpidon asiantuntija."},
    {"role": "user", "content": "Miten tutkimusmenot käsitellään kirjanpidossa?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.1,
        top_p=0.9,
        do_sample=True
    )

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

---

Framework Versions

• PEFT 0.19.1
• Transformers ≥ 4.40.0
• TRL ≥ 0.8.6
• PyTorch ≥ 2.3.0