MiniCPM5-2B-catalan-chat

MiniCPM5-2B-catalan-chat

A conversational Catalan chat model: LoRA fine-tune of openbmb/MiniCPM5-2B on the full projecte-aina/InstruCAT instruction dataset (165,100 samples, 11 task categories, ~43M tokens, 1 epoch).

Model Details

• Base model: openbmb/MiniCPM5-2B (2.5B params, Llama architecture, Apache-2.0)
• Fine-tuned on: projecte-aina/InstruCAT — Catalan instructions converted from caBreu, CatalanQA, CaWikiTC, ceil, CoqCat, GuiaCat, IntoxiCat, NLUCat, Parafraseja, PAWS-ca, sts-ca, teca and WikiCat
• Language: Catalan (ca-ES)
• Task: conversational instruction following
• License: base model Apache-2.0; training data CC-BY-NC-ND-4.0 (non-commercial) — check the InstruCAT license terms before commercial use

Training

Observed training curve (completion-only loss, every 10 steps): 2.39 → 1.48 (step 20) → 1.04 (step 30) → 0.45 (step ~70) → ~0.35–0.40 at the end. Mean token accuracy reached ~0.91–0.92 by the second half of the run. Eval on 1,000 held-out validation samples ran at step 500 without errors.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "luispoveda93/MiniCPM5-2B-catalan-chat",
    dtype="bfloat16",
)
tok = AutoTokenizer.from_pretrained("luispoveda93/MiniCPM5-2B-catalan-chat")

messages = [{"role": "system", "content": "Ets un assistent conversacional que respon sempre en català."},
            {"role": "user", "content": "Explica'm què és la Sardana."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=256)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

The LoRA adapter alone is at luispoveda93/MiniCPM5-2B-catalan-chat-lora.

Limitations

• Trained on single-turn instruction data; no multi-turn Catalan conversations were used.
• InstruCAT is task-oriented (QA, paraphrase, NER, classification, summarization); general chit-chat behavior comes from the base model.
• Non-commercial data license applies to the fine-tuning dataset.

Citation

Please cite the InstruCAT dataset and Projecte AINA if you use this model:

• Dataset: projecte-aina/InstruCAT — Language Technologies Unit ([email protected]) at BSC, funded within project ILENIA (ref. 2022/TL22/00215337).