MiniCPM5-2B-catalan-chat-v2

MiniCPM5-2B-catalan-chat-v2

Conversational Catalan chat model — round 2. A LoRA fine-tune of luispoveda93/MiniCPM5-2B-catalan-chat (itself a LoRA fine-tune of openbmb/MiniCPM5-2B on InstruCAT) focused on multi-turn chat behaviour.

What round 2 added

Round 1 trained on single-turn task instructions (projecte-aina/InstruCAT). Round 2 trains on 23.4K multi-turn Catalan conversations from the chat-dense sources of BSC-LT/ALIA-2606-SFT — the SFT mixture that trained ALIA-40b-instruct-2606:

All conversations CC-BY-4.0 (removes round-1's non-commercial restriction on the training data; note InstruCAT is still in the round-1 lineage). A Catalan system prompt ("Ets un assistent conversacional que respon sempre en català.") was prepended to conversations lacking one. 400 conversations held out for eval.

Training details

• Base: luispoveda93/MiniCPM5-2B-catalan-chat (round-1 merged model)
• Method: LoRA r=32 / α=64, all projection layers, lr 1e-4 cosine (gentler than round 1's 2e-4), warmup 50, 1 epoch, effective batch 32 (4 × 8), max_length 2048 packed, bf16, completion-only loss
• Scale: ~326 packed steps, 2h19m on A100-large
• Observed: train loss 2.44 → ~1.44; eval loss 1.48 → 1.41 → 1.383; token accuracy 0.57 → 0.704 (400-conversation held-out eval)
• Training metrics (trackio)

Artifacts

• This repo: merged bf16 model (5.0 GB)
• Adapter: luispoveda93/MiniCPM5-2B-catalan-chat-v2-lora (on top of the round-1 merged model)
• Round 1: luispoveda93/MiniCPM5-2B-catalan-chat · GGUF

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("luispoveda93/MiniCPM5-2B-catalan-chat-v2", dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("luispoveda93/MiniCPM5-2B-catalan-chat-v2")

messages = [
    {"role": "system", "content": "Ets un assistent conversacional que respon sempre en català."},
    {"role": "user", "content": "Hola! Com estàs avui?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(inputs, max_new_tokens=256)[0][inputs.shape[1]:], skip_special_tokens=True))

Limitations

• 2.5B parameters: limited world knowledge and reasoning depth; Catalan conversational style is the training focus.
• Round 2 used the chat-dense core of the ALIA mixture (~23.4K conversations); the QA corpora (fineweb-edu, RAG, math) were deliberately excluded to prioritize chat behaviour — task-QA skills come from round 1.
• Evaluation is loss/accuracy-based on held-out conversations; no independent Catalan chat benchmark was run.