Qwen1.5-MoE-A2.7B-MMLU-Task-Only

Qwen1.5-MoE-A2.7B-MMLU-Task-Only

Day la LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B, mot mo hinh Mixture-of-Experts, tren task MMLU (Massive Multitask Language Understanding) — cau hoi trac nghiem nhieu linh vuc, chon 1 trong N lua chon (thuong la 4: A/B/C/D).

Prompt format

The following are multiple choice questions (with answers) about <subject>.

<question>
A. <choice_A>
B. <choice_B>
C. <choice_C>
D. <choice_D>
Answer: <letter>

Neu khong co subject (nhu du lieu auxiliary_train.json), header bo phan "about <subject>". Phan sau Answer: la phan model sinh ra. Prompt nay DUNG CHUAN format pho bien khi eval MMLU (lm-evaluation-harness), nen dung TRUC TIEP duoc cho zero-shot eval tren cac benchmark trac nghiem cung dinh dang, vi du MMMLU (ban da ngon ngu cua MMLU): chi can thay cau hoi/lua chon bang du lieu ngon ngu do, giu nguyen cau truc, roi so sanh xac suat token dap an ung voi A/B/C/D... (hoac generate() va lay chu cai dau tien sinh ra).

Cau hinh LoRA

• Layer duoc finetune: [8, 16) trong tong so 24 layer (tuong ung khoang 1L/3 -> 2L/3), giong het setting cua cac code SNLI/SQuAD/alignment.
• Module duoc gan LoRA: attention, router, experts trong khoang layer tren, moi thanh phan mot rank rieng qua rank_pattern cua PEFT:
  • attention: r = 16
  • router: r = 4
  • experts: r = 16
• alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

• L_LM: cross-entropy chuan tren token tiep theo, CHI tinh tren phan <letter><eos> (prompt cau hoi + cac lua chon bi mask, giong instruction-tuning/SFT tieu chuan).
• L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
• lb_loss_coef (lambda) = 0.01
• num_experts = 60, top_k = 4

Du lieu

MMLU, doc tu data/english_task/mmlu/auxiliary_train.json (list object {"question", "choices": [...], "answer": int, "subject"}). Sample nao co full_text vuot qua --max_length bi bo qua (khong truncate, de tranh cat mat chu cai dap an nam o cuoi chuoi).

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png, diagnostics/loss_curve_smoothed.png va diagnostics/accuracy_curve.png (ty le token dung xap xi tren toan bo token cua nhan <letter><eos> — teacher forcing, tinh tren batch train, da lam min moi 50 step; day KHONG phai do chinh xac chuan tren tap eval MMLU/MMMLU, chi la diagnostics theo doi qua trinh train).