Qwen1.5-MoE-A2.7B-Finetuning

Qwen1.5-MoE-A2.7B-Finetuning

LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B (Mixture-of-Experts) tren mix 3 task: SNLI (NLI), SQuAD (extractive QA), MMLU (trac nghiem), train chung trong 1 lan (cac tap duoc tron lai, shuffle).

Task: snli, squad, mmlu — so sample sau khi loc: snli: 549367, squad: 87599, mmlu: 87691.

Prompt format (giu nguyen nhu khi train de eval zero-shot)

SNLI:

Premise: <premise>
Hypothesis: <hypothesis>
Question: What is the relationship between the premise and the hypothesis? Choose one: entailment, neutral, or contradiction.
Answer: <label>

SQuAD:

Context: <context>
Question: <question>
Answer: <answer>

MMLU:

The following are multiple choice questions (with answers) about <subject>.

<question>
A. <choice_A>
B. <choice_B>
C. <choice_C>
D. <choice_D>
Answer: <letter>

Phan sau Answer: la phan model sinh ra va la phan DUY NHAT duoc tinh loss.

Cau hinh LoRA

• Layer duoc finetune: [8, 16) trong tong so 24 layer (khoang 1L/3 -> 2L/3).
• Module gan LoRA: attention (r = 16), router (r = 4), experts (r = 16) trong khoang layer tren (rank rieng qua rank_pattern cua PEFT).
• alpha = 32, dropout = 0.05

Loss

L_total = L_LM + lb_loss_coef * L_LB

• L_LM: cross-entropy chuan, chi tren phan <answer><eos>, chuan hoa theo tong so token nhan cua global batch.
• L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style) tren cac router trong khoang layer duoc finetune.
• lb_loss_coef = 0.01, num_experts = 60, top_k = 4

Training

• 3 epoch, lr = 0.0002, warmup = 0.03, grad clip = 1.0
• Batch: co dinh 8 sample moi batch moi GPU.
• max_length = 512; context SQuAD qua dai duoc windowing quanh vi tri dap an.

Diagnostics

Xem thu muc diagnostics/: loss_log.jsonl (tung step, gom loss/acc theo task), loss_curve.png, loss_curve_smoothed.png, accuracy_curve.png, task_curves_smoothed.png. Accuracy o day la ty le token dung (teacher forcing) tren train batch, KHONG phai do chinh xac chuan tren tap eval.