Qwen1.5-MoE-A2.7B-Finetuning-lb0.01

Qwen1.5-MoE-A2.7B-Finetuning

Day la LoRA adapter finetune tu Qwen/Qwen1.5-MoE-A2.7B, mot mo hinh Mixture-of-Experts.

Cau hinh LoRA

• Layer duoc finetune: [8, 16) trong tong so 24 layer (tuong ung khoang 1L/3 -> 2L/3).
• Module duoc gan LoRA: attention, router, experts trong khoang layer tren, moi thanh phan mot rank rieng qua rank_pattern cua PEFT:
  • attention: r = 16
  • router: r = 4
  • experts: r = 16
• alpha = 32, dropout = 0.05

Loss

Loss MoE tieu chuan:

L_total = L_LM + lb_loss_coef * L_LB

• L_LM: cross-entropy chuan tren token tiep theo.
• L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.
• lb_loss_coef = 0.01
• num_experts = 60, top_k = 4

Du lieu

Alignment data duoc chon qua --alignment_data (flores ntrex ted), gom sample tu cac file: flores.json, ntrex.json, ted.json (moi field ngon ngu trong 1 record duoc coi la 1 sample), shuffle va sort theo do dai token truoc khi gom batch.

Diagnostics

Xem diagnostics/loss_log.jsonl (log theo tung step), diagnostics/loss_curve.png (bieu do tho L_LM / L_LB / L_Total theo tung step) va diagnostics/loss_curve_smoothed.png (cung 3 duong loss nhung da lay trung binh cong moi 50 step lien tiep — de doc xu huong hon vi bieu do tho rat messy o cap do tung step).