Downloads · 30 days
810
18% of all-time downloads
Macro-Nano-Instruct-Finetuning
ducanhdinh/Macro-Nano-Instruct-Finetuning
Macro-Nano-Instruct-Finetuning is a machine learning model from ducanhdinh. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
Day la LoRA adapter finetune tu ATH-MaaS/Marco-Nano-Instruct, mot mo hinh Mixture-of-Experts.
All-time downloads
4.4K
Public
Repo size
92.1 GB
Likes
4
Public
Repo makeup
Click a slice to open those files.
.pt1.2 GB · 67%
At a glance
- Library
- peft
- License
- apache-2.0
- Access
- Public
- Created
- Aug 18, 2026
- Updated
- Aug 30, 2026
- SHA
- 67de7d26
Base models
Model card
From the Hugging Face model README
Macro-Nano-Instruct-Finetuning
Day la LoRA adapter finetune tu ATH-MaaS/Marco-Nano-Instruct, mot mo hinh Mixture-of-Experts.
Cau hinh LoRA
- Layer duoc finetune:
[9, 18)trong tong so28layer (tuong ung khoang 1L/3 -> 2L/3). - Module duoc gan LoRA: attention, router, experts trong khoang layer tren.
- r = 16, alpha = 32, dropout = 0.05
Loss
Loss MoE tieu chuan:
L_total = L_LM + lb_loss_coef * L_LB
L_LM: cross-entropy chuan tren token tiep theo.L_LB: load balancing loss chuan cua MoE (Switch/Mixtral style), tinh tren cac router nam trong khoang layer duoc finetune.lb_loss_coef= Nonenum_experts= 232,top_k= 8
Du lieu
Cau don ngu duoc gom tu 3 bo du lieu alignment: flores.json, bible.json, ntrex.json
(moi field ngon ngu trong 1 record duoc coi la 1 sample), shuffle va sort theo do dai token
truoc khi gom batch.
Diagnostics
Xem diagnostics/loss_log.jsonl (log theo tung step) va diagnostics/loss_curve.png
(bieu do L_LM / L_LB / L_Total theo step).
- Library
- peft
- License
- apache-2.0
- Created
- Aug 18, 2026
- Updated
- Aug 30, 2026