Downloads · 30 days
0
llama-3.1-8b-instruct_LBox-10x2-plus-generalist-lora-moe
Jongbin-kr/llama-3.1-8b-instruct_LBox-10x2-plus-generalist-lora-moe
llama-3.1-8b-instruct_LBox-10x2-plus-generalist-lora-moe is a machine learning model from Jongbin-kr. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as llama3.1.
should probably proofread and complete it, then remove this comment. --
Access
Public
Updated Sep 5, 2026
Repo size
10.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors629 MB · 97%
At a glance
- Library
- transformers
- License
- llama3.1
- Access
- Public
- Created
- Sep 4, 2026
- Updated
- Sep 5, 2026
- SHA
- 83a178d7
Base models
Model card
From the Hugging Face model README
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
llama-3.1-8b-instruct_LBox-10x2-plus-generalist-lora-moe
This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 1.0368
- Router Supervised Loss: 0.7111
- Router Load Balance Loss: 3.1977
- Router Z Loss: 1.7449
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 16
- total_eval_batch_size: 2
- optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 0.03
- num_epochs: 5.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Supervised Loss | Load Balance Loss | Z Loss |
|---|---|---|---|---|---|---|
| 9.2157 | 0.3266 | 500 | 1.2531 | 0.6028 | 3.5467 | 4.5532 |
| 9.3029 | 0.6533 | 1000 | 1.1633 | 0.5261 | 3.7052 | 4.0598 |
| 8.8652 | 0.9799 | 1500 | 1.1281 | 0.6396 | 3.4210 | 3.4540 |
| 9.0210 | 1.3064 | 2000 | 1.0920 | 0.7815 | 3.0531 | 3.0569 |
| 9.4279 | 1.6330 | 2500 | 1.0720 | 0.5916 | 3.4984 | 2.6309 |
| 9.1875 | 1.9597 | 3000 | 1.0659 | 0.7595 | 3.1193 | 2.5054 |
| 8.8329 | 2.2861 | 3500 | 1.0522 | 0.5281 | 3.6835 | 2.2354 |
| 8.9111 | 2.6128 | 4000 | 1.0467 | 0.7138 | 3.2494 | 2.0908 |
| 7.8045 | 2.9394 | 4500 | 1.0469 | 0.6881 | 3.2693 | 2.0084 |
| 7.6419 | 3.2659 | 5000 | 1.0448 | 0.8645 | 2.9374 | 2.0494 |
| 7.9167 | 3.5925 | 5500 | 1.0425 | 0.6480 | 3.3811 | 1.8848 |
| 8.1925 | 3.9192 | 6000 | 1.0374 | 0.7151 | 3.1922 | 1.7747 |
| 7.7807 | 4.2456 | 6500 | 1.0373 | 0.6394 | 3.3661 | 1.7425 |
| 7.9786 | 4.5723 | 7000 | 1.0373 | 0.6628 | 3.3196 | 1.7385 |
| 8.0582 | 4.8989 | 7500 | 1.0369 | 0.7080 | 3.2079 | 1.7431 |
| 7.5259 | 5.0 | 7655 | 1.0368 | 0.7111 | 3.1977 | 1.7449 |
Framework versions
- Transformers 5.9.0
- Pytorch 2.11.0+cu130
- Datasets 4.4.1
- Tokenizers 0.22.2
- Library
- transformers
- License
- llama3.1
- Created
- Sep 4, 2026
- Updated
- Sep 5, 2026