Downloads · 30 days
20
7% of all-time downloads
Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-sni-switch-top1-router-ffn-lora-sft-5ep
llama-3.1-8b-instruct-4x1-moe-sni-switch-top1-router-ffn-lora-sft-5ep is a text generation model from Jongbin-kr. Use it when you need the model to write or continue text. It is set up for peft.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
20
7% of all-time downloads
All-time downloads
307
Public
Repo size
25 GB
Likes
0
Public
Click a slice to open those files.
.safetensors454 MB · 96%
From the Hugging Face model README
This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss | Entropy | Max Load | 0 Utilization | 1 Utilization | 2 Utilization | 3 Utilization | Aux Loss | Z Loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1.2081 | 0.0920 | 200 | 0.8260 | 1.2599 | 0.3383 | 0.2024 | 0.2078 | 0.2515 | 0.3383 | 1.0215 | 2.2505 |
| 0.8482 | 0.1841 | 400 | 0.5173 | 1.1600 | 0.3160 | 0.2220 | 0.2043 | 0.2577 | 0.3160 | 1.0082 | 2.0889 |
| 0.7691 | 0.2761 | 600 | 0.4577 | 1.1238 | 0.2852 | 0.2369 | 0.2301 | 0.2479 | 0.2852 | 0.9989 | 1.7243 |
| 0.5757 | 0.3681 | 800 | 0.4337 | 1.1022 | 0.2784 | 0.2322 | 0.2387 | 0.2508 | 0.2784 | 0.9982 | 1.4809 |
| 0.7538 | 0.4602 | 1000 | 0.4147 | 1.0964 | 0.2776 | 0.2368 | 0.2361 | 0.2495 | 0.2776 | 0.9982 | 1.3930 |
| 0.6009 | 0.5522 | 1200 | 0.4009 | 1.0752 | 0.2775 | 0.2416 | 0.2292 | 0.2518 | 0.2775 | 0.9978 | 1.2824 |
| 0.6409 | 0.6442 | 1400 | 0.3948 | 1.0800 | 0.2721 | 0.2482 | 0.2329 | 0.2468 | 0.2721 | 0.9977 | 1.1922 |
| 0.6234 | 0.7362 | 1600 | 0.3865 | 1.0658 | 0.2742 | 0.2431 | 0.2325 | 0.2503 | 0.2742 | 0.9981 | 1.1593 |
| 0.6476 | 0.8283 | 1800 | 0.3790 | 1.0669 | 0.2724 | 0.2395 | 0.2342 | 0.2538 | 0.2724 | 0.9977 | 1.0729 |
| 0.5467 | 0.9203 | 2000 | 0.3731 | 1.0578 | 0.2786 | 0.2427 | 0.2253 | 0.2534 | 0.2786 | 0.9982 | 1.0895 |
| 0.5881 | 1.0120 | 2200 | 0.3720 | 1.0542 | 0.2670 | 0.2433 | 0.2382 | 0.2515 | 0.2670 | 0.9978 | 0.9643 |
| 0.5913 | 1.1040 | 2400 | 0.3682 | 1.0633 | 0.2688 | 0.2392 | 0.2382 | 0.2539 | 0.2688 | 0.9980 | 0.9374 |
| 0.6345 | 1.1960 | 2600 | 0.3676 | 1.0745 | 0.2654 | 0.2455 | 0.2364 | 0.2528 | 0.2654 | 0.9983 | 0.8719 |
| 0.5550 | 1.2881 | 2800 | 0.3653 | 1.0772 | 0.2693 | 0.2504 | 0.2242 | 0.2562 | 0.2693 | 0.9980 | 0.8824 |
| 0.6406 | 1.3801 | 3000 | 0.3629 | 1.0768 | 0.2669 | 0.2433 | 0.2337 | 0.2560 | 0.2669 | 0.9982 | 0.8385 |
| 0.5551 | 1.4721 | 3200 | 0.3605 | 1.0921 | 0.2605 | 0.2451 | 0.2364 | 0.2581 | 0.2605 | 0.9983 | 0.7794 |
| 0.5237 | 1.5641 | 3400 | 0.3583 | 1.0725 | 0.2678 | 0.2480 | 0.2337 | 0.2505 | 0.2678 | 0.9983 | 0.8032 |
| 0.5682 | 1.6562 | 3600 | 0.3562 | 1.0831 | 0.2649 | 0.2484 | 0.2354 | 0.2514 | 0.2649 | 0.9984 | 0.7675 |
| 0.6292 | 1.7482 | 3800 | 0.3552 | 1.0862 | 0.2710 | 0.2411 | 0.2337 | 0.2542 | 0.2710 | 0.9988 | 0.7706 |
| 0.6170 | 1.8402 | 4000 | 0.3545 | 1.0816 | 0.2715 | 0.2428 | 0.2322 | 0.2534 | 0.2715 | 0.9986 | 0.7594 |
| 0.5174 | 1.9323 | 4200 | 0.3524 | 1.1005 | 0.2625 | 0.2441 | 0.2427 | 0.2508 | 0.2625 | 0.9987 | 0.7037 |
| 0.4999 | 2.0239 | 4400 | 0.3619 | 1.0682 | 0.2729 | 0.2388 | 0.2342 | 0.2540 | 0.2729 | 0.9985 | 0.7198 |
| 0.6737 | 2.1160 | 4600 | 0.3625 | 1.0872 | 0.2694 | 0.2442 | 0.2342 | 0.2521 | 0.2694 | 0.9984 | 0.6582 |
| 0.4839 | 2.2080 | 4800 | 0.3630 | 1.1029 | 0.2723 | 0.2421 | 0.2378 | 0.2479 | 0.2723 | 0.9983 | 0.6369 |
| 0.5249 | 2.3000 | 5000 | 0.3630 | 1.0953 | 0.2684 | 0.2427 | 0.2368 | 0.2521 | 0.2684 | 0.9985 | 0.6422 |
| 0.4198 | 2.3921 | 5200 | 0.3653 | 1.0877 | 0.2676 | 0.2431 | 0.2346 | 0.2547 | 0.2676 | 0.9983 | 0.6281 |
| 0.5578 | 2.4841 | 5400 | 0.3624 | 1.1022 | 0.2668 | 0.2450 | 0.2350 | 0.2533 | 0.2668 | 0.9987 | 0.6181 |
| 0.5427 | 2.5761 | 5600 | 0.3635 | 1.1063 | 0.2727 | 0.2409 | 0.2339 | 0.2525 | 0.2727 | 0.9988 | 0.6284 |
| 0.5417 | 2.6681 | 5800 | 0.3620 | 1.1144 | 0.2748 | 0.2416 | 0.2351 | 0.2485 | 0.2748 | 0.9987 | 0.6128 |
| 0.5170 | 2.7602 | 6000 | 0.3601 | 1.1085 | 0.2627 | 0.2433 | 0.2419 | 0.2520 | 0.2627 | 0.9989 | 0.5982 |
| 0.4715 | 2.8522 | 6200 | 0.3602 | 1.1005 | 0.2699 | 0.2424 | 0.2336 | 0.2540 | 0.2699 | 0.9986 | 0.6079 |
| 0.4694 | 2.9442 | 6400 | 0.3583 | 1.1124 | 0.2612 | 0.2437 | 0.2453 | 0.2497 | 0.2612 | 0.9990 | 0.5831 |
| 0.4236 | 3.0359 | 6600 | 0.3786 | 1.1054 | 0.2581 | 0.2412 | 0.2467 | 0.2540 | 0.2581 | 0.9991 | 0.5539 |
| 0.5597 | 3.1279 | 6800 | 0.3780 | 1.0992 | 0.2615 | 0.2436 | 0.2443 | 0.2506 | 0.2615 | 0.9989 | 0.5545 |
| 0.4722 | 3.2200 | 7000 | 0.3818 | 1.1015 | 0.2676 | 0.2413 | 0.2384 | 0.2526 | 0.2676 | 0.9986 | 0.5543 |
| 0.4085 | 3.3120 | 7200 | 0.3768 | 1.1077 | 0.2680 | 0.2422 | 0.2387 | 0.2511 | 0.2680 | 0.9987 | 0.5421 |
| 0.4337 | 3.4040 | 7400 | 0.3800 | 1.1032 | 0.2756 | 0.2388 | 0.2335 | 0.2521 | 0.2756 | 0.9987 | 0.5542 |
| 0.4900 | 3.4960 | 7600 | 0.3799 | 1.1099 | 0.2690 | 0.2394 | 0.2409 | 0.2506 | 0.2690 | 0.9986 | 0.5164 |
| 0.5026 | 3.5881 | 7800 | 0.3808 | 1.1082 | 0.2686 | 0.2423 | 0.2366 | 0.2524 | 0.2686 | 0.9984 | 0.5229 |
| 0.4861 | 3.6801 | 8000 | 0.3824 | 1.1076 | 0.2676 | 0.2407 | 0.2382 | 0.2535 | 0.2676 | 0.9985 | 0.5246 |
| 0.5168 | 3.7721 | 8200 | 0.3768 | 1.1151 | 0.2679 | 0.2399 | 0.2432 | 0.2490 | 0.2679 | 0.9988 | 0.5145 |
| 0.4322 | 3.8642 | 8400 | 0.3793 | 1.1163 | 0.2702 | 0.2394 | 0.2407 | 0.2497 | 0.2702 | 0.9989 | 0.5198 |
| 0.5146 | 3.9562 | 8600 | 0.3785 | 1.1114 | 0.2673 | 0.2421 | 0.2392 | 0.2515 | 0.2673 | 0.9986 | 0.5158 |
| 0.4307 | 4.0479 | 8800 | 0.3985 | 1.1146 | 0.2680 | 0.2390 | 0.2417 | 0.2513 | 0.2680 | 0.9987 | 0.4994 |
| 0.3639 | 4.1399 | 9000 | 0.4026 | 1.1089 | 0.2676 | 0.2408 | 0.2418 | 0.2497 | 0.2676 | 0.9988 | 0.4995 |
| 0.5228 | 4.2319 | 9200 | 0.4049 | 1.1114 | 0.2665 | 0.2408 | 0.2423 | 0.2504 | 0.2665 | 0.9987 | 0.4914 |
| 0.4428 | 4.3239 | 9400 | 0.4033 | 1.1194 | 0.2703 | 0.2413 | 0.2392 | 0.2492 | 0.2703 | 0.9986 | 0.4821 |
| 0.4561 | 4.4160 | 9600 | 0.4091 | 1.1195 | 0.2649 | 0.2416 | 0.2443 | 0.2492 | 0.2649 | 0.9988 | 0.4769 |
| 0.4293 | 4.5080 | 9800 | 0.4053 | 1.1076 | 0.2678 | 0.2408 | 0.2403 | 0.2511 | 0.2678 | 0.9985 | 0.4861 |
| 0.3607 | 4.6000 | 10000 | 0.4047 | 1.1197 | 0.2679 | 0.2411 | 0.2418 | 0.2492 | 0.2679 | 0.9987 | 0.4718 |
| 0.4211 | 4.6921 | 10200 | 0.4041 | 1.1204 | 0.2684 | 0.2403 | 0.2408 | 0.2505 | 0.2684 | 0.9987 | 0.4762 |
| 0.4513 | 4.7841 | 10400 | 0.4025 | 1.1133 | 0.2671 | 0.2427 | 0.2403 | 0.2499 | 0.2671 | 0.9986 | 0.4818 |
| 0.4728 | 4.8761 | 10600 | 0.4044 | 1.1175 | 0.2668 | 0.2419 | 0.2424 | 0.2489 | 0.2668 | 0.9988 | 0.4724 |
| 0.3628 | 4.9682 | 10800 | 0.4040 | 1.1168 | 0.2664 | 0.2409 | 0.2431 | 0.2496 | 0.2664 | 0.9988 | 0.4747 |
| 0.3354 | 5.0 | 10870 | 0.4042 | 1.1169 | 0.2668 | 0.2413 | 0.2429 | 0.2490 | 0.2668 | 0.9988 | 0.4745 |