Downloads · 30 days
6
3% of all-time downloads
Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-lbox-switch-top1-router-ffn-lora-aux-anneal-5ep
llama-3.1-8b-instruct-4x1-moe-lbox-switch-top1-router-ffn-lora-aux-anneal-5ep is a text generation model from Jongbin-kr. Use it when you need the model to write or continue text. It is set up for peft.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
6
3% of all-time downloads
All-time downloads
240
Public
Repo size
15.9 GB
Likes
0
Public
Click a slice to open those files.
.safetensors454 MB · 96%
From the Hugging Face model README
This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss | 0 Utilization | 1 Utilization | 2 Utilization | 3 Utilization | Aux Loss | Aux Loss Coef | Aux Loss Weighted | Entropy | Max Load | Z Loss |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.7309 | 0.1391 | 200 | 0.5833 | 0.2458 | 0.2666 | 0.1606 | 0.3270 | 1.0206 | 0.0100 | 0.0102 | 1.2026 | 0.3270 | 2.1134 |
| 0.3526 | 0.2782 | 400 | 0.2207 | 0.2417 | 0.2314 | 0.2422 | 0.2847 | 1.0003 | 0.0100 | 0.0100 | 1.2367 | 0.2847 | 1.2949 |
| 0.2874 | 0.4173 | 600 | 0.1551 | 0.2463 | 0.2286 | 0.2637 | 0.2614 | 1.0014 | 0.0100 | 0.0100 | 1.2412 | 0.2637 | 0.9875 |
| 0.2707 | 0.5564 | 800 | 0.1329 | 0.2456 | 0.2313 | 0.2723 | 0.2508 | 1.0029 | 0.0100 | 0.0100 | 1.2459 | 0.2723 | 0.8069 |
| 0.2531 | 0.6956 | 1000 | 0.1150 | 0.2494 | 0.2384 | 0.2728 | 0.2394 | 1.0041 | 0.0100 | 0.0100 | 1.2527 | 0.2728 | 0.6966 |
| 0.2354 | 0.8347 | 1200 | 0.1102 | 0.2522 | 0.2506 | 0.2628 | 0.2344 | 1.0039 | 0.0100 | 0.0100 | 1.2429 | 0.2628 | 0.6025 |
| 0.2414 | 0.9738 | 1400 | 0.0987 | 0.2523 | 0.2302 | 0.2778 | 0.2397 | 1.0037 | 0.0100 | 0.0100 | 1.2471 | 0.2778 | 0.5361 |
| 0.2122 | 1.1127 | 1600 | 0.0902 | 0.2506 | 0.2488 | 0.2622 | 0.2384 | 1.0030 | 0.0097 | 0.0098 | 1.2436 | 0.2622 | 0.4932 |
| 0.2037 | 1.2518 | 1800 | 0.0848 | 0.2512 | 0.2506 | 0.2602 | 0.2380 | 1.0027 | 0.0094 | 0.0095 | 1.2447 | 0.2602 | 0.4695 |
| 0.1917 | 1.3909 | 2000 | 0.0842 | 0.2584 | 0.2483 | 0.2541 | 0.2392 | 1.0018 | 0.0091 | 0.0091 | 1.2486 | 0.2584 | 0.4339 |
| 0.1926 | 1.5300 | 2200 | 0.0786 | 0.2533 | 0.2455 | 0.2520 | 0.2492 | 1.0004 | 0.0088 | 0.0088 | 1.2441 | 0.2533 | 0.3998 |
| 0.1876 | 1.6691 | 2400 | 0.0754 | 0.2545 | 0.2494 | 0.2531 | 0.2430 | 1.0011 | 0.0085 | 0.0085 | 1.2450 | 0.2545 | 0.3876 |
| 0.1841 | 1.8082 | 2600 | 0.0761 | 0.2612 | 0.2489 | 0.2537 | 0.2361 | 1.0018 | 0.0082 | 0.0082 | 1.2478 | 0.2612 | 0.3662 |
| 0.1733 | 1.9474 | 2800 | 0.0688 | 1.2420 | 0.2556 | 0.2556 | 0.2455 | 0.2541 | 0.2448 | 1.0008 | 0.0079 | 0.0079 | 0.3501 |
| 0.1608 | 2.0862 | 3000 | 0.0694 | 1.2506 | 0.2666 | 0.2497 | 0.2478 | 0.2666 | 0.2359 | 1.0031 | 0.0076 | 0.0076 | 0.3352 |
| 0.1509 | 2.2254 | 3200 | 0.0676 | 1.2419 | 0.2575 | 0.2575 | 0.2495 | 0.2552 | 0.2378 | 1.0015 | 0.0072 | 0.0073 | 0.3297 |
| 0.1544 | 2.3645 | 3400 | 0.0619 | 1.2514 | 0.2525 | 0.2516 | 0.2525 | 0.2473 | 0.2486 | 1.0005 | 0.0069 | 0.0069 | 0.3201 |
| 0.1363 | 2.5036 | 3600 | 0.0637 | 1.2480 | 0.2522 | 0.2522 | 0.2517 | 0.2492 | 0.2469 | 1.0007 | 0.0066 | 0.0066 | 0.3111 |
| 0.1391 | 2.6427 | 3800 | 0.0616 | 1.2526 | 0.2534 | 0.2531 | 0.2478 | 0.2534 | 0.2456 | 1.0005 | 0.0063 | 0.0063 | 0.2999 |
| 0.1287 | 2.7818 | 4000 | 0.0609 | 1.2547 | 0.2566 | 0.2556 | 0.2566 | 0.2520 | 0.2358 | 1.0025 | 0.0060 | 0.0060 | 0.2911 |
| 0.1338 | 2.9209 | 4200 | 0.0581 | 1.2520 | 0.2583 | 0.2541 | 0.2461 | 0.2583 | 0.2416 | 1.0012 | 0.0057 | 0.0057 | 0.2874 |
| 0.1117 | 3.0598 | 4400 | 0.0571 | 1.2506 | 0.2583 | 0.2553 | 0.2442 | 0.2583 | 0.2422 | 1.0009 | 0.0054 | 0.0054 | 0.2806 |
| 0.1044 | 3.1989 | 4600 | 0.0573 | 1.2490 | 0.2520 | 0.2520 | 0.2519 | 0.2516 | 0.2445 | 1.0007 | 0.0051 | 0.0051 | 0.2732 |
| 0.0988 | 3.3380 | 4800 | 0.0562 | 1.2536 | 0.2553 | 0.2553 | 0.25 | 0.2508 | 0.2439 | 1.0009 | 0.0047 | 0.0047 | 0.2667 |
| 0.0909 | 3.4772 | 5000 | 0.0557 | 1.2546 | 0.2567 | 0.2567 | 0.2459 | 0.2542 | 0.2431 | 1.0008 | 0.0044 | 0.0044 | 0.2617 |
| 0.0889 | 3.6163 | 5200 | 0.0557 | 1.2528 | 0.2550 | 0.2550 | 0.2509 | 0.2531 | 0.2409 | 1.0011 | 0.0041 | 0.0041 | 0.2589 |
| 0.0861 | 3.7554 | 5400 | 0.0533 | 1.2549 | 0.2553 | 0.2553 | 0.2448 | 0.2552 | 0.2447 | 1.0006 | 0.0038 | 0.0038 | 0.2531 |
| 0.0768 | 3.8945 | 5600 | 0.0521 | 1.2566 | 0.2547 | 0.2547 | 0.2536 | 0.2483 | 0.2434 | 1.0008 | 0.0035 | 0.0035 | 0.2505 |
| 0.0653 | 4.0334 | 5800 | 0.0516 | 1.2561 | 0.2555 | 0.2555 | 0.25 | 0.2516 | 0.2430 | 1.0008 | 0.0032 | 0.0032 | 0.2482 |
| 0.0627 | 4.1725 | 6000 | 0.0516 | 1.2552 | 0.2570 | 0.2519 | 0.2570 | 0.2470 | 0.2441 | 1.0011 | 0.0029 | 0.0029 | 0.2441 |
| 0.0552 | 4.3116 | 6200 | 0.0505 | 1.2588 | 0.2536 | 0.2536 | 0.2519 | 0.2512 | 0.2433 | 1.0009 | 0.0025 | 0.0026 | 0.2404 |
| 0.0533 | 4.4507 | 6400 | 0.0510 | 1.2591 | 0.2528 | 0.2528 | 0.2523 | 0.2522 | 0.2427 | 1.0010 | 0.0022 | 0.0022 | 0.2384 |
| 0.0443 | 4.5898 | 6600 | 0.0497 | 1.2600 | 0.2548 | 0.2534 | 0.2505 | 0.2548 | 0.2412 | 1.0012 | 0.0019 | 0.0019 | 0.2373 |
| 0.0381 | 4.7289 | 6800 | 0.0486 | 1.2613 | 0.2548 | 0.2548 | 0.2519 | 0.2528 | 0.2405 | 1.0012 | 0.0016 | 0.0016 | 0.2365 |
| 0.0338 | 4.8681 | 7000 | 0.0481 | 1.2599 | 0.2541 | 0.2541 | 0.2536 | 0.2528 | 0.2395 | 1.0014 | 0.0013 | 0.0013 | 0.2346 |
| 0.0280 | 5.0 | 7190 | 0.0476 | 1.2596 | 0.2548 | 0.2527 | 0.2548 | 0.2522 | 0.2403 | 1.0014 | 0.0010 | 0.0010 | 0.2350 |