Downloads · 30 days
10
3% of all-time downloads
qing-yao/babylm-opt-sva-decay-0.1-from8000-11473589
babylm-opt-sva-decay-0.1-from8000-11473589 is a text generation model from qing-yao. Use it when you need the model to write or continue text. It is set up for transformers.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
10
3% of all-time downloads
All-time downloads
319
Public
Parameters
110M
6.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.pt4.4 GB · 62%
From the Hugging Face model README
This model is a fine-tuned version of models/babylm-default_seed-42_1e-3 on the qing-yao/slightly-cleaner-babylm dataset. It achieves the following results on the evaluation set:
More information needed
More information needed
More information needed
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen |
|---|---|---|---|---|
| 7.0783 | 0.1371 | 250 | 7.0315 | 16384000 |
| 5.3787 | 0.2741 | 500 | 5.3901 | 32768000 |
| 4.9499 | 0.4112 | 750 | 4.9824 | 49152000 |
| 4.8444 | 0.5482 | 1000 | 4.7563 | 65536000 |
| 4.6264 | 0.6853 | 1250 | 4.5572 | 81920000 |
| 4.4358 | 0.8224 | 1500 | 4.3957 | 98304000 |
| 4.3009 | 0.9594 | 1750 | 4.2668 | 114688000 |
| 4.1937 | 1.0965 | 2000 | 4.1604 | 131029760 |
| 4.0318 | 1.2336 | 2250 | 4.0687 | 147413760 |
| 4.0821 | 1.3706 | 2500 | 3.9881 | 163797760 |
| 4.0435 | 1.5077 | 2750 | 3.9142 | 180181760 |
| 3.8267 | 1.6447 | 3000 | 3.8492 | 196565760 |
| 3.8228 | 1.7818 | 3250 | 3.7909 | 212949760 |
| 3.8113 | 1.9189 | 3500 | 3.7387 | 229333760 |
| 3.6935 | 2.0559 | 3750 | 3.6923 | 245675520 |
| 3.7052 | 2.1930 | 4000 | 3.6491 | 262059520 |
| 3.6055 | 2.3300 | 4250 | 3.6096 | 278443520 |
| 3.616 | 2.4671 | 4500 | 3.5757 | 294827520 |
| 3.5368 | 2.6042 | 4750 | 3.5387 | 311211520 |
| 3.5042 | 2.7412 | 5000 | 3.5091 | 327595520 |
| 3.5091 | 2.8783 | 5250 | 3.4801 | 343979520 |
| 3.4449 | 3.0154 | 5500 | 3.4553 | 360321280 |
| 3.4085 | 3.1524 | 5750 | 3.4325 | 376705280 |
| 3.3963 | 3.2895 | 6000 | 3.4129 | 393089280 |
| 3.4197 | 3.4265 | 6250 | 3.3905 | 409473280 |
| 3.4021 | 3.5636 | 6500 | 3.3745 | 425857280 |
| 3.3401 | 3.7007 | 6750 | 3.3566 | 442241280 |
| 3.366 | 3.8377 | 7000 | 3.3367 | 458625280 |
| 3.3422 | 3.9748 | 7250 | 3.3252 | 475009280 |
| 3.2944 | 4.1118 | 7500 | 3.3128 | 491351040 |
| 3.4211 | 4.2489 | 7750 | 3.3001 | 507735040 |
| 3.2671 | 4.3860 | 8000 | 3.2890 | 524119040 |
| 3.2172 | 4.5230 | 8250 | 3.2819 | 540503040 |
| 3.2342 | 4.6601 | 8500 | 3.2685 | 556887040 |
| 3.272 | 4.7971 | 8750 | 3.2604 | 573271040 |
| 3.2223 | 4.9342 | 9000 | 3.2518 | 589655040 |
| 3.1633 | 5.0713 | 9250 | 3.2433 | 605996800 |