Downloads · 30 days
17
18% of all-time downloads
horizon-pythia-ft-1b
crumb/horizon-pythia-ft-1b
horizon-pythia-ft-1b is a text generation model from crumb. Use it when you need the model to write or continue text. It is set up for transformers.
All-time downloads
95
Public
Repo size
8.1 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin4 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- gpt_neox
- Access
- Public
- Created
- Nov 9, 2023
- Updated
- Nov 10, 2023
- SHA
- 7b13d755
Try a prompt
Model card
From the Hugging Face model README
See the blog post!
pythia-1b "reweighted" (very short continued pretraining) with a mix of data from the crumbly Horizon dataset found via crumb's "Paying 'Attention' to your Dataset" method on pythia-70m -> pythia-160m -> and pythia-410m at different stages.
hparams:
LR: 1e-5
SCHEDULE: cosine with 20% warmup from 0, cooldown to 0
BS: 64
CTX: 2048
everything else you can think of is set to it's default value in the huggingface trainer
validation loss (loss on data not used in training, lower is better)
| model | arxiv | github | books | wiki | webtext |
|---|---|---|---|---|---|
| horizon-pythia-ft-1b | 2.13 | 1.30 | 2.00 | 2.22 | 2.71 |
| pythia-1b | 2.21 | 1.30 | 2.02 | 2.29 | 2.72 |
optimized mixture after 12 training runs, starting with 100 samples each:
| subset | documents |
|---|---|
| arxiv | 608 |
| github | 226 |
| books | 613 |
| wiki | 1438 |
| webtext | 8516 |
benchmarks that i can actually run in reasonable time:
| model | arc | truthfulqa | winogrande |
|---|---|---|---|
| horizon-pythia-ft-1b | 28.24 | 41.13 | 53.75 |
| pythia-1b deduped* | 29.1 | 38.94 | 53.59 |
| the actual pythia model isnt on the leaderboard and im really tired and dont want to open the eval script again and wait |
- Task
- Text Generation
- Library
- transformers
- Type
- gpt_neox
- Languages
- en
- Created
- Nov 9, 2023
- Updated
- Nov 10, 2023