Downloads · 30 days
169
1% of all-time downloads
Mixsmol-4x400M-v0.1-epoch2
vilm/Mixsmol-4x400M-v0.1-epoch2
Mixsmol-4x400M-v0.1-epoch2 is a text generation model from vilm. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This is the first checkpoint (Epoch 1) of Mixsmol-4x400M-v0.1 Note that this is an experimental in data mixing. Therefore, we only trained the model on 50B tokens (95% English and 5% Vietnamese) to test the following:…
All-time downloads
11.9K
Public
Parameters
1.8B
3.5 GB on disk
Likes
5
Public
Repo makeup
Click a slice to open those files.
.safetensors3.5 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mixtral
- Access
- Public
- Created
- Jan 28, 2024
- Updated
- Jan 30, 2024
- SHA
- 3741dbdb
Try a prompt
Model card
From the Hugging Face model README
Mixsmol-4x400M-v0.1 by Ontocord
This is the first checkpoint (Epoch 1) of Mixsmol-4x400M-v0.1 Note that this is an experimental in data mixing. Therefore, we only trained the model on 50B tokens (95% English and 5% Vietnamese) to test the following:
- Reasoining capabilities through high-quality synthetic textbooks data pretraining
- Crosslingual understanding through machine translation and multilingual + multiple tasks pretraining
After verifying our hypothesis with this run, we will schedule a second run on bigger data and compute for it to achieve its maximum capability.
Data
- Synthetic Textbooks: 8M samples
- RefinedWeb: 1M samples
- RedPajama-v2: 500K samples
- MathPile: Everything
- ThePile: MiniPile Subset
- GoodWiki
- The Stack Smol XL
- The Vault: train_small split
- Instruction Pretraining: 250k samples
- Task
- Text Generation
- Library
- transformers
- Type
- mixtral
- License
- apache-2.0
- Created
- Jan 28, 2024
- Updated
- Jan 30, 2024