Downloads · 30 days
9
0% of all-time downloads
pretrain-wura
llama-lang-adapt/pretrain-wura
pretrain-wura is a text generation model from llama-lang-adapt. Use it when you need the model to write or continue text. It is set up for transformers.
We continual pre-train meta-llama/Llama-2-7b-hf on monolingual WURA corpus for 20 languages. All languages are uniformly sampled.
All-time downloads
3.6K
Public
Repo size
27 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin27 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- llama
- Access
- Public
- Created
- Jan 26, 2024
- Updated
- Jan 26, 2024
- SHA
- 188b1cec
Try a prompt
Datasets
Model card
From the Hugging Face model README
We continual pre-train meta-llama/Llama-2-7b-hf on monolingual WURA corpus for 20 languages. All languages are uniformly sampled.
Important Parameters
- num_gpus: 8
- max_steps: 8000 # see here
- gradient_accumulation_steps: 16
- per_device_batch_size: 2
- learning_rate: 2e-5
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- Created
- Jan 26, 2024
- Updated
- Jan 26, 2024