Downloads · 30 days
256
75% of all-time downloads
Mila_1L
Ffohturk/Mila_1L
Mila_1L is a text generation model from Ffohturk. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
This is a one-layer base model with the LlaMA 2 architecture trained on 6B tokens of the algebraic-stack part of the Proof-pile 2 dataset. \ It's output distribution is thus mostly concerned with code. The tokenizer i…
All-time downloads
342
Public
Parameters
37M
74.4 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors73.9 MB · 99%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- llama2
- Model type
- llama
- Access
- Public
- Created
- Apr 3, 2024
- Updated
- Apr 3, 2024
- SHA
- 640fc4a4
Try a prompt
Datasets
Model card
From the Hugging Face model README
This is a one-layer base model with the LlaMA 2 architecture trained on 6B tokens of the algebraic-stack part of the Proof-pile 2 dataset.
It's output distribution is thus mostly concerned with code.
The tokenizer is the LlaMA 2 one. I used the following hyper parameters:
d<sub>model</sub> = 512
d<sub>ff</sub> = 2048
n<sub>heads</sub> = 4
n<sub>ctx</sub> = 1024
For the training I used AdamW with weight decay = 0.05 and cosine annealing with 5000 warmup steps and maximum learning rate 1e-4. We used BF16 precision.
Train loss: 2.6228
Test loss: 2.7490
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- llama2
- Created
- Apr 3, 2024
- Updated
- Apr 3, 2024