Downloads · 30 days
0
0% of all-time downloads
sc2-1b-data-ablation
RaymondLi/sc2-1b-data-ablation
sc2-1b-data-ablation is a machine learning model from RaymondLi. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
1B-parameter models trained on Python-only datasets. In the different branches, models are trained on different versions of the Stack: - stack v1 - stack v2 - permissive - stack v2 - permissive and unlicensed
All-time downloads
85
Public
Repo size
79 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.1 MB · 100%
At a glance
- Access
- Public
- Created
- Oct 10, 2023
- Updated
- Oct 11, 2023
- SHA
- 1ac955a5
Model card
From the Hugging Face model README
1B-parameter models trained on Python-only datasets. In the different branches, models are trained on different versions of the Stack:
- stack v1
- stack v2 - permissive
- stack v2 - permissive and unlicensed
24 layers, a hidden-size of 2048 and 16 attention heads (multiquery). The learning-rate is set to $4\times10^{-4}$ after a warmup of $1000$ steps and follows a cosine decay to $4\times10^{-5}$ at the end of training. Trained with a batch size of 128 samples of 8192 tokens each, for $100$k iterations, such that the model sees $100$B tokens at the end of training. We use a FIM-rate of $0.5$, the same tokenizer as StarCoder (except for tokenizer ablations) and learned absolute positional embeddings.
- Created
- Oct 10, 2023
- Updated
- Oct 11, 2023