Downloads · 30 days
0
smalLM-v0.1
KickItLikeShika/smalLM-v0.1
smalLM-v0.1 is a machine learning model from KickItLikeShika. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
- W&B run: https://wandb.ai/kickitlikeshika-cision/smallLM/runs/gxtldf56?nw=nwuserkickitlikeshika - Tokenizer: GPT-2 Pre-Trained Tokenizer - Hellaswag Acc: 0.3778 - Val Loss: 2.99 - Training Loss: 3.03 - Training Data…
Access
Public
Updated Oct 28, 2024
Repo size
3.3 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin1.8 GB · 100%
At a glance
- Access
- Public
- Created
- Aug 17, 2024
- Updated
- Oct 28, 2024
- SHA
- 72ea20f0
Model card
From the Hugging Face model README
SmalLM V0.1
- W&B run: https://wandb.ai/kickitlikeshika-cision/smallLM/runs/gxtldf56?nw=nwuserkickitlikeshika
- Tokenizer: GPT-2 Pre-Trained Tokenizer
- Hellaswag Acc: 0.3778
- Val Loss: 2.99
- Training Loss: 3.03
- Training Data: Fineweb EDU 10B tokens sample
- Training Steps: 19073
- LR: 6e-4
- Seq Len: 1024
- Vocab Size: 50304
- Number of Layers: 32
- Number of Attention Heads: 32
- Embedding Dim: 1024
- Model Params: 450M
- Created
- Aug 17, 2024
- Updated
- Oct 28, 2024