Downloads · 30 days
3
4% of all-time downloads
simple-100m-pretrain-v1
Dodosoomro/simple-100m-pretrain-v1
simple-100m-pretrain-v1 is a machine learning model from Dodosoomro. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is a 120M parameter decoder-only transformer trained from scratch on 1 billion tokens of high-quality web data, math, and code.
All-time downloads
76
Public
Parameters
146M
292 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors292 MB · 100%
At a glance
- Model type
- llama
- Access
- Public
- Created
- Apr 14, 2026
- Updated
- Apr 14, 2026
- SHA
- 43543b75
Model card
From the Hugging Face model README
simple-100m-pretrain-v1
This is a 120M parameter decoder-only transformer trained from scratch on 1 billion tokens of high-quality web data, math, and code.
Model Details
- Architecture: Llama-based (32 Layers)
- Attention: Grouped Query Attention (GQA) 2:1 Ratio
- Hidden Size: 512
- Intermediate Size: 1408
- Context Length: 1024
- Tokenizer: GPT-2
- Total Tokens: 1 Billion
Training Configuration
- Hardware: TPU v5e-8 (Kaggle)
- Optimizer: AdamW (Peak LR: 5e-4)
- Schedule: Cosine decay with 2% warmup
- Precision: bfloat16
Benchmarks (Zero-Shot)
| Metric | Task | Value |
|---|---|---|
| Perplexity | Training-Validation | 35.04 |
| Accuracy | ARC-Easy | 31.10% |
| Accuracy | PIQA | 54.13% |
| Accuracy | HellaSwag | 26.22% |
| Accuracy | TruthfulQA (MC1) | 27.42% |
| Accuracy | GSM8K (5-shot) | 1.36% |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Dodosoomro/simple-100m-pretrain-v1")
tokenizer = AutoTokenizer.from_pretrained("Dodosoomro/simple-100m-pretrain-v1")
prompt = "The future of artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))
- Type
- llama
- Created
- Apr 14, 2026
- Updated
- Apr 14, 2026