Downloads · 30 days
16
3% of all-time downloads
pythia-70m-ce-matched
ddgoede/pythia-70m-ce-matched
pythia-70m-ce-matched is a machine learning model from ddgoede. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A Pythia-70M model trained from scratch with cross-entropy on the deduplicated Pile, using the exact training code, data pipeline, seed and schedule as ddgoede/pythia-70m-distilled-from-410m, with the distillation ter…
All-time downloads
615
Public
Parameters
70.4M
43.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors282 MB · 100%
At a glance
- License
- apache-2.0
- Model type
- gpt_neox
- Access
- Public
- Created
- Aug 17, 2026
- Updated
- Aug 17, 2026
- SHA
- 2c8d0cf0
Base models
Datasets
Model card
From the Hugging Face model README
Pythia-70M, cross-entropy, recipe-matched control
A Pythia-70M model trained from scratch with cross-entropy on the deduplicated
Pile, using the exact training code, data pipeline, seed and schedule as
ddgoede/pythia-70m-distilled-from-410m,
with the distillation term switched off (alpha = 0).
Its purpose is to serve as a controlled baseline: comparing it against the
distilled student isolates the effect of the loss function, since every other
part of the recipe is shared. The public EleutherAI/pythia-70m-deduped
checkpoint is trained by a different recipe (LR decayed to a 0.1x floor rather
than to zero, fp16 rather than bf16, weight decay 0.1 rather than 0.01,
gradient clipping, and a different global data order), so a comparison against
it confounds the loss with the recipe.
Checkpoints
One branch (revision) per checkpoint step, following the Pythia schedule:
step1, step2, step4, ..., step512, then step1000 through step143000 in
steps of 1000 (153 revisions).
from transformers import GPTNeoXForCausalLM, AutoTokenizer
model = GPTNeoXForCausalLM.from_pretrained(
"ddgoede/pythia-70m-ce-matched", revision="step143000"
)
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-70m-deduped")
Training details
| Objective | cross-entropy (alpha = 0) |
| Data | EleutherAI/the_pile_deduplicated, streamed, shuffle buffer 10,000, seed 42 |
| Steps | 143,000 |
| Batch | 1024 sequences x 2048 tokens (2M tokens/step, ~293B tokens total) |
| Optimizer | AdamW, lr 1e-3, betas (0.9, 0.95), weight decay 0.01 |
| Schedule | linear warmup 1430 steps, cosine decay to zero |
| Precision | bf16 mixed precision |
| Hardware | 16x A5000 (2 nodes) |
Training code: training/llm/train_distill.py in the
wiggle-room-hypothesis-project
repository.
- Type
- gpt_neox
- License
- apache-2.0
- Languages
- en
- Created
- Aug 17, 2026
- Updated
- Aug 17, 2026