CreekwardGoat-500K

CreekwardGoat-500K

CreekwardGoat-500K is a compact, decoder-only transformer developed to explore the capabilities of tiny language models. Despite its remarkably small scale of around 500 thousand parameters, it achieves very competitive performance within its size class on established evaluation benchmarks, including BLiMP and ARC-Easy.

Model Details

Benchmarks

All results were evaluated using the lm-evaluation-harness.

Usage

Inference

Run:

python inference.py

Training / Reproduction

This model is fully transparent in training and reproducible from scratch.

Download the training data:

python get_training_data.py

Start training:

python train.py

Training configurations including seed and number of training tokens can be modified in config.py.

If you leave the files unchanged, theoretically, this should reproduce the training trajectory as well as the final model output.

Training Data

The model was trained on a 300 million token subset of FineWeb-Edu-Dedup, taken from the HuggingFaceTB/smollm-corpus repository. The specific shard was selected due to its relatively small size to reduce storage and preprocessing overhead.

• Source repository: HuggingFaceTB/smollm-corpus
• Dataset license: Open Data Commons Attribution License (ODC-By) 1.0

Limitations & Disclaimer

This model is an experimental personal research artifact and is not intended for production or consumer use. It is released as a base (pre-trained) model and does not support conversational interaction. Due to its extremely small scale (≈500K parameters), this model is fundamentally incapable of reliable output. It may generate:

• Factually inaccurate or hallucinated content

• Grammatically garbled or nonsensical text

• Unexpected or biased outputs

Generated outputs do not represent the author's views. Users are solely responsible for verifying outputs, ensuring compliance with all applicable laws, and bearing all consequences of use or redistribution. The author accepts no responsibility for model outputs or downstream use.

Citations

• Vaswani et al., Attention Is All You Need
• Kim et al., Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Shazeer, GLU Variants Improve Transformer
• Zhang and Sennrich, Root Mean Square Layer Normalization
• Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
• Gao et al., A Framework for Few-Shot Language Model Evaluation, Zenodo, v0.4.0, 2023.