zoof-v1.2-394M

Zoof-394M (v1.2.2)

<div align="center">

A clean, optimized, and interpretable implementation of a decoder-only Transformer in PyTorch.

</div>

Model Summary

zoof-394M (v1.2.2) is a general-purpose language model pre-trained on 79 Billion tokens of high-quality educational data. It demonstrates how modern architectural choices and high-quality data can yield competitive performance in the sub-400M parameter regime, even with limited compute.

Despite its compact size, zoof-394M achieves competitive zero-shot performance on reasoning and commonsense benchmarks, often rivaling models trained on significantly larger datasets.

💻 Usage

Please refer to the Zoof repository.

Training Data

• Pre-Training: fineweb-edu

📊 Performance & Benchmarks

The following table compares zoof-v1.2-394M (in zero shot setting) against industry-standard baselines in the 300-500M parameter class.

Note: Zoof achieves these scores with ~2% of the training compute used for SmolLM2 (79B vs 4T tokens), highlighting the efficiency of the FineWeb-Edu dataset.

⚙️ Technical Specifications

• Architecture: Decoder-only Transformer with RMSNorm (Pre-Norm).
• Positional Embeddings: Rotary Positional Embeddings (RoPE).
• Attention: Flash Attention via F.scaled_dot_product_attention.
• Initialization: Depth-scaled initialization ($1/\sqrt{2L}$) for residual stability.
• Context Length: 1024 Tokens.
• Parameters: 394M.
• Chat Model: Zoof-v1.2-394M-chat
• Language: English