zoof-v1.2-394M-chat

Zoof-394M-Chat (v1.2.2)

<div align="center">

The instruction-tuned variant of the zoof-v1.2-394M Small Language Model.

</div>

Model Summary

zoof-394M-chat is the instruction-tuned version of zoof-394M. It has been fine-tuned using Supervised Fine-Tuning (SFT) to follow instructions, answer questions, and engage in dialogue.

This model retains the efficiency of the base model while offering significantly improved coherence and adherence to user prompts.

💻 Usage

Please refer to the Zoof repository.

Training Data

• Pre-Training: fineweb-edu
• SFT: No Robots, WildChat-4.8M, Magpie Pro 300k Filtered and WizardLM Evol Instruct 70k.

📊 Performance & Benchmarks

The following table compares zoof-v1.2-394M (base model) against industry-standard baselines in the 300-500M parameter class.

Note: Zoof achieves these scores with ~2% of the training compute used for SmolLM2 (79B vs 4T tokens), highlighting the efficiency of the FineWeb-Edu dataset.

⚙️ Technical Specifications

• Architecture: Decoder-only Transformer with RMSNorm (Pre-Norm).
• Positional Embeddings: Rotary Positional Embeddings (RoPE).
• Attention: Flash Attention via F.scaled_dot_product_attention.
• Initialization: Depth-scaled initialization (1/\sqrt{2L}) for residual stability.
• Context Length: 1024 Tokens.
• Parameters: 394M.
• Base Model: zoof-v1.2-394M
• Language: English