phoenix-350m-base

Phoenix-350M

A 350M-parameter decoder-only language model trained from scratch on ~10B tokens of FineWeb-Edu, on a single consumer GPU (RTX 3080 Ti, 12 GB). Released as a base model (no instruction tuning).

The architecture is a modern LLaMA/Qwen-style decoder, and the weights are provided in the stock Qwen3 format so they load with plain transformers (and export cleanly to ONNX / vLLM).

Highlights

• Chinchilla-optimal (~28.5 tokens/param) — trained to the compute-efficient point for its size.
• Punches above its token weight on knowledge/reasoning: on ARC it matches or beats models trained on 30×–180× more data, thanks to FineWeb-Edu's dense educational content.
• Final validation perplexity 15.6 (~0.86 bits/byte on its own held-out set).

Architecture

Training

• Data: FineWeb-Edu (sample-10BT), ~9.99B tokens, one epoch.
• Optimizer: 8-bit AdamW (bitsandbytes), lr 6e-4, WSD schedule (2000 warmup, final-10% linear decay).
• Precision: bf16, gradient checkpointing, global batch 524,288 tokens.
• Hardware: 1× RTX 3080 Ti (12 GB), ~7 days.

Evaluation (0-shot, lm-evaluation-harness, acc_norm where conventional)

Bits-per-byte (same-corpus, WikiText-103): Phoenix 0.98 vs Pythia-410M 0.86 vs Pythia-160M 1.19.

How to read this: Phoenix trained on 10B tokens — 30×–400× less data than the peers above. It's competitive-to-better on ARC (knowledge/reasoning, where FineWeb-Edu shines — it beats Apple's OpenELM-450M on both ARC-easy and ARC-challenge despite 180× less data), but its smaller token budget shows on HellaSwag, PIQA, WinoGrande and language-modeling metrics (BPB, LAMBADA), which scale with raw token count. MMLU is ~random at this scale, as expected for a 350M base model.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("shreyash-pandey-katni/phoenix-350m-base")
model = AutoModelForCausalLM.from_pretrained("shreyash-pandey-katni/phoenix-350m-base")
ids = tok("The process of photosynthesis begins when", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60)[0], skip_special_tokens=True))

ONNX weights are provided under onnx/ for optimum / onnxruntime / transformers.js.

Limitations

• Base model — not instruction-tuned or aligned; it completes text, it doesn't follow chat instructions.
• English only, ~10B tokens — limited world knowledge; will hallucinate and repeat.
• No safety/RLHF filtering. Use for research and as a small-model baseline, not production.

License

Apache-2.0. Trained on FineWeb-Edu (ODC-BY).