axion-250m

Axion 250M

Axion 250M is a 234,914,816-parameter decoder-only causal language model trained from scratch on 5 billion tokens from FineWeb-Edu. It is an experimental base model and has not been instruction-tuned or safety-aligned.

Architecture

The implementation uses RMSNorm, causal scaled dot-product attention, GELU feed-forward layers, and untied input/output embeddings. The model was trained with a custom three-node PyTorch pipeline, with one RTX 3090 per node.

Training and Evaluation

• Training corpus: HuggingFaceFW/fineweb-edu, sample-10BT configuration
• Training tokens: 5,000,003,584 processed
• Validation split: final 10 million tokens reserved from the prepared corpus
• Final validation cross-entropy: 3.0035
• Final validation perplexity: 20.16
• Best observed validation cross-entropy: 2.9967
• Best observed validation perplexity: 20.02

The reported final score used 8,192 held-out tokens over four evaluation batches. These measurements are project-internal language-modeling metrics and are not benchmark scores.

A larger post-training evaluation over 524,288 held-out tokens produced a cross-entropy of 3.0709 and perplexity of 21.56. This larger sample is the more reliable estimate of general validation performance.

Standard Benchmarks

The full evaluation splits were run zero-shot with lm-eval 0.4.12 using BF16 inference. Normalized accuracy is reported for multiple-choice tasks where the harness provides it.

The exact task versions, sample counts, standard errors, and runner settings are stored in evaluation/lm_eval_0_4_12_zero_shot.json.

Usage

This release uses the native Axion PyTorch architecture. It is not compatible with transformers.AutoModelForCausalLM.

git clone https://huggingface.co/ikdataml/axion-250m
cd axion-250m
python -m pip install torch pyyaml sentencepiece safetensors
python generate.py --prompt "The universe" --max-new-tokens 80

The weights are stored in model.safetensors. The exact model and training configuration is in configs/axion_250m.yaml, and the tokenizer is in tokenizer/axion16k.model.

Limitations

This is a small experimental base model trained on web text. It can produce incorrect, repetitive, biased, unsafe, or incoherent text. It should not be used for factual, medical, legal, financial, or safety-critical decisions. Users are responsible for evaluating outputs for their application.

License

The model code and weights are released under the MIT License. FineWeb-Edu is an external training dataset and remains subject to its own terms and source content licenses.