Aris-375M-v0.1

Aris-375M-v0.1

Aris-375M-v0.1 is a 375 million parameter decoder-only language model trained from scratch using the Aris 96K tokenizer.

This release serves as the initial public research checkpoint of the Aris model family.

Model Details

• Parameters: ~375M
• Architecture: Llama-style causal decoder
• Tokenizer: joseph-ai/Aris-Tokenizer-96K
• Context length: 4096
• Training: packed-token pretraining
• Training tokens: approximately 5B

Evaluation

The model was evaluated on FineWeb-Edu validation data and standard zero-shot language model benchmarks.

Zero-shot lm-eval results:

Intended Use

This model is intended for research into small language model pretraining, data mixture design, and tokenizer design.

Limitations

This is a base model and has not been instruction tuned. It may produce incorrect, repetitive, biased, or unsafe text. It should not be used for production decision-making or user-facing assistant behavior without further safety and alignment training.

Training Procedure

The model was trained with packed sequences using multi-GPU bf16 training.

Key hyperparameters from the final training stage:

• learning_rate: 5e-05
• train_batch_size: 2
• eval_batch_size: 2
• num_devices: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 64
• optimizer: AdamW fused
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100

Framework Versions

• Transformers 5.12.1
• PyTorch 2.4.1+cu124
• Datasets 5.0.0
• Tokenizers 0.22.2