Downloads · 30 days
3
6% of all-time downloads
orion-atlas-1b
asvuep/orion-atlas-1b
orion-atlas-1b is a machine learning model from asvuep. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Cendrix AI — orion-atlas-7b | GitHub | Paper
All-time downloads
50
Public
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.py11 KB · 71%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Mar 24, 2026
- Updated
- Mar 24, 2026
- SHA
- bf39d1d1
Model card
From the Hugging Face model README
Orion Atlas 1B
Cendrix AI — orion-atlas-7b | GitHub | Paper
Status
🟡 Training in progress — Currently at iter ~11,000 / 50,000. Weights will be published upon completion (~March 25, 2026).
This repository contains the model architecture, config, and training details. Weights incoming.
Architecture
Orion Atlas 1B is the first model in the Orion Atlas family — a custom transformer architecture built from scratch by Cendrix AI.
| Property | Value |
|---|---|
| Parameters | 1,147,766,784 (~1.15B) |
| Layers | 24 |
| Attention heads | 16 (GQA: 4 KV heads) |
| Hidden dim | 2048 |
| FFN | SwiGLU |
| Normalization | RMSNorm |
| Position | RoPE with YaRN context extension |
| Context | 4,096 tokens (extensible via YaRN) |
| Tokenizer | Custom SentencePiece, 32K vocab |
| Architecture | Mini-Llama style: RoPE, GQA, SwiGLU, RMSNorm, Flash Attention |
Model Family
| Model | Params | Status |
|---|---|---|
| Orion Atlas 1B | 1.15B | 🟡 Training |
| Orion Atlas 3B | ~3B | 📋 Planned |
| Orion Atlas 7B | 8.77B | 🏗️ Architecture released |
| Orion Atlas 14B | ~14B | 📋 Planned |
| Orion Atlas 37B | ~37B | 📋 Planned |
The 7B model uses a Mamba-2 Hybrid + Differential Attention architecture — the first known combination of these techniques. See orion-atlas-7b.
Training
- Pre-training data: FineWeb-Edu, SlimPajama, StarCoder, The Stack, OpenWebMath, Cosmopedia, GPT4All, Tulu-SFT (~17B tokens)
- Hardware: NVIDIA H100 80GB HBM3 (RunPod)
- Framework: Custom PyTorch training loop
- Optimizer: AdamW, lr=3e-4, cosine decay
- Batch: 40 × 2048 tokens (effective ~82K tokens/step)
- Speed: ~50,000 tok/s
Design Goals
Built for agentic tasks — tool calling, structured JSON output, multi-step reasoning. The 1B serves as the seed model for progressive distillation up to the 37B flagship.
Usage
This model uses a custom architecture (not HuggingFace Transformers compatible). See model.py for inference code.
# Coming with weights release
Citation
@misc{palermini2026orionatlas,
title={Orion Atlas: A Mamba-2 Hybrid Architecture with Differential Attention for Agentic Language Models},
author={Avery Palermini},
year={2026},
institution={Cendrix AI},
}
© 2026 Cendrix AI · Grove City, OH
- License
- apache-2.0
- Languages
- en
- Created
- Mar 24, 2026
- Updated
- Mar 24, 2026