Aetheris-MoE-300M-A125M-base

Aetheris: Hybrid Mamba-MoE (294M)

Developed by: Pomilon Intelligence Lab Status: 🟡 Experimental / Research Preview Source Code: GitHub - Pomilon-Intelligence-Lab/Aetheris

Aetheris is an experimental language model exploring the intersection of State Space Models (SSM) and Mixture-of-Experts (MoE) architectures.

We designed this model to investigate efficient scaling on consumer hardware. By combining Mamba's linear-time sequence modeling with the sparse capacity of MoE, Aetheris aims to maximize parameter count while minimizing inference latency.

🧪 Architecture & Design

This project tests the hypothesis: Can we interleave dense Mamba blocks with sparse MoE layers to create a model that is parameter-rich but computationally light?

The architecture follows a strict alternating pattern:

1. SSM Blocks (Odd Layers): Dense Mamba blocks responsible for sequence mixing and memory.
2. MoE Blocks (Even Layers): Sparse router layers that direct tokens to 1 of 4 experts.

📊 Technical Specifications

Due to the sparse nature of the MoE layers, approximately 43% of the parameters remain inactive during any given inference step.

📉 Training Status

Training is currently in progress on a single NVIDIA RTX 5000.

• Current Step: 17,000
• Current Loss: ~1.8167
• Dataset: A subset of SlimPajama-627B

⚠️ Performance Notice: Aetheris is currently in a "proof-of-concept" state. While it generates coherent English syntax, it does not yet possess strong reasoning capabilities. It is intended for architectural analysis rather than downstream tasks.

🚀 Usage & Inference

Since Aetheris utilizes a custom architecture not yet supported by standard Transformers, you must use the custom inference code provided in our repository.

# 1. Clone the repository
git clone https://github.com/Pomilon/Aetheris.git
cd Aetheris

# 2. Install dependencies
pip install -r requirements.txt

# 3. Run generation (Ensure you have downloaded the model weights first)
python -m aetheris.cli.main generate --prompt "The future of AI is" --checkpoint_dir path/to/checkpoints_folder

📚 Acknowledgements

This research builds upon foundational work in the field:

• Mamba: Gu & Dao (2023)
• Mixture of Experts: Shazeer et al. (2017)

---

📝 Author's Note (The "Scuffed" Reality)

Hi, Pomilon here! While the text above sounds official, here is the reality:

This is a "learning by doing" experiment. I wanted to see if I could smash these two architectures together on my laptop without it exploding. I built this from scratch to learn, so don't expect GPT-4 performance! It's currently in the "babbling coherently" phase.

If you manage to break it (or fix it), let me know!