Downloads · 30 days
62
15% of all-time downloads
Kiy-K/moe-gpt-neox-125m-v1.5
moe-gpt-neox-125m-v1.5 is a text generation model from Kiy-K. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Mixture of Experts Enhanced Language Model
Downloads · 30 days
62
15% of all-time downloads
All-time downloads
402
Public
Parameters
162M
3.8 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors704 MB · 99%
From the Hugging Face model README
Mixture of Experts Enhanced Language Model
This is an efficient Mixture of Experts (MoE) enhancement of the GPT-NeoX-125M architecture. By adding sparse expert routing, this model achieves better performance with significantly lower computational costs compared to traditional dense models.
| Feature | Specification |
|---|---|
| Base Model | Kiy-K/moe-gpt-neox-125m-v1.1 |
| Architecture | MoE-Enhanced GPT-NeoX |
| Total Parameters | ~125M (base) + MoE layers |
| MoE Experts | 8 specialized experts |
| Top-K Routing | 2 (sparse activation) |
| Training Method | Frozen base + MoE layer training |
| Training Hardware | Kaggle 2x Tesla T4 GPUs |
| Format | 🔒 Safetensors (secure) |
The model enhances every other transformer layer with a MoE routing system:
Input Token → Gating Network → Route to Top-2 Experts → Weighted Combination → Output
Key Components:
Advantages:
Training Steps: 6,000
Batch Size: 16
Gradient Accumulation: 2
Effective Batch Size: 96
Learning Rate: 1e-4
Warmup Ratio: 0.06
Optimizer: AdamW (β1=0.9, β2=0.95, weight_decay=0.1)
Max Sequence Length: 512
Training Time: ~3 hours
Trained on diverse, high-quality datasets:
| Dataset | Samples | Purpose |
|---|---|---|
| FineWeb-Edu | 7,000 | Educational web content |
| UltraChat 200k | 7,000 | Conversational data |
| The Stack (Python) | 7,000 | Code understanding |
Total Training Samples: 21,000
Data Quality:
import torch
from transformers import AutoTokenizer, GPTNeoXForCausalLM
# Load tokenizer (from Pythia for better template support)
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/pythia-160m")
# Load base model
base_model = GPTNeoXForCausalLM.from_pretrained("Kiy-K/moe-gpt-neox-125m-v1.1")
# For full MoE functionality, you'll need the MoEGPTNeoX wrapper
# (See architecture code in repository)
# Basic generation example (with base model)
text = "def fibonacci(n):"
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
outputs = base_model.generate(
**inputs,
max_length=100,
temperature=0.7,
top_p=0.9,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
from safetensors.torch import load_file
# Load MoE weights
moe_weights = load_file("moe_layers.safetensors")
# Apply to your MoEGPTNeoX wrapper
# (Full implementation in training script)
Recommended settings:
# For creative tasks (code, stories)
generation_config = {
"temperature": 0.7,
"top_p": 0.9,
"top_k": 50,
"repetition_penalty": 1.1
}
# For factual tasks (Q&A, documentation)
generation_config = {
"temperature": 0.3,
"top_p": 0.85,
"top_k": 40,
"repetition_penalty": 1.0
}
This model excels at:
Best For:
class MoELayer:
hidden_size: 512
intermediate_size: 2048
num_experts: 8
top_k: 2
dropout: 0.1
# Gating Network
gate: Linear(512 -> 8) # Routes to experts
# Expert Networks
experts: [
ExpertNetwork(512 -> 2048 -> 512) # x8
]
# Load Balancing Loss
auxiliary_loss: 0.01 * load_balancing_loss
Initial Loss: ~3.5
Final Loss: ~2.1
Best Loss: ~2.0
Training Steps: 6,000
All 8 experts show balanced activation patterns thanks to load balancing loss, ensuring efficient parameter usage.
.
├── model.safetensors # Base GPT-NeoX weights (secure format)
├── moe_layers.safetensors # MoE expert weights (secure format)
├── moe_config.json # MoE architecture configuration
├── config.json # Model configuration
├── tokenizer.json # Tokenizer vocabulary
├── tokenizer_config.json # Tokenizer settings
├── special_tokens_map.json # Special token mappings
├── training_info.json # Training metrics & config
└── README.md # This file
All model weights use the safetensors format, providing:
Training data methodology ensures:
pip install torch>=2.0.0
pip install transformers>=4.35.0
pip install safetensors>=0.4.0
| Use Case | Minimum | Recommended |
|---|---|---|
| Inference | 2GB VRAM | 4GB VRAM |
| Fine-tuning | 8GB VRAM | 16GB VRAM |
| CPU Inference | 4GB RAM | 8GB RAM |
# Use float16 for faster inference
model = model.half().to("cuda")
# Enable torch.compile (PyTorch 2.0+)
model = torch.compile(model)
# Use gradient checkpointing for fine-tuning
model.gradient_checkpointing_enable()
v1.5 (Current)
v1.1 (Base)
If you use this model in your research or projects, please cite:
@misc{moe_gpt_neox_125m_v15,
author = {Kiy-K},
title = {MoE GPT-NeoX 125M v1.5: Efficient Mixture of Experts Language Model},
year = {2025},
publisher = {HuggingFace},
url = {https://huggingface.co/Kiy-K/moe-gpt-neox-125m-v1.5}
}
Found a bug or have a suggestion? Please open an issue on the repository!
Apache 2.0 License
This model inherits the Apache 2.0 license from the base GPT-NeoX model. You are free to:
See LICENSE for full terms.
Built with ❤️ for the open-source AI community
Everyone deserves their own AI model - not just expensive API access.