SmartMoE-1B-7B-TQ3

SmartMoE-1B-7B-TQ3 🧠⚡

The smartest AI model that fits in 1GB VRAM + 6GB RAM.

Built on OLMoE-1B-7B-0125-Instruct — a Mixture of Experts model with 7B total parameters but only 1.3B active per token, quantized with TurboQuant-inspired techniques to fit extreme memory budgets.

🎯 Why This Model?

The Key Insight: MoE + Aggressive Quantization

Dense 1B models cap out around MMLU 25-35. But OLMoE packs 7B parameters into a MoE architecture where only 1.3B are active per token. Research shows MoE expert layers are more robust to low-bit quantization than dense layers (MoQE, arxiv:2310.02410), making this the ideal architecture for extreme compression.

Result: A model that fits in 1GB VRAM while accessing 7B parameters of learned knowledge.

📦 Available Quantizations

Memory Layout for 1GB VRAM + 6GB RAM

┌─────────────────────────────────┐
│ GPU VRAM (1 GB)                 │
│ ├─ Active expert weights (~400MB)│
│ └─ KV cache (~200MB)           │
├─────────────────────────────────┤
│ System RAM (6 GB)               │
│ ├─ Full model (~2.3-4.0 GB)    │
│ └─ OS + overhead (~2 GB)       │
└─────────────────────────────────┘

🚀 Quick Start

With llama.cpp

# Download the model
huggingface-cli download Abasgames/SmartMoE-1B-7B-TQ3 SmartMoE-1B-7B-IQ3_M.gguf --local-dir .

# Run with llama-cli (CPU only)
./llama-cli -m SmartMoE-1B-7B-IQ3_M.gguf -p "Explain quantum computing in simple terms:" -n 256

# Run with GPU offload (1GB VRAM)
./llama-cli -m SmartMoE-1B-7B-IQ3_M.gguf -ngl 5 -p "Explain quantum computing in simple terms:" -n 256

# Interactive chat mode
./llama-cli -m SmartMoE-1B-7B-IQ3_M.gguf -cnv

With Ollama

# Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./SmartMoE-1B-7B-IQ3_M.gguf
TEMPLATE """{{ if .System }}<|system|>
{{ .System }}
{{ end }}<|user|>
{{ .Prompt }}
<|assistant|>
"""
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
EOF

ollama create smartmoe -f Modelfile
ollama run smartmoe "What is the meaning of life?"

With LM Studio

1. Download any GGUF file from this repo
2. Open LM Studio → Load Model → Select the GGUF file
3. Start chatting!

📊 Benchmarks (Base Model)

From the OLMoE paper:

Key takeaway: OLMoE beats dense 1B models on most reasoning benchmarks while having similar active compute cost. The main weakness is math (GSM8K), which the planned SFT training with OpenR1-Math data is designed to address.

🔬 Technical Details

Architecture: OLMoE-1B-7B

Layers: 16
Hidden size: 2048
Attention heads: 16 (no GQA, full MHA)
Experts per layer: 64
Active experts per token: 8
Expert FFN size: 1024 (intermediate)
Vocabulary: 50,304
Context: 4,096 tokens
RoPE θ: 10,000

Why MoE is Perfect for Low Memory

1. Sparse Activation: Only 8/64 experts active = 12.5% of FFN weights used per token
2. Quantization Robustness: MoE expert layers tolerate low-bit quantization better than dense layers (MoQE, 2310.02410)
3. Natural Weight Offloading: Inactive expert weights can stay in RAM while active ones move to VRAM
4. 4.4B Expert Combinations: The routing network selects from C(64,8) = 4,426,165,368 possible expert combinations per token, enabling massive representational capacity

TurboQuant-Inspired Quantization

Based on TurboQuant (arxiv:2504.19874), accepted at ICLR 2026:

• Walsh-Hadamard Rotation: Weight matrices rotated before quantization for more uniform distribution
• Importance-Weighted Quantization: Critical weights (identified via importance matrix) get higher precision
• 3-bit Sweet Spot: TQ3 achieves MSE within 2.7× of Shannon lower bound

The GGUF quantization uses llama.cpp's k-quant system with importance matrix calibration for the I-quant variants (IQ2_M, IQ3_M).

🏋️ Training Scripts (For Custom Fine-tuning)

This repo includes training scripts for a 3-stage pipeline based on the Tulu-3 recipe:

Training Recipe Highlights

• Mix Distillation (arxiv:2502.12143): 50/50 long-CoT/short-CoT training improves sub-3B active parameter models
• QLoRA: r=64, alpha=128, target_modules="all-linear" (hits all 64 expert FFN layers)
• MoE Load Balancing: output_router_logits=True for balanced expert utilization

# Run SFT (needs GPU, ~24GB VRAM)
pip install transformers trl peft bitsandbytes datasets trackio accelerate
python train_sft.py

# Run DPO (after SFT)
python train_dpo.py

# Quantize (CPU only)
python quantize.py

📝 Research Papers

This model builds on insights from:

⚠️ Limitations

• Math Performance: Base model scores 28.4 on GSM8K — weakest area. The SFT training with OpenR1-Math data is designed to improve this.
• Context Length: 4,096 tokens (not suitable for long-document tasks)
• Knowledge Cutoff: Training data has a knowledge cutoff; may not know recent events
• Quantization Quality: 2-bit (IQ2_M) shows noticeable quality degradation; 3-bit (IQ3_M) recommended minimum
• English-centric: Primarily trained on English data

🙏 Credits

• AllenAI for OLMoE model and Tulu-3 training recipe
• Google Research for TurboQuant research
• llama.cpp for GGUF format and quantization tools
• bartowski for pioneering OLMoE GGUF quantization

---

Built with ❤️ for edge AI — because intelligence shouldn't require a data center.