GOBA-OLMoE-Expert-Tuned

GOBA-OLMoE-Expert-Tuned: Domain-Specialized MoE via Expert Tuning

3 domain-specialized variants | JA / Finance / Code | No general performance loss | GGUF Q4_K_M | Apache 2.0

Domain-adapted variants of OLMoE-1B-7B-0125-Instruct created using Expert Tuning — a novel approach that repurposes low-importance expert slots in Mixture-of-Experts models for domain-specific adaptation, as an alternative to LoRA or full fine-tuning.

Highlights

• 3 domain variants: Japanese, Finance, and Code — each tuned independently
• Zero general performance loss: MMLU and GSM8K scores remain within ±2pp of the original
• Domain improvements confirmed: JMMLU +4.5pp, HumanEval+ +10pp, EDINET-Bench Fraud Detection +16pp
• Drop-in replacement: works with llama.cpp with no code changes
• Apache 2.0: fully open for commercial use

Included Models

Benchmark Results

General Benchmarks (no domain bias)

Domain-Specific Benchmarks

Note: OLMoE-1B-7B has 1.3B active parameters, so absolute scores are lower than larger models. The relative improvements from Expert Tuning are the key result.

Model Details

What is Expert Tuning?

Expert Tuning is a novel domain adaptation technique for Mixture-of-Experts (MoE) models. Instead of adding external adapters (LoRA) or fine-tuning all parameters, it identifies low-importance experts within the existing MoE architecture and replaces them with domain-specialized experts trained via knowledge distillation.

Key advantages over LoRA:

• No additional parameters at inference time (experts replace existing slots)
• Native MoE routing automatically directs domain-relevant tokens to specialized experts
• Compatible with quantized GGUF inference — no adapter merging needed

Method overview:

1. Compute importance scores for all experts across layers
2. Select bottom-k experts as candidates for replacement
3. Train domain-specific experts using cross-expert knowledge distillation with domain data
4. Insert trained experts back into the GGUF with lossless Q4_K/Q6_K quantization

Training Data

How to Use

With llama.cpp

# Run with llama-server (e.g., Japanese-tuned variant)
llama-server \
  -m OLMoE-ja-tuned.gguf \
  --port 8090 \
  -ngl 99 \
  -c 4096

# Query via OpenAI-compatible API
curl http://localhost:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "olmoe",
    "messages": [{"role": "user", "content": "日本の金融政策について説明してください"}],
    "max_tokens": 512
  }'

With moe-stream

moe-stream OLMoE-ja-tuned.gguf --server --preload-gates --preload-attn

Technical Notes

• Q4_K/Q6_K quantizer: Custom Python implementation matching llama.cpp reference, with correct sub-block min clamping, qs packing order, and symmetric signed encoding
• Insertion CosSim: 0.9974 (quantized vs. original trained weights), confirming lossless insertion
• Training CosSim: 0.831 average (teacher-student similarity after KD), indicating meaningful domain adaptation while preserving expert structure
• 64 trained experts per model: 16 layers x 4 experts/layer replaced

Limitations

• OLMoE-1B-7B has only 1.3B active parameters, limiting absolute performance on complex tasks
• Domain benchmarks use moderate sample sizes (20-200 questions); larger evaluations may show different effect sizes
• Finance-tuned model shows prediction bias toward fraud detection, with regression on earnings forecasting tasks
• Expert Tuning effectiveness scales with the number of experts per layer; models with fewer experts (e.g., 8-16) have less capacity for domain injection

Citation

@misc{goba2026expert,
  title={Expert Tuning: Domain Adaptation via Expert Slot Repurposing in Mixture-of-Experts Models},
  author={GOBA AI Labs},
  year={2026},
  url={https://huggingface.co/goba-ai-labs/GOBA-OLMoE-Expert-Tuned}
}

Related Models

• PrunedHub-GPT-OSS-20B-28x — Lossless expert pruning for GPT-OSS-20B
• PrunedHub-Qwen3-30B-A3B-EN-MxMoE — Mixed-quantization MoE pruning
• PrunedHub-Qwen3-30B-A3B-JA-MxMoE — Language-aware MoE pruning

---

Built by GOBA AI Labs — Making large MoE models practical on consumer hardware.