Qwen3-30B-A3B-MoBE16-nosm-lr02

mobe_qwen3_30b_a3b_nosoftmax_b16_lr02

num_B compression-stress sweep (B=16). Control arm: MoBE with the softmax/simplex mixing removed (free real coefficients, no sparse apparatus), at the free-coefficient LR optimum (lr=0.02). Isolates the simplex relaxation from the sparse machinery; should track Sparse-minimal at every num_B. See docs/findings_log.md.

• Method: mobe
• Base model: Qwen/Qwen3-30B-A3B
• Compressed: gate_proj, up_proj over layers 0-48 (down_proj kept dense)
• Settings: mobe, num_B=16, activation=tanh

See config.toml and training.json for the full run configuration and per-layer reconstruction stats.

Evaluation

No eval results attached.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-MoBE16-nosm-lr02",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-MoBE16-nosm-lr02", trust_remote_code=True)

The factored experts are reconstructed at runtime via the bundled Qwen3MoBEForCausalLM.