Qwen3-30B-A3B-MoBE16-softmax-lr07

mobe_qwen3_30b_a3b_b16

num_B compression-stress sweep (B=16). MoBE-softmax arm at its LR optimum (lr=0.07). Shrinks the shared dictionary to 16 bases (tighter simplex / convex hull) to test whether the simplex constraint's cost grows with compression vs Sparse and no-softmax MoBE. See docs/findings_log.md.

• Method: mobe
• Base model: Qwen/Qwen3-30B-A3B
• Compressed: gate_proj, up_proj over layers 0-48 (down_proj kept dense)
• Settings: mobe, num_B=16, activation=tanh

See config.toml and training.json for the full run configuration and per-layer reconstruction stats.

Evaluation

No eval results attached.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-MoBE16-softmax-lr07",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-MoBE16-softmax-lr07", trust_remote_code=True)

The factored experts are reconstructed at runtime via the bundled Qwen3MoBEForCausalLM.