Downloads · 30 days
7
15% of all-time downloads
AverageMetaheuristicsEnjoyer/qwen3-30b-a3b-sparse-mobe-b32-minimal-lr02-h100
qwen3-30b-a3b-sparse-mobe-b32-minimal-lr02-h100 is a text generation model from AverageMetaheuristicsEnjoyer. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Sparse-MoBE stripped of LISTA priors, at its own Block-J LR optimum (lr=0.02). Robustness check alongside the lr=0.07 (match-MoBE) run. See docs/sparsemobemechanism.md.
Downloads · 30 days
7
15% of all-time downloads
All-time downloads
46
Public
Parameters
23.3B
46.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors46.6 GB · 100%
From the Hugging Face model README
Sparse-MoBE stripped of LISTA priors, at its own Block-J LR optimum (lr=0.02). Robustness check alongside the lr=0.07 (match-MoBE) run. See docs/sparse_mobe_mechanism.md.
sparse_mobegate_proj, up_proj over layers 0-48 (down_proj kept dense)See config.toml and training.json for the full run configuration and per-layer reconstruction stats.
| Task | Metric | Value |
|---|---|---|
| arc_challenge | sample_len | 1172.0000 |
| arc_challenge | acc,none | 0.5324 |
| arc_challenge | acc_stderr,none | 0.0146 |
| arc_challenge | acc_norm,none | 0.5486 |
| arc_challenge | acc_norm_stderr,none | 0.0145 |
| arc_easy | sample_len | 2376.0000 |
| arc_easy | acc,none | 0.7967 |
| arc_easy | acc_stderr,none | 0.0083 |
| arc_easy | acc_norm,none | 0.7959 |
| arc_easy | acc_norm_stderr,none | 0.0083 |
| winogrande | sample_len | 1267.0000 |
| winogrande | acc,none | 0.7182 |
| winogrande | acc_stderr,none | 0.0126 |
| piqa | sample_len | 1838.0000 |
| piqa | acc,none | 0.7845 |
| piqa | acc_stderr,none | 0.0096 |
| piqa | acc_norm,none | 0.7992 |
| piqa | acc_norm_stderr,none | 0.0093 |
| hellaswag | sample_len | 10042.0000 |
| hellaswag | acc,none | 0.5601 |
| hellaswag | acc_stderr,none | 0.0050 |
| hellaswag | acc_norm,none | 0.7390 |
| hellaswag | acc_norm_stderr,none | 0.0044 |
| openbookqa | sample_len | 500.0000 |
| openbookqa | acc,none | 0.3400 |
| openbookqa | acc_stderr,none | 0.0212 |
| openbookqa | acc_norm,none | 0.4500 |
| openbookqa | acc_norm_stderr,none | 0.0223 |
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"AverageMetaheuristicsEnjoyer/qwen3-30b-a3b-sparse-mobe-b32-minimal-lr02-h100",
trust_remote_code=True,
torch_dtype="bfloat16",
device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/qwen3-30b-a3b-sparse-mobe-b32-minimal-lr02-h100", trust_remote_code=True)
The factored experts are reconstructed at runtime via the bundled Qwen3SparseMoBEForCausalLM.