Downloads · 30 days
10
17% of all-time downloads
AverageMetaheuristicsEnjoyer/qwen3-30b-a3b-td-moe-ep-whiten-out-fix
qwen3-30b-a3b-td-moe-ep-whiten-out-fix is a text generation model from AverageMetaheuristicsEnjoyer. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
TD-MoE Tucker decomposition on Qwen3-30B-A3B, compression matched to MoBE/Sparse-MoBE b32 (37.5% removed per gate/up matrix, 62.5% kept). Expert-preserving, OUTPUT whitening (gradient covariance).
Downloads · 30 days
10
17% of all-time downloads
All-time downloads
59
Public
Parameters
23.3B
93.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors46.6 GB · 100%
From the Hugging Face model README
TD-MoE Tucker decomposition on Qwen3-30B-A3B, compression matched to MoBE/Sparse-MoBE b32 (37.5% removed per gate/up matrix, 62.5% kept). Expert-preserving, OUTPUT whitening (gradient covariance).
td_moegate_proj, up_proj over layers 0-48 (down_proj kept dense)See config.toml and training.json for the full run configuration and per-layer reconstruction stats.
| Task | Metric | Value |
|---|---|---|
| arc_challenge | sample_len | 1172.0000 |
| arc_challenge | acc,none | 0.2201 |
| arc_challenge | acc_stderr,none | 0.0121 |
| arc_challenge | acc_norm,none | 0.2645 |
| arc_challenge | acc_norm_stderr,none | 0.0129 |
| arc_easy | sample_len | 2376.0000 |
| arc_easy | acc,none | 0.3489 |
| arc_easy | acc_stderr,none | 0.0098 |
| arc_easy | acc_norm,none | 0.3346 |
| arc_easy | acc_norm_stderr,none | 0.0097 |
| winogrande | sample_len | 1267.0000 |
| winogrande | acc,none | 0.5122 |
| winogrande | acc_stderr,none | 0.0140 |
| piqa | sample_len | 1838.0000 |
| piqa | acc,none | 0.5571 |
| piqa | acc_stderr,none | 0.0116 |
| piqa | acc_norm,none | 0.5484 |
| piqa | acc_norm_stderr,none | 0.0116 |
| hellaswag | sample_len | 10042.0000 |
| hellaswag | acc,none | 0.2956 |
| hellaswag | acc_stderr,none | 0.0046 |
| hellaswag | acc_norm,none | 0.3278 |
| hellaswag | acc_norm_stderr,none | 0.0047 |
| openbookqa | sample_len | 500.0000 |
| openbookqa | acc,none | 0.1580 |
| openbookqa | acc_stderr,none | 0.0163 |
| openbookqa | acc_norm,none | 0.2900 |
| openbookqa | acc_norm_stderr,none | 0.0203 |
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"AverageMetaheuristicsEnjoyer/qwen3-30b-a3b-td-moe-ep-whiten-out-fix",
trust_remote_code=True,
torch_dtype="bfloat16",
device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/qwen3-30b-a3b-td-moe-ep-whiten-out-fix", trust_remote_code=True)
The factored experts are reconstructed at runtime via the bundled Qwen3TDMoEForCausalLM.