Downloads · 30 days
5
2% of all-time downloads
Qwen3-30B-A3B-SparseMoBE16-minimal-lr02
AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-SparseMoBE16-minimal-lr02
Qwen3-30B-A3B-SparseMoBE16-minimal-lr02 is a text generation model from AverageMetaheuristicsEnjoyer. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
numB compression-stress sweep (B=16). Sparse-MoBE minimal (simplex relaxed, LISTA priors stripped) at its LR optimum (lr=0.02). Pairs with mobeb16 / nosoftmaxb16 to test whether relaxing the simplex helps more as the…
All-time downloads
293
Public
Parameters
20.9B
41.7 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors41.7 GB · 100%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Model type
- qwen3_moe
- Access
- Public
- Created
- Jul 15, 2026
- Updated
- Jul 15, 2026
- SHA
- 88fbebbe
Try a prompt
Base models
Model card
From the Hugging Face model README
sparse_qwen3_30b_b16_minimal_lr02
num_B compression-stress sweep (B=16). Sparse-MoBE minimal (simplex relaxed, LISTA priors stripped) at its LR optimum (lr=0.02). Pairs with mobe_b16 / nosoftmax_b16 to test whether relaxing the simplex helps more as the shared dictionary tightens. See docs/findings_log.md.
- Method:
sparse_mobe - Base model: Qwen/Qwen3-30B-A3B
- Compressed:
gate_proj, up_projover layers 0-48 (down_projkept dense) - Settings: sparse_mobe, num_B=16, activation=tanh
See config.toml and training.json for the full run configuration and per-layer reconstruction stats.
Evaluation
No eval results attached.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-SparseMoBE16-minimal-lr02",
trust_remote_code=True,
torch_dtype="bfloat16",
device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-SparseMoBE16-minimal-lr02", trust_remote_code=True)
The factored experts are reconstructed at runtime via the bundled Qwen3SparseMoBEForCausalLM.
- Task
- Text Generation
- Type
- qwen3_moe
- License
- apache-2.0
- Created
- Jul 15, 2026
- Updated
- Jul 15, 2026