Downloads · 30 days
14
6% of all-time downloads
deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100
AverageMetaheuristicsEnjoyer/deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100
deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100 is a text generation model from AverageMetaheuristicsEnjoyer. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Sparse-MoBE stripped of LISTA priors (T=0, S=0, square encoder, random w-init) on DeepSeek-V2-Lite-Chat, at its own Block-J LR optimum (lr=0.02). Compared against MoBE mobedeepseekv2lite (lr=0.07). Layer 0 is dense (f…
All-time downloads
221
Public
Parameters
13.3B
26.6 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors26.6 GB · 100%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Model type
- deepseek_v2
- Access
- Public
- Created
- May 31, 2026
- Updated
- May 31, 2026
- SHA
- 23af2023
Try a prompt
Base models
Model card
From the Hugging Face model README
sparse_deepseek_v2_lite_b4_bs16_minimal_lr02
Sparse-MoBE stripped of LISTA priors (T=0, S=0, square encoder, random w-init) on DeepSeek-V2-Lite-Chat, at its own Block-J LR optimum (lr=0.02). Compared against MoBE mobe_deepseek_v2_lite (lr=0.07). Layer 0 is dense (first_k_dense_replace=1) so start_layer=1. See docs/sparse_mobe_mechanism.md.
- Method:
sparse_mobe - Base model: deepseek-ai/DeepSeek-V2-Lite-Chat
- Compressed:
gate_proj, up_projover layers 1-27 (down_projkept dense) - Settings: sparse_mobe, num_B=4, activation=tanh
See config.toml and training.json for the full run configuration and per-layer reconstruction stats.
Evaluation
| Task | Metric | Value |
|---|---|---|
| arc_challenge | sample_len | 1172.0000 |
| arc_challenge | acc,none | 0.4787 |
| arc_challenge | acc_stderr,none | 0.0146 |
| arc_challenge | acc_norm,none | 0.4787 |
| arc_challenge | acc_norm_stderr,none | 0.0146 |
| arc_easy | sample_len | 2376.0000 |
| arc_easy | acc,none | 0.7673 |
| arc_easy | acc_stderr,none | 0.0087 |
| arc_easy | acc_norm,none | 0.7437 |
| arc_easy | acc_norm_stderr,none | 0.0090 |
| winogrande | sample_len | 1267.0000 |
| winogrande | acc,none | 0.6969 |
| winogrande | acc_stderr,none | 0.0129 |
| piqa | sample_len | 1838.0000 |
| piqa | acc,none | 0.7916 |
| piqa | acc_stderr,none | 0.0095 |
| piqa | acc_norm,none | 0.8041 |
| piqa | acc_norm_stderr,none | 0.0093 |
| hellaswag | sample_len | 10042.0000 |
| hellaswag | acc,none | 0.5798 |
| hellaswag | acc_stderr,none | 0.0049 |
| hellaswag | acc_norm,none | 0.7565 |
| hellaswag | acc_norm_stderr,none | 0.0043 |
| openbookqa | sample_len | 500.0000 |
| openbookqa | acc,none | 0.3460 |
| openbookqa | acc_stderr,none | 0.0213 |
| openbookqa | acc_norm,none | 0.4680 |
| openbookqa | acc_norm_stderr,none | 0.0223 |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"AverageMetaheuristicsEnjoyer/deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100",
trust_remote_code=True,
torch_dtype="bfloat16",
device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100", trust_remote_code=True)
The factored experts are reconstructed at runtime via the bundled DeepseekV2SparseMoBEForCausalLM.
- Task
- Text Generation
- Type
- deepseek_v2
- License
- apache-2.0
- Created
- May 31, 2026
- Updated
- May 31, 2026