Downloads · 30 days
241
7% of all-time downloads
chestnutlzj/MoE-Qwen-4x1.8B-pretrain-50000-ckpt
MoE-Qwen-4x1.8B-pretrain-50000-ckpt is a text generation model from chestnutlzj. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This is a MoE model project constructed based on the Qwen 1.8B model. In this project, we concatenated 4 original models and trained them using special training methods.
Downloads · 30 days
241
7% of all-time downloads
All-time downloads
3.2K
Public
Parameters
4.3B
8.5 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors8.5 GB · 100%
How the weights are stored.
BF164.3B · 100%
From the Hugging Face model README
This is a MoE model project constructed based on the Qwen 1.8B model. In this project, we concatenated 4 original models and trained them using special training methods.
This model is a checkpoint model for the continue pretraining stage.

| Groups | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| boolq | 0 | acc | 0.6508 | ± | 0.0083 |
| ceval-valid | 0 | acc | 0.5290 | ± | 0.1912 |
| 0 | acc_norm | 0.5290 | ± | 0.1912 | |
| cmmlu | 0 | acc | 0.5087 | ± | 0.1237 |
| 0 | acc_norm | 0.5087 | ± | 0.1237 | |
| mathqa | 0 | acc | 0.2647 | ± | 0.0081 |
| 0 | acc_norm | 0.2693 | ± | 0.0081 | |
| mmlu | 0 | acc | 0.4353 | ± | 0.0830 |
| - stem | 0 | acc | 0.3809 | ± | 0.0659 |
| - social_sciences | 0 | acc | 0.4959 | ± | 0.0708 |
| - other | 0 | acc | 0.4844 | ± | 0.0744 |
| - humanities | 0 | acc | 0.3998 | ± | 0.0849 |
This project is open source under the Tongyi Qianwen Research License Agreement. You can view the complete license agreement in this link: [https://github.com/QwenLM/Qwen/blob/main/Tongyi%20Qianwen%20RESEARCH%20LICENSE%20AGREEMENT].
During the use of this project, please ensure that your usage behavior complies with the terms and conditions of the license agreement.