Downloads · 30 days
70
0% of all-time downloads
MoEMoEKKung/Frankenstein-MoE-en-10.7Bx4
Frankenstein-MoE-en-10.7Bx4 is a text generation model from MoEMoEKKung. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-nc-sa-4.0.
To initialize the gate projection weight of the MoE layer, the H6 trainset was sampled and used. We sampled 400 and selected the final 30 with low PPL.
Downloads · 30 days
70
0% of all-time downloads
All-time downloads
30.4K
Public
Parameters
36.1B
72.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors72.2 GB · 100%
From the Hugging Face model README
To initialize the gate projection weight of the MoE layer, the H6 trainset was sampled and used. We sampled 400 and selected the final 30 with low PPL.
trufulqa used gpt4 to generate data.
in progress