Downloads · 30 days
79
0% of all-time downloads
MoEMoEKKung/Frankenstein-MoE-en-10.7Bx6
Frankenstein-MoE-en-10.7Bx6 is a text generation model from MoEMoEKKung. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-nc-sa-4.0.
To initialize the gate projection weight of the MoE layer, the H6 trainset was sampled and used. We sampled 400 and selected the final 30 with low PPL.
Downloads · 30 days
79
0% of all-time downloads
All-time downloads
30.2K
Public
Parameters
53B
106 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors106 GB · 100%
From the Hugging Face model README
To initialize the gate projection weight of the MoE layer, the H6 trainset was sampled and used. We sampled 400 and selected the final 30 with low PPL.
trufulqa used gpt4 to generate data.
in progress