Downloads · 30 days
20
17% of all-time downloads
gptj-4x6b-moe-test
TearGosling/gptj-4x6b-moe-test
gptj-4x6b-moe-test is a text generation model from TearGosling. Use it when you need the model to write or continue text. It is set up for transformers.
This is a test of converting the architecture of GPT-J 6B into a mixture-of-experts model. It is initialized with 4 experts (2 active) with otherwise the same configuration as GPT-J-6B, making this a 17B parameter mod…
All-time downloads
119
Public
Parameters
17.3B
34.7 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors34.7 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- gptj_moe
- Access
- Public
- Created
- Apr 6, 2024
- Updated
- Apr 6, 2024
- SHA
- ee26d26a
Try a prompt
Model card
From the Hugging Face model README
This is a test of converting the architecture of GPT-J 6B into a mixture-of-experts model. It is initialized with 4 experts (2 active) with otherwise the same configuration as GPT-J-6B, making this a 17B parameter model in total.
The model weights were initialized randomly - not loaded from the pretrained GPT-J - for testing purposes. This model is not useable for any downstream purposes unless you're trying to generate absolute schizo babble - in which case, this model is perfect for your use-case. You have been warned.
Be sure to pass trust_remote_code=True into AutoModelForCausalLM.from_pretrained if you still want to use this model for some god-forsaken reason.
- Task
- Text Generation
- Library
- transformers
- Type
- gptj_moe
- Created
- Apr 6, 2024
- Updated
- Apr 6, 2024