Downloads · 30 days
80
0% of all-time downloads
mixtralmerge-8x7B-rebalanced-test
chargoddard/mixtralmerge-8x7B-rebalanced-test
mixtralmerge-8x7B-rebalanced-test is a text generation model from chargoddard. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-nc-4.0.
This is a dumb experiment - don't expect it to be good!
All-time downloads
38.8K
Public
Parameters
46.7B
187 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin93.4 GB · 50%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- cc-by-nc-4.0
- Model type
- mixtral
- Access
- Public
- Created
- Jan 4, 2024
- Updated
- May 30, 2025
- SHA
- 997691ff
Try a prompt
Datasets
Model card
From the Hugging Face model README
This is a dumb experiment - don't expect it to be good!
I merged a few Mixtral models together then tuned only the routing parameters. There was a pretty steep drop in loss with only a bit of training - went from ~0.99 to ~.7 over about ten million tokens.
I'm hoping this after-the-fact balancing will have reduced some of the nasty behavior typical of current tunes. But maybe it just made it even dumber! We'll see.
Uses ChatML format.
Will update with more details if it turns out promising.
- Task
- Text Generation
- Library
- transformers
- Type
- mixtral
- License
- cc-by-nc-4.0
- Created
- Jan 4, 2024
- Updated
- May 30, 2025