Downloads · 30 days
30
1% of all-time downloads
abideen/AlphaMonarch-daser
AlphaMonarch-daser is a text generation model from abideen. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-nc-4.0.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
30
1% of all-time downloads
All-time downloads
5.4K
Public
Parameters
7.2B
14.5 GB on disk
Likes
11
Public
Click a slice to open those files.
.safetensors14.5 GB · 100%
From the Hugging Face model README

AlphaMonarch-daser is a mixture of two techniques that are LaserQlora and Dora. This model is a DPO fine-tuned of mlabonne/NeuralMonarch-7B using the argilla/OpenHermes2.5-dpo-binarized-alpha preference dataset. I have fine-tuned this model only on half of the projections, but have achieved better results as compared to the version released AlphaMonarch-dora. I have trained this model for 1080 steps. Comparison of AlphaMonarch, AlphaMonarch-laser, AlphaMonarch-daser, and AlphaMonarch-dora on the OpenLLM leaderboard are:
On YALL leaderboard: AlphaMonarch-daser > AlphaMonarch-dora > AlphaMonarch > AlphaMonarch-laser

On OpenLLM bench: AlphaMonarch-laser > AlphaMonarch > AlphaMonarch-daser > AlphaMonarch-dora

The following hyperparameters were used during training: