Downloads · 30 days
51
4% of all-time downloads
Mistral-Small-Drummer-22B
nbeerbower/Mistral-Small-Drummer-22B
Mistral-Small-Drummer-22B is a text generation model from nbeerbower. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
All-time downloads
1.4K
Public
Parameters
22.2B
44.5 GB on disk
Likes
14
Public
Repo makeup
Click a slice to open those files.
.safetensors44.5 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- other
- Model type
- mistral
- Access
- Public
- Created
- Sep 26, 2024
- Updated
- Oct 12, 2024
- SHA
- 5195dfd5
Try a prompt
Base models
Model card
From the Hugging Face model README
Mistral-Small-Drummer-22B
mistralai/Mistral-Small-Instruct-2409 finetuned on jondurbin/gutenberg-dpo-v0.1 and nbeerbower/gutenberg2-dpo.
Method
ORPO tuned with 2xA40 on RunPod for 1 epoch.
learning_rate=4e-6,
lr_scheduler_type="linear",
beta=0.1,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
gradient_accumulation_steps=8,
optim="paged_adamw_8bit",
num_train_epochs=1,
Dataset was prepared using Mistral-Small Instruct format.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 29.45 |
| IFEval (0-Shot) | 63.31 |
| BBH (3-Shot) | 40.12 |
| MATH Lvl 5 (4-Shot) | 16.69 |
| GPQA (0-shot) | 12.42 |
| MuSR (0-shot) | 9.80 |
| MMLU-PRO (5-shot) | 34.39 |
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- License
- other
- Created
- Sep 26, 2024
- Updated
- Oct 12, 2024