Downloads · 30 days
9
3% of all-time downloads
brucethemoose/jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction
jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction is a text generation model from brucethemoose. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Downloads · 30 days
9
3% of all-time downloads
All-time downloads
359
Public
Repo size
18 GB
Likes
4
Public
Click a slice to open those files.
.safetensors18 GB · 100%
From the Hugging Face model README

Just a fiction oriented 4bpw exl2 quantization of https://huggingface.co/jondurbin/bagel-dpo-34b-v0.2
Quantized on 300K tokens of two Vicuna format chats, a sci fi story and a fiction story at a long context. This should yield better storywriting performance than the default exl2 quantization.
If anyone wants sizes other than 4bpw, for more/less context or smaller GPUs, just ask.
Being a Yi model, try running a lower temperature with ~0.05 MinP, a little repitition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run "hot" by default.
24GB GPUs can run Yi-34B-200K models at 45K-75K context with exllamav2, and performant UIs like exui. I go into more detail in this post
First pass:
python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -om /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -ml 32768 -mr 7 -ss 4096 -b 4.0 -hb 6 -nr
Second pass:
python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -m /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -l 12288 -r 25 -ml 32768 -mr 9 -ss 4096 -b 4.0 -hb 6 -cf /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction -nr