Downloads · 30 days
9
3% of all-time downloads
brucethemoose/jondurbin_bagel-dpo-34b-v0.2-exl2-6bpw-fiction
jondurbin_bagel-dpo-34b-v0.2-exl2-6bpw-fiction is a text generation model from brucethemoose. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Downloads · 30 days
9
3% of all-time downloads
All-time downloads
273
Public
Repo size
26.4 GB
Likes
3
Public
Click a slice to open those files.
.safetensors26.4 GB · 100%
From the Hugging Face model README

Just a fiction oriented 6bpw exl2 quantization of https://huggingface.co/jondurbin/bagel-dpo-34b-v0.2
Quantized on 300K tokens of two Vicuna format chats, a sci fi story and a fiction story at a long context. This should yield better storywriting performance than the default exl2 quantization.
Being a Yi model, try running a lower temperature with ~0.05 MinP, a little repitition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run "hot" by default.
24GB GPUs can run Yi-34B-200K models at 45K-75K context with exllamav2, and performant UIs like exui. I go into more detail in this post
First pass:
python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -om /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -ml 32768 -mr 7 -ss 4096 -b 4.0 -hb 6 -nr
Second pass:
python convert.py --in_dir /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2 -o /home/alpha/FastModels/scratch -m /home/alpha/FastModels/bagelmeas.json --cal_dataset /home/alpha/Documents/stories.parquet -l 12288 -r 25 -ml 32768 -mr 9 -ss 4096 -b 4.0 -hb 6 -cf /home/alpha/FastModels/jondurbin_bagel-dpo-34b-v0.2-exl2-4bpw-fiction -nr