Downloads · 30 days
12
3% of all-time downloads
brucethemoose/Capybara-Tess12-Yi-34B-200K-DARE
Capybara-Tess12-Yi-34B-200K-DARE is a text generation model from brucethemoose. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Downloads · 30 days
12
3% of all-time downloads
All-time downloads
361
Public
Parameters
34.4B
68.8 GB on disk
Likes
6
Public
Click a slice to open those files.
.safetensors68.8 GB · 100%
From the Hugging Face model README
NousResearch/Nous-Capybara-34B and migtissera/Tess-M-v1.2 merged with an experimental implementation of "dare ties" via mergekit. See:
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
https://arxiv.org/abs/2311.03099
https://github.com/cg123/mergekit/tree/dare-tokenizer
Merged with the following config, and the tokenizer from Yi Llamafied:
models:
- model: /home/alpha/Storage/Models/Raw/larryvrh_Yi-34B-200K-Llamafied
# no parameters necessary for base model
- model: /home/alpha/Storage/Models/Raw/migtissera_Tess-M-v1.2
parameters:
weight: 0.62
density: 0.55
- model: /home/alpha/Storage/Models/Raw/Nous-Capybara-34B
parameters:
weight: 0.56
density: 0.55
merge_method: dare_ties
base_model: /home/alpha/Storage/Models/Raw/larryvrh_Yi-34B-200K-Llamafied
parameters:
int8_mask: true
dtype: bfloat16
SYSTEM: {system_message}
USER: {prompt}
ASSISTANT:
Being a Yi model, try disabling the BOS token and/or running a lower temperature with MinP if output doesn't seem right.
Sometimes the model "spells out" the stop token as </s> like Capybara, so you may need to add </s> as an additional stopping condition.
Credits:
https://github.com/cg123/mergekit/tree/dare-tokenizer
https://huggingface.co/NousResearch/Nous-Capybara-34B/
https://huggingface.co/migtissera/Tess-M-v1.2