speechless-mistral-7b-dare-0.85

speechlessai/speechless-mistral-7b-dare-0.85

speechless-mistral-7b-dare-0.85 is a text generation model from speechlessai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.

AWQ model(s) for GPU inference. GPTQ models for GPU inference, with multiple quantisation parameter options. 2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference

Downloads · 30 days

0% of all-time downloads

All-time downloads

35.1K

Public

Parameters

7.2B

15 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors15 GB · 100%

Parameter types

How the weights are stored.

BF167B · 96%

Try a prompt

Model card

From the Hugging Face model README

Experiment for DARE(Drop and REscale), most of the delta parameters can be directly set to zeros without affecting the capabilities of SFT LMs and larger models can tolerate a higher proportion of discarded parameters.

Merged with below DARE models.

weight_mask_rate: 0.85 / use_weight_rescale: True / mask_stratery: random / scaling_coefficient: 1.0

Model	Average	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K
Intel/neural-chat-7b-v3-1	61.59	66.21	83.64	62.37	59.65	78.14	19.56
migtissera/SynthIA-7B-v1.3	59.34	62.12	83.45	62.65	51.37	78.85	17.59
bhenrym14/mistral-7b-platypus-fp16	58.71	63.05	84.15	64.11	45.07	78.53	17.36
jondurbin/airoboros-m-7b-3.1.2	58.75	61.86	83.51	61.91	53.75	77.58	13.87
teknium/CollectiveCognition-v1.1-Mistral-7B	62.92	62.12	84.17	62.35	57.62	75.37	15.62
uukuguy/speechless-mistral-dolphin-orca-platypus-samantha-7b	62.06	64.33	84.4	63.72	52.52	78.37	21.38

speechless-mistral-7b-dare-0.85 (Merge 6 DARE models)	64.69	63.57	84.82	64.29	50.66	79.24	45.56

Task: Text Generation
Library: transformers
Type: mistral
License: llama2
Created: Nov 22, 2023
Updated: Dec 8, 2023