Downloads · 30 days
12
3% of all-time downloads
RichardErkhov/louisbrulenaudet_-_Pearl-7B-slerp-gguf
louisbrulenaudet_-_Pearl-7B-slerp-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
12
3% of all-time downloads
All-time downloads
480
Public
Repo size
82.2 GB
Likes
0
Public
Click a slice to open those files.
.gguf95.2 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
Pearl-7B-slerp - GGUF
| Name | Quant method | Size |
|---|---|---|
| Pearl-7B-slerp.Q2_K.gguf | Q2_K | 2.53GB |
| Pearl-7B-slerp.IQ3_XS.gguf | IQ3_XS | 2.81GB |
| Pearl-7B-slerp.IQ3_S.gguf | IQ3_S | 2.96GB |
| Pearl-7B-slerp.Q3_K_S.gguf | Q3_K_S | 2.95GB |
| Pearl-7B-slerp.IQ3_M.gguf | IQ3_M | 3.06GB |
| Pearl-7B-slerp.Q3_K.gguf | Q3_K | 3.28GB |
| Pearl-7B-slerp.Q3_K_M.gguf | Q3_K_M | 3.28GB |
| Pearl-7B-slerp.Q3_K_L.gguf | Q3_K_L | 3.56GB |
| Pearl-7B-slerp.IQ4_XS.gguf | IQ4_XS | 3.67GB |
| Pearl-7B-slerp.Q4_0.gguf | Q4_0 | 3.83GB |
| Pearl-7B-slerp.IQ4_NL.gguf | IQ4_NL | 3.87GB |
| Pearl-7B-slerp.Q4_K_S.gguf | Q4_K_S | 3.86GB |
| Pearl-7B-slerp.Q4_K.gguf | Q4_K | 4.07GB |
| Pearl-7B-slerp.Q4_K_M.gguf | Q4_K_M | 4.07GB |
| Pearl-7B-slerp.Q4_1.gguf | Q4_1 | 4.24GB |
| Pearl-7B-slerp.Q5_0.gguf | Q5_0 | 4.65GB |
| Pearl-7B-slerp.Q5_K_S.gguf | Q5_K_S | 4.65GB |
| Pearl-7B-slerp.Q5_K.gguf | Q5_K | 4.78GB |
| Pearl-7B-slerp.Q5_K_M.gguf | Q5_K_M | 4.78GB |
| Pearl-7B-slerp.Q5_1.gguf | Q5_1 | 5.07GB |
| Pearl-7B-slerp.Q6_K.gguf | Q6_K | 5.53GB |
| Pearl-7B-slerp.Q8_0.gguf | Q8_0 | 7.17GB |
tags:
03-22-2024 - To date, louisbrulenaudet/Pearl-34B-ties is the "Best 🤝 base merges and moerges model of around 30B" on the Open LLM Leaderboard.
Pearl-7B-slerp is a merge of the following models:
The evaluation was performed using the HuggingFace Open LLM Leaderboard.
| Model | Average | ARC | HellaSwag | MMLU | TruthfulQA | Winogrande | GSM8K | Params (B) |
|---|---|---|---|---|---|---|---|---|
| louisbrulenaudet/Pearl-7B-slerp | 72.75 | 68.00 | 87.16 | 64.04 | 62.35 | 81.29 | 73.62 | 7.24 |
| mistralai/Mixtral-8x7B-Instruct-v0.1 | 72.62 | 70.22 | 87.63 | 71.16 | 64.58 | 81.37 | 60.73 | 46.7 |
| microsoft/phi-2 | 61.33 | 61.09 | 75.11 | 58.11 | 44.47 | 74.35 | 54.81 | 2.78 |
| microsoft/Orca-2-13b | 58.64 | 60.67 | 79.81 | 60.37 | 56.41 | 76.64 | 17.97 | 13 |
| mistralai/Mistral-7B-Instruct-v0.1 | 54.96 | 54.52 | 75.63 | 55.38 | 56.28 | 73.72 | 14.25 | 7.24 |
| meta-llama/Llama-2-7b-hf | 50.97 | 53.07 | 78.59 | 46.87 | 38.76 | 74.03 | 14.48 | 6.74 |
Spherical Linear Interpolation (SLERP) serves as a technique for seamlessly interpolating between two vectors while maintaining a constant rate of change and upholding the geometric properties of the spherical space in which these vectors exist.
Opting for SLERP over traditional linear interpolation is motivated by various considerations. Linear interpolation in high-dimensional spaces may result in a reduction in the magnitude of the interpolated vector, diminishing the scale of weights. Additionally, in many cases, the alteration in the weights' direction conveys more meaningful information, such as feature learning and representation, compared to the magnitude of change.
$$ {\displaystyle \operatorname {slerp} (p_{0},p_{1};t)={\frac {\sin {[(1-t)\Omega }]}{\sin \Omega }}p_{0}+{\frac {\sin[t\Omega ]}{\sin \Omega }}p_{1}.}$$
The implementation of SLERP involves the following steps:
In essence, SLERP provides a robust mechanism for interpolating vectors, offering advantages in preserving directional information and mitigating issues associated with linear interpolation in high-dimensional spaces.
slices:
- sources:
- model: mlabonne/OmniBeagle-7B
layer_range: [0, 32]
- model: WizardLM/WizardMath-7B-V1.1
layer_range: [0, 32]
merge_method: slerp
base_model: mlabonne/OmniBeagle-7B
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5
dtype: bfloat16
!pip install -qU transformers accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "louisbrulenaudet/Pearl-7B-slerp"
messages = [{"role": "user", "content": "What is a large language model?"}]
tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto",
)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
If you use this code in your research, please use the following BibTeX entry.
@misc{louisbrulenaudet2023,
author = {Louis Brulé Naudet},
title = {Pearl-7B-slerp, an xtraordinary 7B model for maths},
year = {2023}
howpublished = {\url{https://huggingface.co/louisbrulenaudet/Pearl-7B-slerp}},
}
If you have any feedback, please reach out at [email protected].