Downloads · 30 days
381
6% of all-time downloads
RichardErkhov/MaziyarPanahi_-_calme-2.2-llama3-70b-gguf
MaziyarPanahi_-_calme-2.2-llama3-70b-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
381
6% of all-time downloads
All-time downloads
6.8K
Public
Repo size
826 GB
Likes
0
Public
Click a slice to open those files.
.gguf874 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
calme-2.2-llama3-70b - GGUF
| Name | Quant method | Size |
|---|---|---|
| calme-2.2-llama3-70b.Q2_K.gguf | Q2_K | 24.56GB |
| calme-2.2-llama3-70b.IQ3_XS.gguf | IQ3_XS | 27.29GB |
| calme-2.2-llama3-70b.IQ3_S.gguf | IQ3_S | 28.79GB |
| calme-2.2-llama3-70b.Q3_K_S.gguf | Q3_K_S | 28.79GB |
| calme-2.2-llama3-70b.IQ3_M.gguf | IQ3_M | 29.74GB |
| calme-2.2-llama3-70b.Q3_K.gguf | Q3_K | 31.91GB |
| calme-2.2-llama3-70b.Q3_K_M.gguf | Q3_K_M | 31.91GB |
| calme-2.2-llama3-70b.Q3_K_L.gguf | Q3_K_L | 34.59GB |
| calme-2.2-llama3-70b.IQ4_XS.gguf | IQ4_XS | 35.64GB |
| calme-2.2-llama3-70b.Q4_0.gguf | Q4_0 | 37.22GB |
| calme-2.2-llama3-70b.IQ4_NL.gguf | IQ4_NL | 37.58GB |
| calme-2.2-llama3-70b.Q4_K_S.gguf | Q4_K_S | 37.58GB |
| calme-2.2-llama3-70b.Q4_K.gguf | Q4_K | 39.6GB |
| calme-2.2-llama3-70b.Q4_K_M.gguf | Q4_K_M | 39.6GB |
| calme-2.2-llama3-70b.Q4_1.gguf | Q4_1 | 41.27GB |
| calme-2.2-llama3-70b.Q5_0.gguf | Q5_0 | 45.32GB |
| calme-2.2-llama3-70b.Q5_K_S.gguf | Q5_K_S | 45.32GB |
| calme-2.2-llama3-70b.Q5_K.gguf | Q5_K | 46.52GB |
| calme-2.2-llama3-70b.Q5_K_M.gguf | Q5_K_M | 46.52GB |
| calme-2.2-llama3-70b.Q5_1.gguf | Q5_1 | 49.36GB |
| calme-2.2-llama3-70b.Q6_K.gguf | Q6_K | 53.91GB |
| calme-2.2-llama3-70b.Q8_0.gguf | Q8_0 | 69.83GB |
language:
This model is a fine-tune (DPO) of meta-llama/Meta-Llama-3-70B-Instruct model.
PS: This fine-tuned model was previously known as MaziyarPanahi/Llama-3-70B-Instruct-DPO-v0.2. It was renamed to avoid any confusion with the original model.
All GGUF models are available here: MaziyarPanahi/calme-2.2-llama3-70b-GGUF
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 37.98 |
| IFEval (0-Shot) | 82.08 |
| BBH (3-Shot) | 48.57 |
| MATH Lvl 5 (4-Shot) | 22.96 |
| GPQA (0-shot) | 12.19 |
| MuSR (0-shot) | 15.30 |
| MMLU-PRO (5-shot) | 46.74 |
| Metric | Value |
|---|---|
| Avg. | 78.96 |
| AI2 Reasoning Challenge (25-Shot) | 72.53 |
| HellaSwag (10-Shot) | 86.22 |
| MMLU (5-Shot) | 80.41 |
| TruthfulQA (0-shot) | 63.57 |
| Winogrande (5-shot) | 82.79 |
| GSM8k (5-shot) | 88.25 |
Top 10 models on the Leaderboard <img src="./llama-qwen2-leaderboard.png" alt="Llama-3-70B finet-tuned models" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
This model uses ChatML prompt template:
<|im_start|>system
{System}
<|im_end|>
<|im_start|>user
{User}
<|im_end|>
<|im_start|>assistant
{Assistant}
You can use this model by using MaziyarPanahi/calme-2.2-llama3-70b as the model name in Hugging Face's
transformers library.
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
from transformers import pipeline
import torch
model_id = "MaziyarPanahi/calme-2.2-llama3-70b"
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
# attn_implementation="flash_attention_2"
)
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True
)
streamer = TextStreamer(tokenizer)
pipeline = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
model_kwargs={"torch_dtype": torch.bfloat16},
streamer=streamer
)
# Then you can use the pipeline to generate text.
messages = [
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
{"role": "user", "content": "Who are you?"},
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
terminators = [
tokenizer.eos_token_id,
tokenizer.convert_tokens_to_ids("<|im_end|>"),
tokenizer.convert_tokens_to_ids("<|eot_id|>") # safer to have this too
]
outputs = pipeline(
prompt,
max_new_tokens=2048,
eos_token_id=terminators,
do_sample=True,
temperature=0.6,
top_p=0.95,
)
print(outputs[0]["generated_text"][len(prompt):])