macadeliccc_-_MBX-7B-v3-DPO-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

MBX-7B-v3-DPO - GGUF

• Model creator: https://huggingface.co/macadeliccc/
• Original model: https://huggingface.co/macadeliccc/MBX-7B-v3-DPO/

Original model description:

license: cc library_name: transformers datasets:

• jondurbin/truthy-dpo-v0.1 model-index:
• name: MBX-7B-v3-DPO results:
  • task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
    • type: acc_norm value: 73.55 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/MBX-7B-v3-DPO name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
    • type: acc_norm value: 89.11 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/MBX-7B-v3-DPO name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
    • type: acc value: 64.91 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/MBX-7B-v3-DPO name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
    • type: mc2 value: 74.0 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/MBX-7B-v3-DPO name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
    • type: acc value: 85.56 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/MBX-7B-v3-DPO name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
    • type: acc value: 69.67 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/MBX-7B-v3-DPO name: Open LLM Leaderboard

---

MBX-7B-v3-DPO

This model is a finetune of flemmingmiguel/MBX-7B-v3 using jondurbin/truthy-dpo-v0.1

Code Example

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("macadeliccc/MBX-7B-v3-DPO")
model = AutoModelForCausalLM.from_pretrained("macadeliccc/MBX-7B-v3-DPO")

messages = [
    {"role": "system", "content": "Respond to the users request like a pirate"},
    {"role": "user", "content": "Can you write me a quicksort algorithm?"}
]
gen_input = tokenizer.apply_chat_template(messages, return_tensors="pt")

Example Output

GGUF

Available here

Exllamav2

Quants are available from bartowski, check them out here

Download the size you want below, VRAM figures are estimates.

Evaluations

EQ-Bench Comparison

<pre>----Benchmark Complete---- 2024-01-30 15:22:18 Time taken: 145.9 mins Prompt Format: ChatML Model: macadeliccc/MBX-7B-v3-DPO Score (v2): 74.32 Parseable: 166.0 --------------- Batch completed Time taken: 145.9 mins --------------- </pre>

Original Model

<pre>----Benchmark Complete---- 2024-01-31 01:26:26 Time taken: 89.1 mins Prompt Format: Mistral Model: flemmingmiguel/MBX-7B-v3 Score (v2): 73.87 Parseable: 168.0 --------------- Batch completed Time taken: 89.1 mins --------------- </pre>

AGIEval

Average: 45.16%

GPT4All

Average: 77.73%

TruthfulQA

Average: 74.62%

Bigbench

Average: 48.83%

Average score: 61.58%

Elapsed time: 02:37:39

Open LLM Leaderboard Evaluation Results

Detailed results can be found here