Sonya-7B-EXL2

Sakura-SOLAR-Instruct

• Model creator: Sanji Watsuki
• Original model: Sonya-7B
• Is a merge of:
  • xDAN-AI/xDAN-L1-Chat-RL-v1
  • jan-hq/stealth-v1.2
  • chargoddard/piano-medley-7b
  • NeverSleep/Noromaid-7b-v0.2
  • athirdpath/NSFW_DPO_vmgb-7b

Quantizations

I also provide zipped quantization because a lot of people find gguf single download convenient. Zipped quantization is relatively smaller in size to download. After extracted, you can use the model folder as usual.

<details> <summary>Parameter Explanation</summary>

• BPW: Bits per weight of the quantized model.
• Folder Size: Size of the model folder in GB when stored on disk or after extracted from zip file.
• Zipped Folder Size: Size of the zipped file after downloaded.
• VRAM Usage: VRAM usage of the model measured using ExLlamav2_HF and 4096 max_seq_len with Oobabooga's Text Generation WebUI.
• Loss: Accuracy loss of the quantized model compared to the original model.
• Description: Recommended GPU and/or settings for the model.

</details>

On the table, I included VRAM usages of both 4096 max_seq_len and 8192 max_seq_len as a comparison. Sonya-7B default max_seq_len is 8192.

Unfortunately I couldn't make higher quants than 7.0bpw. No error on the console, just "Killed" after creating output directory for the model.

Calibration Dataset

• Exllamav2 default dataset

Prompt template: Alpaca

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

Original Info

<div style="display: flex; justify-content: center; align-items: center"> <img src="https://huggingface.co/SanjiWatsuki/Sonya-7B/resolve/main/assets/Sonya.jpg"> </div > <p align="center"> <big><b>Top 1 Performer MT-bench 🤪</b></big> </p>

WTF is This?

Sonya-7B is, at the time of writing, the #1 performing model in MT-Bench first turn, ahead of GPT-4, and overall the #2 model in MT-Bench, to the best of my knowledge. Sonya-7B should be a good all-purpose model for all tasks including assistant, RP, etc.

Sonya-7B has a similar structure to my previous model, Silicon-Maid-7B, and uses a very similar merge. It's a merge of xDAN-AI/xDAN-L1-Chat-RL-v1, Jan-Ai's Stealth v1.2, chargoddard/piano-medley-7b, NeverSleep/Noromaid-7B-v0.2, and athirdpath/NSFW_DPO_vmgb-7b. Sauce is below. Somehow, by combining these pieces, it substantially outscores any of its parents on MT-Bench.

I picked these models because:

• MT-Bench normally correlates well with real world model quality and xDAN performs well on it.
• Almost all models in the mix were Alpaca prompt formatted which gives prompt consistency.
• Stealth v1.2 has been a magic sprinkle that seems to increase my MT-Bench scores.
• I added RP models because it boosted the Writing and Roleplay benchmarks 👀

Based on the parent models, I expect this model to be used with an 8192 context window. Please use NTK scaling alpha of 2.6 to experimentally try out 16384 context.

Let me be candid: Despite the test scores, this model is NOT is a GPT killer. I think it's a very sharp model for a 7B, it probably punches way above its weight for a 7B, but it's still a 7B model. Even for a 7B model, I think it's quirky and has some weird outputs, probably due to how Frankenstein this merge is. Keep your expectations in check 😉

MT-Bench Average Turn

<img src="https://huggingface.co/SanjiWatsuki/Sonya-7B/resolve/main/assets/mt-bench-gpt.png"> <img src="https://huggingface.co/SanjiWatsuki/Sonya-7B/resolve/main/assets/mt-bench-comparison.png">

The Sauce

models:
  - model: xDAN-AI/xDAN-L1-Chat-RL-v1
    parameters:
      weight: 1
      density: 1
  - model: chargoddard/piano-medley-7b
    parameters:
      weight: 0.3
  - model: jan-hq/stealth-v1.2
    parameters:
      weight: 0.2
  - model: NeverSleep/Noromaid-7b-v0.2
    parameters:
      weight: 0.2
  - model: athirdpath/NSFW_DPO_vmgb-7b
    parameters:
      weight: 0.2
merge_method: ties
base_model: mistralai/Mistral-7B-v0.1
parameters:
  density: 0.4
  int8_mask: true
  normalize: true
dtype: bfloat16

There was no additional training, finetuning, or DPO. This is a straight merger.

Prompt Template (Alpaca)

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

I found that this model performed worse with the xDAN prompt format so, despite the heavy weight of xDAN in this merger, I recommeend against its use.

Other Benchmark Stuff

########## First turn ##########

########## Second turn ##########

If you'd like to replicate the MT-Bench run, please ensure that the Alpaca prompt template is applied to the model. I did this by putting "alpaca" in the model path to trigger the AlpacaAdapter.