Mixtral-Fusion-4x7B-Instruct-v0.1

Model Card for Mixtral-Fusion-4x7B-Instruct-v0.1

This model is an experimental model created by merging mistralai/Mixtral-8x7B-Instruct-v0.1 experts.

How we merged experts

Changed to merge using slerp.
Discussion

old merge version
We simply take the average of every two experts.weight.
The same goes for gate.weight.

How To Convert

use colab cpu-high-memory.
convert_mixtral_8x7b_to_4x7b.ipynb

OtherModels

mmnga/Mixtral-Extraction-4x7B-Instruct-v0.1

Usage

pip install git+https://github.com/huggingface/transformers --upgrade
pip install torch accelerate bitsandbytes flash_attn

from transformers import AutoTokenizer, AutoModelForCausalLM, MixtralForCausalLM
import torch

model_name_or_path = "mmnga/Mixtral-Fusion-4x7B-Instruct-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = MixtralForCausalLM.from_pretrained(model_name_or_path, load_in_8bit=True)

text = "[INST] What was John Holt's vision on education? [/INST] "
inputs = tokenizer(text, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))