mixtral-8x7b-quantized-gguf

This repository contains improved Mixtral-8x7B quantized models in GGUF format for use with llama.cpp. The models are fully compatible with the oficial llama.cpp release and can be used out-of-the-box.

The table shows a comparison between these models and the current llama.cpp quantization approach using Wikitext perplexities for a context length of 512 tokens. The "Quantization Error" columns in the table are defined as (PPL(quantized model) - PPL(int8))/PPL(int8). Running the full fp16 Mixtral8x7b model on the systems I have available takes too long, so I'm comparing against the 8-bit quantized model, where I get PPL = 4.1049. From past experience the 8-bit quantization should be basically equivalent to fp16.