alpindale

Mistral-7B-Instruct-v0.2-AQLM-2Bit-1x16

alpindale/Mistral-7B-Instruct-v0.2-AQLM-2Bit-1x16

Mistral-7B-Instruct-v0.2-AQLM-2Bit-1x16 is a text generation model from alpindale. Use it when you need the model to write or continue text. It is set up for transformers.

Took 42 hours to quantize on 4xA40s, at a batch size of 128. I could've went higher, but hindsight. At that batch size, it was using about 25-30 GiB per GPU, utilization remained at 100%.