Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

ISTA-DASLab/Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf

Mistral-7B-v0.1-AQLM-PV-2Bit-1x16-hf is a text generation model from ISTA-DASLab. Use it when you need the model to write or continue text. It is set up for transformers.

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

Downloads · 30 days

1% of all-time downloads

All-time downloads

2.1K

Public

Parameters

1.3B

2.5 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors2.5 GB · 100%

Parameter types

How the weights are stored.

I16872M · 70%

Try a prompt

Model card

From the Hugging Face model README

Official quantization of Mistral-7B-v0.1 using PV-Tuning on top of AQLM.

For this quantization, we used 1 codebook of 16 bits for groups of 8 weights.

Results (0-shot acc):

Model	Quantization	WinoGrande	PiQA	HellaSwag	ArcE	ArcC	Model size, Gb
Mistral-7B-v0.1	None	0.7364	0.8047	0.6115	0.7887	0.4923	14.5
	1x16 (this)	0.7151	0.7976	0.5880	0.7698	0.4514	2.51

The 1x16g16 (1-bit) models are on the way, as soon as we update the inference lib with their respective kernels.

To learn more about the inference, as well as the information on how to quantize models yourself, please refer to the official GitHub repo. The original code for PV-Tuning can be found in the AQLM@pv-tuning branch.

Task: Text Generation
Library: transformers
Type: mistral
Languages: en
Created: May 29, 2024
Updated: May 30, 2024