Downloads · 30 days
11
3% of all-time downloads
m33393/llama-65b-gptq-cuda-4bit-32g-safetensors
llama-65b-gptq-cuda-4bit-32g-safetensors is a text generation model from m33393. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Converted to HF with transformers 4.30.0.dev0, then quantized to 4 bit with GPTQ (Group size 32):
Downloads · 30 days
11
3% of all-time downloads
All-time downloads
361
Public
Repo size
38.5 GB
Likes
2
Public
Click a slice to open those files.
.safetensors38.5 GB · 100%
From the Hugging Face model README
Converted to HF with transformers 4.30.0.dev0, then quantized to 4 bit with GPTQ (Group size 32):
python llama.py ../llama-65b-hf c4 --wbits 4 --true-sequential --act-order --groupsize 32 --save_safetensors 4bit-32g.safetensors
PPL should be marginally better than group size 128 at the cost of more VRAM. An A6000 should still be able to fit it all at full 2048 context.
Note that this model was quantized under GPTQ's cuda branch. Which means it should work with 0cc4m's KoboldAI fork:
https://github.com/0cc4m/KoboldAI