Downloads · 30 days
161
12% of all-time downloads
llama-30b-supercot-4bit-cuda
tsumeone/llama-30b-supercot-4bit-cuda
llama-30b-supercot-4bit-cuda is a text generation model from tsumeone. Use it when you need the model to write or continue text. It is set up for transformers.
Quantized version of this: https://huggingface.co/ausboss/llama-30b-supercot
All-time downloads
1.3K
Public
Repo size
16.9 GB
Likes
14
Public
Repo makeup
Click a slice to open those files.
.safetensors16.9 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- llama
- Access
- Public
- Created
- Apr 23, 2023
- Updated
- Apr 28, 2023
- SHA
- 594531bf
Try a prompt
Model card
From the Hugging Face model README
Quantized version of this: https://huggingface.co/ausboss/llama-30b-supercot
GPTQ quantization using https://github.com/0cc4m/GPTQ-for-LLaMa for compatibility with 0cc4m's fork of KoboldAI
This one is without groupsize to save on VRAM, so that you can enjoy the full 2048 max context if you have 24GB VRAM (or at least get a lot closer to it versus the groupsize version)
Command used to quantize:
python llama.py c:\llama-30b-supercot c4 --wbits 4 --act-order --true-sequential --save_safetensors 4bit.safetensors
Evaluation & Score (Lower is better):
- WikiText2: 4.66
- PTB: 17.64
- C4: 6.50
Groupsize version is here: https://huggingface.co/tsumeone/llama-30b-supercot-4bit-128g-cuda
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- Created
- Apr 23, 2023
- Updated
- Apr 28, 2023