Downloads · 30 days
10
4% of all-time downloads
LLongMA-2-13b-16k-GPTQ-4bit-32g
EyeDeck/LLongMA-2-13b-16k-GPTQ-4bit-32g
LLongMA-2-13b-16k-GPTQ-4bit-32g is a text generation model from EyeDeck. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.
GPTQ quantized version of LLongMA 2 13B 16K as found here: https://huggingface.co/conceptofmind/LLongMA-2-13b-16k
All-time downloads
267
Public
Repo size
8 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors8 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- llama2
- Model type
- llama
- Access
- Public
- Created
- Aug 2, 2023
- Updated
- Aug 2, 2023
- SHA
- 6e67c9c9
Try a prompt
Model card
From the Hugging Face model README
GPTQ quantized version of LLongMA 2 13B 16K as found here: https://huggingface.co/conceptofmind/LLongMA-2-13b-16k
As of writing, that model is missing a model card, so see here to get a general idea: https://huggingface.co/conceptofmind/LLongMA-2-7b
Quantized using GPTQ-for-LLaMA (59cabb9) using the following command:
CUDA_VISIBLE_DEVICES=0 python llama.py <model-path> c4 --wbits 4 --true-sequential --act-order --groupsize 32 --save_safetensors <out-path>/4bit-32g-tsao.safetensors
You'll probably want to run this with ExLlama; use --compress_pos_emb 4.0 and up to --length/--max_seq_len 16384
Using ExLlama, full context fits into 24GB of VRAM with only 1-2GB to spare.
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- llama2
- Created
- Aug 2, 2023
- Updated
- Aug 2, 2023