Downloads · 30 days
46
0% of all-time downloads
llama-2-13b-chat-gptq
compressed-llm/llama-2-13b-chat-gptq
llama-2-13b-chat-gptq is a text generation model from compressed-llm. Use it when you need the model to write or continue text. It is set up for transformers.
All-time downloads
25.8K
Public
Repo size
32 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
Other1.5 KB · 53%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- llama
- Access
- Public
- Created
- Oct 3, 2023
- Updated
- Mar 24, 2024
- SHA
- 3dc38c8e
Try a prompt
Model card
From the Hugging Face model README
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM
model_path = 'efficient-llm/llama-2-13b-chat-gptq'
tokenizer_path = 'meta-llama/Llama-2-7b-hf'
model = AutoGPTQForCausalLM.from_quantized(
model_path,
# inject_fused_attention=False, # or
disable_exllama=True,
device_map='auto',
revision='3bit_128g',
)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, trust_remote_code=True)
input_ids = tokenizer('How are you?', return_tensors='pt').input_ids.to('cuda')
outputs = model.generate(input_ids=input_ids, max_length=128)
print(tokenizer.decode(outputs[0]))
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- Created
- Oct 3, 2023
- Updated
- Mar 24, 2024