Downloads · 30 days
11
4% of all-time downloads
KnutJaegersberg/Galactica-120B-GPTQ-2-bit-64g
Galactica-120B-GPTQ-2-bit-64g is a text generation model from KnutJaegersberg. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-4.0.
Downloads · 30 days
11
4% of all-time downloads
All-time downloads
313
Public
Repo size
36.6 GB
Likes
3
Public
Click a slice to open those files.
.safetensors36.6 GB · 100%
From the Hugging Face model README
Experimental quantization.
Working inference code (regular inference with autogptq does not work without return_token_type_ids=False, didn't get it to work with textgen-webui):
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer, TextGenerationPipeline
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(quantized_model_dir, device="cuda:0", use_triton=False)
input_ids = tokenizer("Question: What is the purpose of life?\n\nAnswer:", return_tensors="pt").input_ids.to("cuda:0")
out = model.generate(input_ids=input_ids,max_length=300)
print(tokenizer.decode(out[0]))
or
print(tokenizer.decode(model.generate(**tokenizer("test is", return_tensors="pt", return_token_type_ids=False).to("cuda:0"))[0]))