Downloads · 30 days
3
6% of all-time downloads
Qwen2.5-14B-Instruct-GPTQ-Marlin
GSJL/Qwen2.5-14B-Instruct-GPTQ-Marlin
Qwen2.5-14B-Instruct-GPTQ-Marlin is a machine learning model from GSJL. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
All-time downloads
47
Public
Parameters
3.3B
9.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors9.9 GB · 100%
Parameter types
How the weights are stored.
F161.7B · 50%
Model card
From the Hugging Face model README
pip install numpy gekko pandas
git clone https://github.com/PanQiWei/AutoGPTQ.git && cd AutoGPTQ
pip install -vvv --no-build-isolation -e .
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer, TextStreamer
model = AutoGPTQForCausalLM.from_quantized(
"GSJL/Qwen2.5-14B-Instruct-GPTQ-Marlin",
use_marlin=True
).to("cuda:0")
tokenizer = AutoTokenizer.from_pretrained(save_dir, use_fast = True)
streamer = TextStreamer(tokenizer, skip_prompt = True, skip_special_tokens=True)
prompt = [{"role":"user","content":"Hi mom!!!!!"}]
inputs = tokenizer.apply_chat_template(
prompt,
return_tensors="pt",
add_generation_prompt = True
).to("cuda:0")
output = model.generate(
input_ids = inputs,
streamer = streamer,
use_cache=True,
do_sample = True,
max_new_tokens = 600
)
- Type
- qwen2
- Created
- Sep 21, 2024
- Updated
- Sep 21, 2024