Downloads · 30 days
8
1% of all-time downloads
hc-mistral-alpaca-merged-awq
parlance-labs/hc-mistral-alpaca-merged-awq
hc-mistral-alpaca-merged-awq is a text generation model from parlance-labs. Use it when you need the model to write or continue text. It is set up for transformers.
4bit AWQ Quantized Version of parlance-labs/hc-mistral-alpaca-merged
All-time downloads
588
Public
Parameters
7.2B
4.2 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors4.2 GB · 100%
Parameter types
How the weights are stored.
I327B · 96%
Try a prompt
Model card
From the Hugging Face model README
4bit AWQ Quantized Version of parlance-labs/hc-mistral-alpaca-merged
This is how to use AutoAWQ to quantize the model.
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
# setup
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
quant_path="hc-mistral-alpaca-merged-awq"
model_path="parlance-labs/hc-mistral-alpaca-merged"
model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# quantize and save model
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
After you save the model you can upload it to the hub
cd hc-mistral-alpaca-merged-awq
huggingface-cli upload parlance-labs/hc-mistral-alpaca-merged-awq .
- Task
- Text Generation
- Library
- transformers
- Type
- mistral
- Created
- Mar 9, 2024
- Updated
- Mar 9, 2024