Downloads · 30 days
3
2% of all-time downloads
OriharaIzaya/calme-3.2-instruct-78b-exl2-mirror
calme-3.2-instruct-78b-exl2-mirror is a text generation model from OriharaIzaya. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
This repository hosts a mirror of the 4.5 bits per weight (bpw) quantization of the calme-3.2-instruct-78b model, leveraging the ExLlamaV2 format for efficient inference with high-context capabilities. This model is a…
Downloads · 30 days
3
2% of all-time downloads
All-time downloads
138
Public
Repo size
41.2 GB
Likes
0
Public
Click a slice to open those files.
.safetensors41.2 GB · 100%
From the Hugging Face model README
This repository hosts a mirror of the 4.5 bits per weight (bpw) quantization of the calme-3.2-instruct-78b model, leveraging the ExLlamaV2 format for efficient inference with high-context capabilities. This model is a Qwen 2.5 finetune. The original exllama quantization is found at calme-3.2-instruct-78b-exl2. This mirror is created for a stable connection and controlled version to run via a container. For all other purposes please use the original quantization.
The quantization process reduces memory usage and inference latency while maintaining high performance for generative text tasks.
This model uses the ChatML prompt template for interaction:
<|im_start|>system
{System}
<|im_end|>
<|im_start|>user
{User}
<|im_end|>
<|im_start|>assistant
{Assistant}
To use this quantized model, ensure you have the ExLlamaV2 library installed. Note that you need the exact exLlama version which was used for the quantization (ExLlamaV2 0.2.6).
pip install exllamav2
from exllamav2 import ExLlamaModel, ExLlamaTokenizer, ExLlamaPipeline
# Load model and tokenizer
model = ExLlamaModel.from_pretrained("OriharaIzaya/calme-3.2-instruct-78b-exl2-4.5bpw")
tokenizer = ExLlamaTokenizer.from_pretrained("OriharaIzaya/calme-3.2-instruct-78b-exl2-4.5bpw")
# Create pipeline
pipeline = ExLlamaPipeline(model, tokenizer)
# Generate text
messages = [{"role": "user", "content": "What is EXL2 quantization?"}]
response = pipeline(messages)
print(response)
For stable running of ExLlama models via OpenAI ord Kobold compatible APIs please use the Official ExLlama API TabbyAPI.
A containerized version of TabbyAPI intended for this model can be found on dockerhub.
To download the model files:
huggingface-cli install huggingface_hub
huggingface-cli login
huggingface-cli download OriharaIzaya/calme-3.2-instruct-78b-exl2-4.5bpw --include "*" --local-dir ./local-folder