Downloads · 30 days
29
7% of all-time downloads
alpindale/magnum-v4-123b-hqq-4bit
magnum-v4-123b-hqq-4bit is a machine learning model from alpindale. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
This repo contains magnum-v4-123b quantized to 4-bit precision using HQQ.
Downloads · 30 days
29
7% of all-time downloads
All-time downloads
404
Public
Parameters
63.6B
66.3 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors66.3 GB · 100%
How the weights are stored.
U860.9B · 96%
From the Hugging Face model README
This repo contains magnum-v4-123b quantized to 4-bit precision using HQQ.
HQQ provides a similar level of precision to AWQ at 4-bit, but with no need for calibration.
This quant was generated using 8xA40s within only 10 minutes.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig
model_path = "anthracite-org/magnum-v4-123b"
quant_config = HqqConfig(nbits=4, group_size=128, axis=1)
model = AutoModelForCausalLM.from_pretrained(model_path,
torch_dtype=torch.float16,
cache_dir='.',
device_map="cuda:0",
quantization_config=quant_config,
low_cpu_mem_usage=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
output_path = "magnum-v4-123b-hqq-4bit"
model.save_pretrained(output_path)
tokenizer.save_pretrained(output_path)
You can perform inference directly with transformers, or using aphrodite:
pip install aphrodite-engine
aphrodite run alpindale/magnum-v4-123b-hqq-4bit -tp 2