Downloads · 30 days
7
0% of all-time downloads
xmadai/Llama-3.1-Nemotron-70B-Instruct-xMADai-INT4
Llama-3.1-Nemotron-70B-Instruct-xMADai-INT4 is a text generation model from xmadai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.1.
This repository contains nvidia/Llama-3.1-Nemotron-70B-Instruct-HF quantized from 16-bit floats to 4-bit integers, using xMAD.ai proprietary technology.
Downloads · 30 days
7
0% of all-time downloads
All-time downloads
1.8K
Public
Repo size
39.8 GB
Likes
4
Public
Click a slice to open those files.
.safetensors39.8 GB · 100%
From the Hugging Face model README
This repository contains nvidia/Llama-3.1-Nemotron-70B-Instruct-HF quantized from 16-bit floats to 4-bit integers, using xMAD.ai proprietary technology.
Accuracy: This xMADified model is the best quantized version of the nvidia/Llama-3.1-Nemotron-70B-Instruct-HF model (40 GB only). See Table 1 below for model quality benchmarks.
Memory-efficiency: The full-precision model is around 140 GB, while this xMADified model is under 40 GB, making it feasible to run on a single 48 GB GPU.
Fine-tuning: These models are fine-tunable over the same reduced (a single 48 GB GPU) hardware in mere 3-clicks. Watch our product demo here
| Model | Arc Challenge | Arc Easy | LAMBADA Standard | LAMBADA OpenAI | MMLU STEM | MMLU Humanities | MMLU Social Sciences | MMLU Other |
|---|---|---|---|---|---|---|---|---|
| xmadai/Llama-3.1-Nemotron-70B-Instruct-xMADai-INT4 (this model) | 63.05 | 86.36 | 71.96 | 75.82 | 75.55 | 80.62 | 87.42 | 83.71 |
| unsloth/Llama-3.1-Nemotron-70B-Instruct-bnb-4bit | 60.32 | 85.35 | 71.01 | 74.64 | 75.1 | 80.06 | 87.33 | 83.71 |
Loading the model checkpoint of this xMADified model requires around 40 GB of VRAM. Hence it can be efficiently run on a single 48 GB GPU.
Package prerequisites:
pip install torch==2.4.0 # Run following if you have CUDA version 11.8: pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu118
pip install transformers accelerate optimum
pip install -vvv --no-build-isolation "git+https://github.com/PanQiWei/[email protected]"
Sample Inference Code
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM
model_id = "xmadai/Llama-3.1-Nemotron-70B-Instruct-xMADai-INT4"
prompt = [
{"role": "system", "content": "You are a helpful assistant, that responds as a pirate."},
{"role": "user", "content": "What's Deep Learning?"},
]
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
inputs = tokenizer.apply_chat_template(
prompt,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
).to("cuda")
model = AutoGPTQForCausalLM.from_quantized(
model_id,
device_map='auto',
trust_remote_code=True,
)
outputs = model.generate(**inputs, do_sample=True, max_new_tokens=1024)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
If you found this model useful, please cite our research paper.
@article{zhang2024leanquant,
title={LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid},
author={Zhang, Tianyi and Shrivastava, Anshumali},
journal={arXiv preprint arXiv:2407.10032},
year={2024},
url={https://arxiv.org/abs/2407.10032},
}
For additional xMADified models, access to fine-tuning, and general questions, please contact us at [email protected] and join our waiting list.