Llama-3.1-8B-Instruct-adpq-4bit-sim

Llama-3.1-8B-Instruct - ADPQ 4-bit Quantized

This work is part of a master thesis. The library used for quantization is available at auto-adpq.

pip install auto-adpq

Model Description

This is a compressed version of meta-llama/Llama-3.1-8B-Instruct created using 4-bit quantization.

This model was quantized to reduce VRAM usage and increase inference speed while maintaining majority of the original model's performance.

Quantization Details

• Original Model: meta-llama/Llama-3.1-8B-Instruct
• Quantization Method: ADPQ (Adaptive Quantization with data-free calibration)
• Precision: 4-bit
• Simulated: Yes

How to Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Tfloow/Llama-3.1-8B-Instruct-adpq-4bit-sim"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

inputs = tokenizer("Hello, world!", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Performance

<table> <thead> <tr> <th width="40%">Model Variant</th> <th width="30%">Quantization Method</th> <th width="30%">PPL (Perplexity)</th> </tr> </thead> <tbody> <tr> <td rowspan="3"><strong>meta-llama/Llama-3.1-8B</strong></td> <td>Baseline</td> <td>4.8693</td> </tr> <tr> <td>BNB</td> <td>5.0733</td> </tr> <tr> <td><strong>AdpQ</strong></td> <td><strong>5.3671</strong></td> </tr> <tr> <td rowspan="5"><strong>meta-llama/Llama-3.1-8B-Instruct</strong></td> <td>Baseline</td> <td>4.9080</td> </tr> <tr> <td>BNB</td> <td>4.9993</td> </tr> <tr> <td><strong>AdpQ</strong></td> <td><strong>5.0069</strong></td> </tr> <tr> <td>AWQ</td> <td>5.0440</td> </tr> <tr> <td>GPTQ</td> <td>nan</td> </tr> <tr> <td rowspan="4"><strong>meta-llama/Llama-3.2-1B</strong></td> <td>Baseline</td> <td>6.5546</td> </tr> <tr> <td><strong>AdpQ 9%</strong></td> <td><strong>6.9491</strong></td> </tr> <tr> <td>BNB</td> <td>6.9971</td> </tr> <tr> <td><strong>AdpQ 2%</strong></td> <td><strong>7.0380</strong></td> </tr> <tr> <td rowspan="3"><strong>meta-llama/Llama-3.2-3B-Instruct</strong></td> <td>Baseline</td> <td>5.7864</td> </tr> <tr> <td>AWQ</td> <td>5.8339</td> </tr> <tr> <td><strong>AdpQ</strong></td> <td><strong>5.9040</strong></td> </tr> </tbody> </table>

How was the model quantized?

import torch
from transformers import AutoModelForCausalLM

from auto_adpq import Auto_AdpQ, AutoAdpQConfig

model_name = "meta-llama/Llama-3.1-8B-Instruct"

# Setup Auto-AdpQ configuration
adpq_config = AutoAdpQConfig(
    group_size=group_size,
    n_iters=250,  # Throw UserWarning if too low
    alpha=0.05,   # The higher, the better the PPL loss but higher overhead
    device="cpu",
    q_bit=4,
    data_packing=False,
    symmetrical_quantization=True,
)

user = "Tfloow"
adpq_model_name = f"{user}/{model_name.split('/')[-1]}-adpq-4bit-sim"

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)

# virtual quantization
quantized = Auto_AdpQ.apply_quantization(model, adpq_config, multi_threaded=16)

model.push_to_hub(adpq_model_name)