Llama-3.1-8B-Quanto-Int8

Llama-3.1-8B-Instruct Quantized (Quanto INT8)

This is a quantized version of meta-llama/Llama-3.1-8B-Instruct using quanto with INT8 weight quantization.

Model Details

• Base Model: meta-llama/Llama-3.1-8B-Instruct
• Quantization Method: quanto
• Weight Precision: INT8 (qint8)
• Original Size: ~16 GB (bfloat16)
• Quantized Size: ~8.5 GB

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from quanto import quantize, freeze, qint8, safe_load

# Load base model structure
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True
)

# Quantize structure and load weights
quantize(model, weights=qint8)
state_dict = safe_load("model.safetensors")  # Use quanto's safe_load
model.load_state_dict(state_dict)
freeze(model)

# Load tokenizer and generate
tokenizer = AutoTokenizer.from_pretrained("tokenlabsdotrun/Llama-3.1-8B-Quanto-Int8")
inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=10)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

This model inherits the Llama 3.1 Community License.