Llama-3.2-3B-Instruct-int8

Llama-3.2-3B-Instruct-INT8

INT8 quantized version of Llama-3.2-3B-Instruct, produced for Nockchain using the Pearl quantization methodology.

Quantization Details

• Method: W8A8 (weight-only INT8 + dynamic activation INT8)
• Weights: Symmetric per-channel quantization
• Activations: Symmetric per-token dynamic quantization
• lm_head and embeddings remain in FP16/BF16

This checkpoint follows the compressed-tensors W8A8 scheme and is compatible with vLLM and Nockchain serving pipeline.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("zkvesl/Llama-3.2-3B-Instruct-int8")
model = AutoModelForCausalLM.from_pretrained(
    "zkvesl/Llama-3.2-3B-Instruct-int8",
    torch_dtype="auto",
    device_map="auto"
)

inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))

Pearl Quantization Methodology

This model was quantized using the Pearl methodology, which ensures bit-exact integer matrix multiplication for inference while maintaining accuracy through GPTQ calibration and SmoothQuant optimization.

References

• Llama-3.2-3B-Instruct
• Pearl: Efficient Inference for LLMs