Mistral-7B-v0.1-da8w8-torchao-v0.17.0

Mistral-7B-v0.1-da8w8-torchao-v0.17.0

Model Overview

• Model Architecture: MistralForCausalLM
  • Input: Text
  • Output: Text
• Source Model: Mistral-7B-v0.1
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.23.0
• Quantization Framework: TorchAO v0.17.0
• Quantization Method: 8-bit Dynamic Activation, 8-bit Weight Quantization, Symmetric
• Compatible Stack:
  • ZenDNN v6.0.0
  • zentorch v2.11.0.2
  • PyTorch v2.11.0
  • TorchAO v0.17.0
  • vLLM v0.23.0

[!NOTE] zentorch v2.11.0.2 for PyTorch v2.11.0 has to be built from source.

Quantization

The model was produced using torchao as shown in the example below. Both activations and weights are quantized to INT8 with symmetric mapping. Activation scales are computed dynamically at runtime per token.

import torch
from transformers import TorchAoConfig, AutoModelForCausalLM, AutoTokenizer
from torchao.quantization import Int8DynamicActivationInt8WeightConfig
from torchao.quantization.quant_primitives import MappingType

MODEL_ID   = "mistralai/Mistral-7B-v0.1"
OUTPUT_DIR = "amd/Mistral-7B-v0.1-da8w8-torchao-v0.17.0"
modules_to_skip = ["lm_head"]

quantization_config = TorchAoConfig(
    Int8DynamicActivationInt8WeightConfig(
        version=2,
        act_mapping_type=MappingType.SYMMETRIC,
    ),
    modules_to_not_convert=modules_to_skip,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype=torch.bfloat16,
    device_map="cpu",
    quantization_config=quantization_config,
    trust_remote_code=True,
)
model.save_pretrained(OUTPUT_DIR)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.save_pretrained(OUTPUT_DIR)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=30, cache_implementation="static")
print(tokenizer.decode(out[0], skip_special_tokens=True))

Quick Start

Requirements

pip install --extra-index-url https://download.pytorch.org/whl/cpu \
            --extra-index-url https://wheels.vllm.ai/cpu/ \
    torch==2.11.0+cpu \
    vllm==0.23.0 \
    torchao==0.17.0 \
    "lm-eval[vllm]==0.4.12" \
    huggingface_hub

CPU runtime libraries (only needed if not already present):

conda install -c conda-forge gperftools=2.17.2 llvm-openmp=18.1.8 --no-deps -y

Recommended environment variables

# TorchInductor + zentorch
export TORCHINDUCTOR_FREEZING=1
export TORCHINDUCTOR_AUTOGRAD_CACHE=0
export VLLM_USE_AOT_COMPILE=0
export ZENDNNL_MATMUL_ALGO=1

# Required CPU runtime libraries
export LD_PRELOAD="<path to lib>/libtcmalloc_minimal.so.4:<path to lib>/libiomp5.so${LD_PRELOAD:+:$LD_PRELOAD}"

Locate the libraries with find / -name 'libtcmalloc_minimal.so.4' and find / -name 'libiomp5.so', then substitute the resulting directory for <path to lib>.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline using lm-evaluation-harness with the vLLM engine.

Evaluation Command

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Mistral-7B-v0.1-da8w8-torchao-v0.17.0,tokenizer=mistralai/Mistral-7B-v0.1,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

• Version Lock: This model is quantized with TorchAO v0.17.0 and is compatible only with PyTorch v2.11.0 / ZenDNN v6.0.0. It will not load correctly on other PyTorch versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.