Downloads · 30 days
18
13% of all-time downloads
amd/Llama-3.1-8B-Instruct-w4a16-tao-symchannel-torchao-v0.17.0
Llama-3.1-8B-Instruct-w4a16-tao-symchannel-torchao-v0.17.0 is a text generation model from amd. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.1.
- Model Architecture: LlamaForCausalLM - Input: Text - Output: Text - Source Model: Llama-3.1-8B-Instruct - Supported Hardware: AMD EPYC (CPU inference) - Preferred Operating System: Linux - Inference Engine: vLLM v0.…
Downloads · 30 days
18
13% of all-time downloads
All-time downloads
134
Public
Parameters
8B
9.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors9.1 GB · 100%
How the weights are stored.
I87B · 87%
From the Hugging Face model README
This model was Built with Llama. This is a quantized version of Llama-3.1-8B-Instruct created by AMD using TorchAO for ZenDNN-optimized CPU inference.
The model was quantized from Llama-3.1-8B-Instruct using TorchAO v0.17.0.
Int4WeightOnlyConfig(granularity=PerAxis(0), mapping_type=MappingType.SYMMETRIC)lm_head and embed_tokensQuantization script:
python woq_sym_channel.py \
--model_name meta-llama/Llama-3.1-8B-Instruct \
--output_dir ./Llama-3.1-8B-Instruct-w4a16-tao-symchannel-torchao-v0.17.0
from vllm import LLM, SamplingParams
model = LLM(
model="amd/Llama-3.1-8B-Instruct-w4a16-tao-symchannel-torchao-v0.17.0",
dtype="bfloat16",
)
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)
torch==2.11.0
torchao==0.17.0
zentorch==2.11.0.1
vllm==0.20.2
For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):
# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)
# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)
Note: Set
LD_PRELOADbefore launching vLLM or any inference script.
The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.
| Benchmark | BF16 Baseline | W4A16-TAO-SymChannel (this model) | Recovery |
|---|---|---|---|
| MMLU (5-shot) | - | - | - |
| GSM8K_COT (8-shot) | - | - | - |
| Perplexity (wikitext2) | - | - | - |
Evaluation results will be updated after benchmarking.
lm_eval \
--model vllm \
--model_args pretrained="amd/Llama-3.1-8B-Instruct-w4a16-tao-symchannel-torchao-v0.17.0" \
--tasks mmlu \
--num_fewshot 5 \
--batch_size auto
This model is distributed under the same license as the source model. See the LICENSE file for details.
Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.