Llama-3.2-1B-Instruct-awq-uint4-asym-g128-bf16-lmhead

Meta-Llama-3.2-1B-Instruct-awq-uint4-asym-g128-bf16

• Introduction

  This model was created by applying Quark with calibration samples from Pile dataset.

• Quantization Stragegy

  • Quantized Layers: All linear layers
  • Weight: int4 symmetric per-group, group_size=128
  • Pre-trained Model data type: BFloat16
  • Scale data type: BFloat16
  • Zero point data type: Int32

• Quick Start

1. Download and install Quark
2. Run the quantization script in the example folder using the following command line:

   export MODEL_DIR = [local model checkpoint folder] or meta-llama/Meta-Llama-3.2-1B-Instruct-hf 
   # single GPU
   python3 quantize_quark.py --model_dir $MODEL_DIR \
                           --data_type bfloat16 \
                           --quant_scheme w_uint4_per_group_asym \
                           --num_calib_data 128 \
                           --quant_algo awq \
                           --dataset pileval_for_awq_benchmark \
                           --seq_len 512 \
                           --output_dir $MODEL_NAME-awq-uint4-asym-g128-bf16-lmhead \
                           --exclude_layers \
                           --model_export quark_safetensors \
                           --custom_mode awq \
   # cpu
   python3 quantize_quark.py --model_dir $MODEL_DIR \
                           --data_type bfloat16 \
                           --quant_scheme w_uint4_per_group_asym \
                           --num_calib_data 128 \
                           --quant_algo awq \
                           --dataset pileval_for_awq_benchmark \
                           --seq_len 512 \
                           --output_dir $MODEL_NAME-awq-uint4-asym-g128-bf16-lmhead \
                           --exclude_layers \
                           --model_export quark_safetensors \
                           --custom_mode awq \
                           --device cpu
   

Deployment

Quark has its own export format, quark_safetensors, which is compatible with autoAWQ exports.

Evaluation

Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize_quark.py. The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.

Evaluation scores

<table> <tr> <td><strong>Benchmark</strong> </td> <td><strong>Meta-Llama-3.2-1B-Instruct-hf(Bfloat16) </strong> </td> <td><strong>Meta-Llama-3.2-1B-Instruct-awq-uint4-asym-g128-bf16(this model)</strong> </td> </tr> <tr> <td>Perplexity-wikitext2 </td> <td>13.1586 </td> <td>14.8471 </td> </tr> </table> #### License Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.