Downloads · 30 days
0
amd/gemma-2-2b-awq-uint4-asym-g128-lmhead-g32-fp16-onnx-hybrid_v2
gemma-2-2b-awq-uint4-asym-g128-lmhead-g32-fp16-onnx-hybrid_v2 is a text generation model from amd. Use it when you need the model to write or continue text. The card lists the license as gemma.
- Introduction This model was created by applying Quark with calibration samples from Pile dataset. - Quantization Strategy - Quantized Layers: All linear layers - Weight: uint4 asymmetric per-group. groupsize=32 for…
Downloads · 30 days
0
Access
Public
Updated Jun 23, 2025
Repo size
4.2 GB
Likes
0
Public
Click a slice to open those files.
.data2.6 GB · 61%
From the Hugging Face model README
export MODEL_DIR = [local model checkpoint folder] or google/gemma-2-2b
# single GPU
python quantize_quark.py --model_dir $MODEL_DIR \
--output_dir output_dir $MODEL_NAME-awq-uint4-asym-g128-lmhead-g32-fp16 \
--quant_scheme w_uint4_per_group_asym \
--num_calib_data 128 \
--quant_algo awq \
--dataset pileval_for_awq_benchmark \
--model_export hf_format \
--group_size 128 \
--group_size_per_layer lm_head 32 \
--data_type float16 \
--exclude_layers
# cpu
python quantize_quark.py --model_dir $MODEL_DIR \
--output_dir output_dir $MODEL_NAME-awq-uint4-asym-g128-lmhead-g32-fp16 \
--quant_scheme w_uint4_per_group_asym \
--num_calib_data 128 \
--quant_algo awq \
--dataset pileval_for_awq_benchmark \
--model_export hf_format \
--group_size 128 \
--group_size_per_layer lm_head 32 \
--data_type float16 \
--exclude_layers \
--device cpu
Quark has its own export format, quark_safetensors, which is compatible with autoAWQ exports.
Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize_quark.py. The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.
Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights reserved.