Downloads · 30 days
0
amd/Llama-2-7b-chat-hf-awq-g128-int4-asym-fp16-onnx-dml
Llama-2-7b-chat-hf-awq-g128-int4-asym-fp16-onnx-dml is a text generation model from amd. Use it when you need the model to write or continue text. The card lists the license as llama2.
- Introduction - Quantization Tool: Quark 0.6.0 - OGA Model Builder: v0.5.1 - Quantization Strategy - AWQ / Group 128 / Asymmetric / UINT4 Weights / FP16 activations - Excluded Layers: None
Downloads · 30 days
0
Access
Public
Updated Dec 3, 2024
Repo size
3.7 GB
Likes
0
Public
Click a slice to open those files.
.data3.7 GB · 100%
From the Hugging Face model README
python3 quantize_quark.py \
--model_dir "$model" \
--output_dir "$output_dir" \
--quant_scheme w_uint4_per_group_asym \
--num_calib_data 128 \
--quant_algo awq \
--dataset pileval_for_awq_benchmark \
--seq_len 512 \
--model_export quark_safetensors \
--data_type float16 \
--exclude_layers [] \
--custom_mode awq
python builder.py \
-i <quantized safetensor model dir> \
-o <oga model output dir> \
-p int4 \
-e dml