Downloads · 30 days
0
BoruiXu/phi3_mini_amd_NPU
phi3_mini_amd_NPU is a machine learning model from BoruiXu. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
0
Access
Public
Updated Jul 4, 2024
Repo size
4.2 GB
Likes
0
Public
Click a slice to open those files.
.pt4.2 GB · 100%
From the Hugging Face model README
run phi3-mini on AMD NPU
phi3_mini_awq_4bit_no_flash_attention.pt, use awq quantization to get the quantization model.python run_awq.py --task decode --target aie --w_bit 4reference:https://github.com/amd/RyzenAI-SW/tree/main/example/transformers
As the quantization of phi-3, may refer to https://github.com/mit-han-lab/llm-awq/pull/183
PS: The performance is similar to that on CPU(7640hs).