Downloads · 30 days
0
phi3_mini_amd_NPU
BoruiXu/phi3_mini_amd_NPU
phi3_mini_amd_NPU is a machine learning model from BoruiXu. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Access
Public
Updated Jul 4, 2024
Repo size
4.2 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt4.2 GB · 100%
At a glance
- Access
- Public
- Created
- Jul 4, 2024
- Updated
- Jul 4, 2024
- SHA
- fc3d3def
Model card
From the Hugging Face model README
run phi3-mini on AMD NPU
- If no
phi3_mini_awq_4bit_no_flash_attention.pt, use awq quantization to get the quantization model. - Put modeling_phi3.py in this repo into the phi-3-mini folder.
- Modify the file path in the run_awq.py
- run
python run_awq.py --task decode --target aie --w_bit 4
reference:https://github.com/amd/RyzenAI-SW/tree/main/example/transformers
As the quantization of phi-3, may refer to https://github.com/mit-han-lab/llm-awq/pull/183
PS: The performance is similar to that on CPU(7640hs).
- Created
- Jul 4, 2024
- Updated
- Jul 4, 2024