Downloads · 30 days
0
Llama-2-13b-chat-hf-onnx-int4
jojo1899/Llama-2-13b-chat-hf-onnx-int4
Llama-2-13b-chat-hf-onnx-int4 is a machine learning model from jojo1899. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama2.
This is an INT4 quantized version of the Llama-2-13b-chat-hf model. The Python packages used in creating this model are as follows:
Access
Public
Updated Nov 7, 2024
Repo size
7.6 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.data7.6 GB · 100%
At a glance
- License
- llama2
- Access
- Public
- Created
- Jun 5, 2024
- Updated
- Nov 7, 2024
- SHA
- 1195ae54
Model card
From the Hugging Face model README
This is an INT4 quantized version of the Llama-2-13b-chat-hf model. The Python packages used in creating this model are as follows:
onnx==1.16.1
onnxruntime-directml==1.20.0
onnxruntime-genai-directml==0.4.0
torch==2.5.1
transformers==4.45.2
This quantized model is created using the following command:
python -m onnxruntime_genai.models.builder -m meta-llama/Llama-2-13b-chat-hf -e dml -p int4 --extra_options {"int4_block_size"=128} -o ./Llama-2-13b-chat-hf-onnx-int4
onnxruntime_genai.models.builder quantizes the model using MatMul4BitsQuantizer from matmul_4bits_quantizer.py of onnxruntime/quantization/ with the "DEFAULT" method.
- License
- llama2
- Created
- Jun 5, 2024
- Updated
- Nov 7, 2024