llama-3_1-8b-instruct-ov-int4

jojo1899/llama-3_1-8b-instruct-ov-int4

llama-3_1-8b-instruct-ov-int4 is a machine learning model from jojo1899. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.1.

This is an INT4 quantized version of the meta-llama/Llama-3.1-8B-Instruct model. The Python packages used in creating this model are as follows:

Downloads · 30 days

3% of all-time downloads

All-time downloads

346

Public

Repo size

13 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin4.1 GB · 100%

At a glance

License: llama3.1
Model type: llama
Access: Public
Created: Oct 30, 2024
Updated: Feb 26, 2025
SHA: 2e3653e2

Model card

From the Hugging Face model README

This is an INT4 quantized version of the meta-llama/Llama-3.1-8B-Instruct model. The Python packages used in creating this model are as follows:

openvino==2025.0.0
optimum==1.24.0
optimum-intel==1.22.0
nncf==2.15.0
torch==2.6.0
transformers==4.48.3

This quantized model is created using the following command:

optimum-cli export openvino --model "meta-llama/Llama-3.1-8B-Instruct" --weight-format int4 --group-size 128 --sym --ratio 1 --all-layers ./llama-3_1-8b-instruct-ov-int4

For more details, run the following command from your Python environment: optimum-cli export openvino --help

INFO:nncf:Statistics of the bitwidth distribution:

Num bits (N)	% all parameters (layers)	% ratio-defining parameters (layers)
4	100% (226 / 226)	100% (226 / 226)

Type: llama
License: llama3.1
Created: Oct 30, 2024
Updated: Feb 26, 2025