Llama-2-13b-chat-hf-ov-int4

jojo1899/Llama-2-13b-chat-hf-ov-int4

Llama-2-13b-chat-hf-ov-int4 is a text generation model from jojo1899. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama2.

This is an INT4 quantized version of the meta-llama/Llama-2-13b-chat-hf model. The Python packages used in creating this model are as follows:

Downloads · 30 days

2% of all-time downloads

All-time downloads

1.3K

Public

Repo size

21.8 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.bin6.7 GB · 100%

At a glance

Task: Text Generation
Library: transformers
License: llama2
Model type: llama
Access: Public
Created: Jun 7, 2024
Updated: Nov 7, 2024
SHA: 0869f093

Try a prompt

Model card

From the Hugging Face model README

This is an INT4 quantized version of the meta-llama/Llama-2-13b-chat-hf model. The Python packages used in creating this model are as follows:

openvino==2024.5.0rc1
optimum==1.23.3
optimum-intel==1.20.1
nncf==2.13.0
torch==2.5.1
transformers==4.46.2

This quantized model is created using the following command:

optimum-cli export openvino --model "meta-llama/Llama-2-13b-chat-hf" --weight-format int4 --group-size 128 --sym --ratio 1 --all-layers ./Llama-2-13b-chat-hf-ov-int4

For more details, run the following command from your Python environment: optimum-cli export openvino --help

INFO:nncf:Statistics of the bitwidth distribution:

Num bits (N)	% all parameters (layers)	% ratio-defining parameters (layers)
4	100% (282 / 282)	100% (282 / 282)

Task: Text Generation
Library: transformers
Type: llama
License: llama2
Created: Jun 7, 2024
Updated: Nov 7, 2024