Meta-Llama-Guard-2-8B-exl2

Exl2 quants for Meta-Llama-Guard-2-8B

Automatically quantized using the auto quant script from hf-scripts

Llama Guard 2 8B is the moderation tuned llama3-8B model by Meta.
Use this moderation model to protect your platform from unwanted generations.
BF16 weights are recommended for optimal accuracy in production environments.
This model has the following harm categories:

<table align="center"> <thead> <tr> <th colspan="2">Harm categories</th> </tr> </thead> <tbody> <tr> <td>S1: Violent Crimes</td> <td>S2: Non-Violent Crimes</td> </tr> <tr> <td>S3: Sex-Related Crimes</td> <td>S4: Child Sexual Exploitation</td> </tr> <tr> <td>S5: Specialized Advice</td> <td>S6: Privacy</td> </tr> <tr> <td>S7: Intellectual Property</td> <td>S8: Indiscriminate Weapons </td> </tr> <tr> <td>S9: Hate</td> <td>S10: Suicide &amp; Self-Harm</td> </tr> <tr> <td>S11: Sexual Content</td> <td></td> </tr> </tbody> </table>

BPW:

3.0
4.0
5.0
6.0
6.5
8.0
measurement.json

How to download:

oobabooga's downloader

use something like download-model.py to download with python requests.
Install requirements:

pip install requests tqdm

Example for downloading 8bpw:

python download-model.py Anthonyg5005/Meta-Llama-Guard-2-8B-exl2:8.0bpw

huggingface-cli

You may also use huggingface-cli
To install it, install python hf-hub

pip install huggingface-hub

Example for 8bpw:

huggingface-cli download Anthonyg5005/Meta-Llama-Guard-2-8B-exl2 --local-dir Llama-Guard-2-8B-exl2-8bpw --revision 8.0bpw

Git LFS (not recommended)

I would recommend the http downloaders over using git, they can resume downloads if failed and are much easier to work with.
Make sure to have git and git LFS installed.
Example for 8bpw download with git:

Have LFS file skip disabled

# windows
set GIT_LFS_SKIP_SMUDGE=0
# linux
export GIT_LFS_SKIP_SMUDGE=0

Clone repo branch

git clone https://huggingface.co/Anthonyg5005/Meta-Llama-Guard-2-8B-exl2 -b 8.0bpw