Hermes-4-405B-exl3-5bpw

Hermes 4 — Llama-3.1 405B EXL 3 5.00bpw

5.00 BPW H8 exllamav3 quant of Hermes 4 405B.

 -- A perplexity:  1.50484401
 -- B perplexity:  1.52952951
 -- A label in top-K:
      K = 1: 0.8938
      K = 2: 0.9486
      K = 3: 0.9640
      K = 4: 0.9714
      K = 5: 0.9757
 -- B label in top-K:
      K = 1: 0.8893
      K = 2: 0.9466
      K = 3: 0.9628
      K = 4: 0.9702
      K = 5: 0.9747
 -- Top-K agreement, A vs B:
      K = 1: 0.9722
      K = 2: 0.8527
      K = 3: 0.6881
      K = 4: 0.5185
      K = 5: 0.3663
 -- KL divergence (A, B):  0.01720790
 -- KL divergence (B, A):  0.01654647

command used to generate this quant

ulimit -n 100000
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True uv run python convert.py -i /path-to/Hermes-4-405B \
                  -o /path-to/final/hermes4-405b-5bpw \
                  -w /path-to/workdir \
                  -b 5.0 \
                  -hq \
                  -ss 2048 \
                  -cpi 3600 \
                  -hb 8 \
                  -pm \
                  -d 0,1,2,3,4,5,6,7 \
                  -dr 1,1,1,1,1,1,1,1

<img src="https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/roT9o5bMYBtQziRMlaSDf.jpeg" width="300" style="float:center" />