Hermes-4-405B-exl3-3bpw

Hermes 4 — Llama-3.1 405B EXL 3 3.00bpw

3.00 BPW H8 exllamav3 quant of Hermes 4 405B.

It can run at about 90 t/s PP and 11 t/s TG at 5k ctx on 8 x 3090 Ti setup. Once thought to be impossible to run locally, not anymore. PPL and KLD jump is worse than expected, if I'll have too much time I'll try to prepare optimized quants.

 -- A perplexity:  1.50484602
 -- B perplexity:  1.93515072
 -- A label in top-K:
      K = 1: 0.8938
      K = 2: 0.9485
      K = 3: 0.9640
      K = 4: 0.9714
      K = 5: 0.9757
 -- B label in top-K:
      K = 1: 0.8252
      K = 2: 0.9103
      K = 3: 0.9384
      K = 4: 0.9523
      K = 5: 0.9604
 -- Top-K agreement, A vs B:
      K = 1: 0.8787
      K = 2: 0.5509
      K = 3: 0.2793
      K = 4: 0.1232
      K = 5: 0.0491
 -- KL divergence (A, B):  0.36757545
 -- KL divergence (B, A):  0.23848714

command used to generate this quant

ulimit -n 100000
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True uv run python convert.py -i /path-to/Hermes-4-405B \
                  -o /path-to/final/hermes4-405b-3bpw \
                  -w /path-to/workdir \
                  -b 3.0 \
                  -ss 2048 \
                  -cpi 3600 \
                  -hb 8 \
                  -pm \
                  -d 0,1,2,3,4,5,6,7 \
                  -dr 1,1,1,1,1,1,1,1

<img src="https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/roT9o5bMYBtQziRMlaSDf.jpeg" width="300" style="float:center" />