Meta_Llama-3.1-8b-it_iMat_Custom_Quant_Stategies-GGUF

Experimental .GGUF quants for https://huggingface.co/google/gemma-2-9b-it accordingly to LCPP PR (based on b_3529 and now b_3565 for the newer ones) : https://github.com/ggerganov/llama.cpp/pull/8836

These experimental quant strategies revisiting Ikawrakow's work are displaying a slight decrease of perplexity, including per bpw (from 10%+ for the lowest quants to 0.x% for the highest ones). This is significant enough to encourage you folks to test them, and provide feedback if pertinent.

The iMatrix I use is based on Group Merged V3 and enriched with a bit of French, a bit of Serbian, and a bit of Croatian languages.

As usual, the name of the quants are a bit pompous, because they are numbered on the type of tensor quant mainly used as a base for the FFN.

ARC and PPL-512 DATA (Get the last data on the main post of the PR thread) :

IQ1_XS - Unusable on <30B models
PR
1.94 GB (1.93 BPW)
1.81 GiB (1.93 BPW)
PPL over 564 chunks for n_ctx=512 = 40.0024 +/- 0.27710

PR2
1.98 GB (1.97 BPW)
1.84 GiB (1.97 BPW)
PPL over 564 chunks for n_ctx=512 = 33.5198 +/- 0.24187


IQ1_S - Unusable on <30B models
Master
2.01 GB (2.00 BPW)
1.87 GiB (2.00 BPW)
PPL over 564 chunks for n_ctx=512 = 61.2817 +/- 0.41707

PR
2.05 GB (2.04 BPW)
1.91 GiB (2.04 BPW)
PPL over 564 chunks for n_ctx=512 = 25.2524 +/- 0.17651

PR2
2.06 GB (2.05 BPW)
1.91 GiB (2.05 BPW)
PPL over 564 chunks for n_ctx=512 = 24.2661 +/- 0.16923


IQ1_M
Master
2.15 GB (2.15 BPW)
2.01 GiB (2.15 BPW)
PPL over 564 chunks for n_ctx=512 = 26.3761 +/- 0.18200

PR
2.14 GB (2.13 BPW)
1.99 GiB (2.13 BPW)
PPL over 564 chunks for n_ctx=512 = 20.0588 +/- 0.14001

PR2
2.15 GB (2.14 BPW)
2.00 GiB (2.14 BPW)
PPL over 564 chunks for n_ctx=512 = 18.8721 +/- 0.13233

PR3
2.16 GB (2.15 BPW)
2.01 GiB (2.15 BPW)
PPL over 564 chunks for n_ctx=512 = 18.7469 +/- 0.13140


IQ1_XL
PR
2.21 GB (2.21 BPW)
2.06 GiB (2.21 BPW)
PPL over 564 chunks for n_ctx=512 = 18.5500 +/- 0.12753

PR2
2.23 GB (2.22 BPW)
2.08 GiB (2.22 BPW)
PPL over 564 chunks for n_ctx=512 = 17.4537 +/- 0.11995

PR3
2.25 GB (2.25 BPW)
2.10 GiB (2.25 BPW)
PPL over 564 chunks for n_ctx=512 = 17.3669 +/- 0.11928

PR4
2.28 GB (2.27 BPW)
2.12 GiB (2.27 BPW)
PPL over 564 chunks for n_ctx=512 = 15.5944 +/- 0.10615

PR5
2.28 GB (2.27 BPW)
2.12 GiB (2.27 BPW)
PPL over 564 chunks for n_ctx=512 = 15.4713 +/- 0.10524


IQ2_XXS
Master
2.39 GB (2.38 BPW)
2.23 GiB (2.38 BPW)
PPL over 564 chunks for n_ctx=512 = 15.2572 +/- 0.10267

PR
2.38 GB (2.37 BPW)
2.22 GiB (2.37 BPW)
PPL over 564 chunks for n_ctx=512 = 13.8073 +/- 0.09290

PR2
2.40 GB (2.39 BPW)
2.23 GiB (2.39 BPW)
PPL over 564 chunks for n_ctx=512 = 12.9671 +/- 0.08687

PR3
2.42 GB (2.41 BPW)
2.25 GiB (2.41 BPW)
PPL over 564 chunks for n_ctx=512 = 12.5074 +/- 0.08360

PR4
2.42 GB (2.41 BPW)
2.26 GiB (2.41 BPW)
PPL over 564 chunks for n_ctx=512 = 12.4244 +/- 0.08294


IQ2_XS
Master
2.60 GB (2.59 BPW)
2.42 GiB (2.59 BPW)
PPL over 564 chunks for n_ctx=512 = 11.7483 +/- 0.07776

PR
2.52 GB (2.51 BPW)
2.35 GiB (2.51 BPW)
PPL over 564 chunks for n_ctx=512 = 11.6639 +/- 0.07805

PR2
2.53 GB (2.52 BPW)
2.36 GiB (2.52 BPW)
PPL over 564 chunks for n_ctx=512 = 11.5685 +/- 0.07742

PR3
2.58 GB (2.57 BPW)
2.40 GiB (2.57 BPW)
PPL over 564 chunks for n_ctx=512 = 11.3031 +/- 0.07514

PR4
2.59 GB (2.58 BPW)
2.42 GiB (2.58 BPW)
PPL over 564 chunks for n_ctx=512 = 10.9291 +/- 0.07270

PR5
2.60 GB (2.59 BPW)
2.42 GiB (2.59 BPW)
PPL over 564 chunks for n_ctx=512 = 10.8794 +/- 0.07229


IQ2_S
Master
2.75 GB (2.74 BPW)
2.56 GiB (2.74 BPW)
PPL over 564 chunks for n_ctx=512 = 10.5180 +/- 0.06976

PR (fail)
2.71 GB (2.70 BPW)
2.52 GiB (2.70 BPW)
PPL over 564 chunks for n_ctx=512 = 10.7010 +/- 0.07027

PR2
2.75 GB (2.74 BPW)
2.56 GiB (2.74 BPW)
PPL over 564 chunks for n_ctx=512 = 10.3728 +/- 0.06806


IQ2_M
Master
2.94 GB (2.93 BPW)
2.74 GiB (2.93 BPW)
PPL over 564 chunks for n_ctx=512 = 9.5935 +/- 0.06228

PR
2.93 GB (2.92 BPW)
2.73 GiB (2.92 BPW)
PPL over 564 chunks for n_ctx=512 = 9.4125 +/- 0.06039


IQ2_XL
PR
2.99 GB (2.98 BPW)
2.78 GiB (2.98 BPW)
PPL over 564 chunks for n_ctx=512 = 9.3122 +/- 0.05973

PR2
3.11 GB (3.10 BPW)
2.90 GiB (3.10 BPW)
PPL over 564 chunks for n_ctx=512 = 9.0378 +/- 0.05764

PR3
3.14 GB (3.13 BPW)
2.93 GiB (3.13 BPW)
PPL over 564 chunks for n_ctx=512 = 8.8604 +/- 0.05620


IQ3_XXS

Master
Size : 3.04 GiB (3.25 BPW)
PPL 512 wikitext : 8.4985 +/- 0.05402

PR (good)
Size : 3.11 GiB (3.32 BPW)
PPL 512 wikitext : 8.3274 +/- 0.05334

PR2 (so so)
llm_load_print_meta: model size       = 3.08 GiB (3.29 BPW)
llm_load_print_meta: model size       = 3.30 GB (3.29 BPW)
Final estimate: PPL 512 = 8.3906 +/- 0.05329

Let's keep the first PR


IQ3_XS

Master
Size : 3.27 GiB (3.50 BPW)
PPL 512 wikitext : 8.2019 +/- 0.05167

PR (ok)
Size : 3.24 GiB (3.47 BPW)
PPL 512 wikitext : 8.1762 +/- 0.05176


IQ3_S

Master
Size : 3.42 GiB (3.66 BPW)
PPL 512 wikitext : 7.9894 +/- 0.05020

PR (good)
Size : 3.41 GiB (3.64 BPW)
PPL 512 wikitext : 7.9067 +/- 0.05022


IQ3_M

Master
Size : 3.52 GiB (3.76 BPW)  
PPL 512 wikitext : 7.9263 +/- 0.04943

PR (good)
Size : 3.49 GiB (3.73 BPW)
PPL 512 wikitext : 7.8704 +/- 0.04951


IQ3_XL

PR (good)
Size : 3.71 GiB (3.97 BPW)
PPL 512 wikitext : 7.7225 +/- 0.04946


IQ3_XXL

PR (good, the benefit seems meager but the token embeddings pushed form IQ3_S to IQ4_XS explains +0.05BPW of it,
and this tensor doesn't run in VRAM but in RAM)
Size : 3.83 GiB (4.09 BPW)
PPL 512 wikitext : 7.6720 +/- 0.04892


IQ3_XXXL

PR (good)
Size : 3.97 GiB (4.24 BPW)
PPL 512 wikitext : 7.5920 +/- 0.04839


IQ4_XS

Master
Size : 4.13 GiB (4.42 BPW)
Arc-C 299     49.16387960    
Arc-E 570     72.10526316     
PPL 512 wikitext : 7.5226 +/- 0.04820


IQ4_XSR

PR (good)
Size : 4.16 GiB (4.45 BPW)
Arc-C 299    
Arc-E 570      
PPL 512 wikitext : 7.5072 +/- 0.04814


FP16

MASTER : Gemma 2 9b It F16.
Size : 14.96 GiB (16.00 BPW)
Arc-C 299     49.49832776
Arc-E 570     73.85964912
PPL 512 wikitext : 7.3224 +/- 0.04674