gemma-4-12b-qat-q4

Quanted using chat formatted imatrix + multimodal data. Shows promising results. Since 12b is vitless I thought it would be a good idea to try and activate the parts of the model that wouldn't usually activate during calibration.

Regular q4_0 from google is a bit off, as found by unsloth. I ran some KLD on my machine and confirmed this. I also compared unsloth's attempt with mine, and I managed to squeeze a bit more out of it.

Achieved by modifying imatrix source a bit: https://github.com/ganon3264/llama.cpp/tree/imatrixtemplate

/// UNSLOTH:
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       5.1266 ±    0.1416       0.00605 ±    0.00281       0.00997 ±    0.00123     3.524 ±  0.243 %    97.744 ±  0.147 %
   2       8.0217 ±    0.1716       0.01063 ±    0.00200       0.00938 ±    0.00092     3.448 ±  0.193 %    97.475 ±  0.110 %
 
====== Perplexity statistics ======
Mean PPL(Q)                   :   8.021674 ±   0.171569
Mean PPL(base)                :   7.936872 ±   0.167715
Cor(ln(PPL(Q)), ln(PPL(base))):  99.56%
Mean ln(PPL(Q)/PPL(base))     :   0.010628 ±   0.002001
Mean PPL(Q)/PPL(base)         :   1.010685 ±   0.002022
Mean PPL(Q)-PPL(base)         :   0.084802 ±   0.016294
 
====== KL divergence statistics ======
Mean    KLD:   0.009379 ±   0.000924
Maximum KLD:  10.733146
99.9%   KLD:   1.677376
99.0%   KLD:   0.063664
95.0%   KLD:   0.013149
90.0%   KLD:   0.007874
Median  KLD:   0.001235
10.0%   KLD:   0.000021
 5.0%   KLD:   0.000005
 1.0%   KLD:   0.000000
 0.1%   KLD:  -0.000002
Minimum KLD:  -0.000007
 
====== Token probability statistics ======
Mean    Δp: -0.031 ± 0.024 %
Maximum Δp: 88.001%
99.9%   Δp: 35.327%
99.0%   Δp:  6.107%
95.0%   Δp:  2.285%
90.0%   Δp:  1.183%
75.0%   Δp:  0.176%
Median  Δp: -0.000%
25.0%   Δp: -0.213%
10.0%   Δp: -1.264%
 5.0%   Δp: -2.342%
 1.0%   Δp: -6.733%
 0.1%   Δp: -34.392%
Minimum Δp: -98.842%
RMS Δp    :  3.448 ± 0.193 %
Same top p: 97.475 ± 0.110 %
 
/// THIS REPO:
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       5.1410 ±    0.1421       0.00886 ±    0.00287       0.00934 ±    0.00105     3.564 ±  0.241 %    97.744 ±  0.147 %
   2       8.0570 ±    0.1725       0.01502 ±    0.00195       0.00841 ±    0.00068     3.498 ±  0.187 %    97.602 ±  0.107 %
 
====== Perplexity statistics ======
Mean PPL(Q)                   :   8.057000 ±   0.172500
Mean PPL(base)                :   7.936872 ±   0.167715
Cor(ln(PPL(Q)), ln(PPL(base))):  99.59%
Mean ln(PPL(Q)/PPL(base))     :   0.015022 ±   0.001952
Mean PPL(Q)/PPL(base)         :   1.015135 ±   0.001982
Mean PPL(Q)-PPL(base)         :   0.120128 ±   0.016176
 
====== KL divergence statistics ======
Mean    KLD:   0.008406 ±   0.000676
Maximum KLD:   6.672385
99.9%   KLD:   1.298410
99.0%   KLD:   0.066219
95.0%   KLD:   0.014449
90.0%   KLD:   0.008712
Median  KLD:   0.001343
10.0%   KLD:   0.000024
 5.0%   KLD:   0.000006
 1.0%   KLD:   0.000000
 0.1%   KLD:  -0.000001
Minimum KLD:  -0.000016
 
====== Token probability statistics ======
Mean    Δp: -0.103 ± 0.024 %
Maximum Δp: 88.643%
99.9%   Δp: 31.379%
99.0%   Δp:  6.149%
95.0%   Δp:  2.217%
90.0%   Δp:  1.171%
75.0%   Δp:  0.156%
Median  Δp: -0.000%
25.0%   Δp: -0.263%
10.0%   Δp: -1.451%
 5.0%   Δp: -2.616%
 1.0%   Δp: -7.183%
 0.1%   Δp: -39.452%
Minimum Δp: -88.443%
RMS Δp    :  3.498 ± 0.187 %
Same top p: 97.602 ± 0.107 %