gemma-4-31B-it-nf4

gemma-4-31B-it, NF4

google/gemma-4-31B-it quantised to 4-bit NF4 with bitsandbytes, double quantisation on, bfloat16 compute. Nothing else is changed. Use of this model is governed by the Gemma Terms of Use.

It exists so that a Space can load the host in about 17 GB instead of pulling 62 GB and quantising at startup.

Why this exact quantiser

We read the layer-47 hidden state out of this model and hand it to a separate 0.6B decoder and a linear retrieval head, both of which were fitted on bf16 states. So the question is whether quantisation moves the state out of the distribution they expect. We measured it on 100 COCO val2017 images.

Anisotropy on this backbone is 0.9874, so raw cosine is not usable: matched bf16/NF4 pairs read 0.9986 while mismatched pairs, different pictures across the two precisions, read 0.9867. Centered on the pool mean, matched pairs are 0.9631 (worst 0.9056) against a mismatched floor of 0.0237.

Retrieval survives: an NF4 query lands on its own image at rank 1 in 100 of 100 cases. Deeper ranking does move, so the two indices are not interchangeable. Verbalisation is content-stable but not token-stable: 8 of 8 sampled pairs name the same subject and 2 of 8 are identical strings, so any quoted example sentence should say which precision produced it.

AWQ and GGUF builds of this model exist and are probably fine, but they were not the ones we measured.