IQ2_NL-IQ3_NL-Test-Models

Experimental IQ2_NL & IQ3_NL Tests

Please note that these models cannot be used with standard llama.cpp. See quantize: add IQ2_NL and IQ3_NL types (CPU)

Quantization test results for NVIDIA-Nemotron-Nano-9B-v2, comparing standard llama.cpp quantization against a fork that adds two new non-linear block types: IQ2_NL and IQ3_NL.

The test model uses the nemotron_h hybrid Mamba-2 / attention architecture, whose tensor dimensions are divisible by 32 but not by 256. Every K-quant (Q2_K … Q6_K) and every I-quant (IQ1_M, IQ2_XXS, …) in llama.cpp operates on 256-element super-blocks. When a tensor's row length is not a multiple of 256, the quantizer cannot use the requested type and falls back to a 32-element block type.

NVIDIA-Nemotron-Nano-9B-v2 triggers this on almost every tensor in the model:

The consequence, taken directly from the quantization logs:

warning: blk.31.ffn_up.weight - ncols 4480 not divisible by 256 (required for type q2_K) -> falling back to q4_0
llama_model_quantize_impl: WARNING: 90 of 341 tensor(s) required fallback quantization

90 of 341 tensors, or rather 90 of the 122 quantizable tensors (73.8%) bypass the requested type entirely. And because the lowest-bpw 32-block type available in standard llama.cpp is Q4_0/IQ4_NL at 4.50 bpw, that fallback establishes a hard floor. A Q2_K quantization of this model does not produce a 2-bit model; it lands at ~4.50 bpw, and a IQ1_M lands at ~4.36 bpw. The sub-4 bpw region is unreachable.

The fork adds IQ2_NL and IQ3_NL non-linear quantization types built on the same layout as the existing IQ4_NL, extending the family downward.

Because these are 32-block types, nemotron_h's 4480 and 15680 geometries are no longer a barrier. Fallback count collapses from 90 → 1:

The single remaining fallback is output.weight, requested as Q5_K but quantized to Q5_1 (6.00 bpw, 32-block).

1. Benefits

a) 32-block divisibility restores the low-bit type options

The 256-element super-block is a high-quality throughput optimisation and is an excellent option for architectures with power-of-two-friendly geometries, but a hard wall for everything else. nemotron_h and many recent architectures are the second case: neither d_model = 4480 nor d_ffn = 15680 is a whole number of super-blocks (17.5 and 61.25 respectively), so they fall back to other, usually sub-optimal, types.

IQ2_NL and IQ3_NL sidestep this entirely by inheriting IQ4_NL's 32-element block:

• Fallbacks drop from 90 / 341 to 1 / 341.
• The requested type is honoured on 120 & 121 of 122 quantizable tensors, where the naive Q2_K and Q3_K recipes honour only 27.
• The delivered bit-rate finally tracks the requested bit-rate: IQ2_NL asks for 2.50 bpw and delivers 2.80 bpw overall (the overhead being FP32 norms, Q5_1 output, and IQ3_NL embeddings), where Q2_K asks for 2.6258 bpw and delivers 4.50.

The new types generalize well beyond Nemotron. Any architecture with non-power-of-two hidden dimensions like hybrid SSM/Mamba models, pruned or distilled checkpoints, models with unusual GQA head counts, and most width-searched Neural Architecture Search (NAS) derivatives hit the same wall, and the fix is the same.

b) bypassing the 4.5 bpw floor imposed by IQ4_NL

The complete set of 32-block quantization types in llama.cpp is:

The minimum is 4.50 bpw. For a model where the 256-block types are unavailable, that value is an absolute floor on the entire model. Every "aggressive" quant type collapses onto it: IQ1_M → 4.36 bpw, Q2_K → 4.50 bpw, Q3_K → 4.84 bpw. Three nominally distinct types spanning 1.75–3.44 bpw all land within 0.5 bpw of each other, at more than double the requested budget.

IQ2_NL (2.50) and IQ3_NL (3.50) extend the non-linear family below that floor:

IQ3_NL wins on size and on every quality metric, but not on speed. It is smaller than IQ1_M on disk and simultaneously better on perplexity, KL divergence, token-probability RMS error, and top-1 agreement, but generates tokens ~35% slower (tg128 35.53 vs 54.70 t/s, pp1024+tg1024 66.38 vs 98.43 t/s). For a 9B model this moves the practical deployment target from "needs 5 GiB" to "fits in 4 GiB" and with better quality, provided that throughput cost is acceptable.

IQ2_NL is a valuable capability extension. At 2.80 bpw it produces a model 17.6% the size of F16, an option that previously did not exist at any quality, but the fidelity cost is substantial: PPL +50.28%, mean KLD 0.432, and top-1 agreement of only 71.1%. Roughly three tokens in ten change rank against the F16 reference. This is characteristic of ~2.5 bpw quantization generally rather than a defect of the type, and it is the expected shape of the curve; but it means IQ2_NL should be understood as a last-resort type for hard VRAM ceilings, not a default. IQ3_NL is where the fork's practical value sits.

Standard IQ1_M is outperformed once a coherent 2.5 bpw fallback exists.

Key Takeaways

• IQ3_NL outperforms the entire sub-5-bpw range. At 3.81 GiB / 3.67 bpw it is 15.8% smaller than the smallest model standard llama.cpp can build (IQ1_M, 4.52 GiB / 4.36 bpw) while scoring materially better on every metric: PPL 8.3177 vs 8.6505, mean KLD 0.0784 vs 0.1292 (−39.4%), top-1 agreement 86.05% vs 83.88%.
• IQ2_NL removes the 4.5 bpw floor. At 2.91 GiB / 2.80 bpw it is 35.7% smaller than anything the standard types can produce for this model.
• IQ2_NL also outperforms IQ1_M. It is 7.8% smaller and 6.8% lower KLD than the fork's own IQ1_M.
• Nothing at Q4_K and above changes. Those four quantized models are bit-for-bit identical across the two test runs; the quantization recipes never select the new types at those levels.

2. Disadvantages

1. Codebook efficiency. At 2.50 bpw, IQ2_NL sits at the same bit-rate as IQ2_S and above IQ2_XS (2.31) and IQ2_XXS (2.06), which spend their bits on lattice codebooks rather than a per-32 scale. On well-aligned models the super-block I-quants should still win at the low end. The NL family's value is availability, not peak efficiency.
2. Kernel maturity. Quantization wall-clock is ~2.7× the standard path (41.8 s vs 15.8 s for the 2-bit type on this test). More significantly, IQ3_NL inference is materially slower: tg128 35.53 t/s against 51.16 for standard Q3_K (−30.6%) and 54.70 for standard IQ1_M (−35.0%); pp1024+tg1024 66.38 against 92.00 and 98.43 (−27.8% and −32.6%). Prompt processing is unaffected (pp512 600.65 vs 600.77), which points at the token-generation dot-product path specifically. The regression does not generalise to the new types as a class: IQ2_NL posts the fastest tg128 in the benchmark table at 57.86 t/s (+2.3% over standard Q2_K) and the fork's IQ1_M is +2.1% over its standard counterpart, both while being substantially smaller. That isolates the cost to IQ3_NL's kernel rather than to 32-element blocks in general.
3. The mid-to-high types are untouched. Q4_K through Q8_0 are bit identical between the two runs. Those recipes still fall back to Q5_0/Q5_1/Q8_0, inflating Q4_K to 5.87 bpw and Q6_K to 8.22 bpw. Routing those fallbacks to IQ4_NL would shrink Q4_K on this architecture with no change to the new types.
4. output.weight remains a fallback. The recipe requests Q5_K and gets Q5_1 at 6.00 bpw for a 1120 MiB tensor (~420 MiB quantized), or ~14% of the entire IQ2_NL model. An IQ5_NL, or simply requesting IQ4_NL instead, would recover a meaningful fraction of that.
5. Single architecture, single model. Every conclusion here is drawn from one 9B hybrid-SSM checkpoint against one calibration corpus (imatrix). The 256-divisibility argument generalises well but the specific metrics results do not.

3. Comparative Performance Tables

Lower is better for every metric except Top-1.

Columns prefixed S· describe the standard (naive) build; columns prefixed F· describe the fork build. Rows 4–7 are identical between the two test runs.

As row 3 shows, IQ3_NL is 24.0% smaller than the naive Q3_K it replaces, but that naive Q3_K is a 4.84 bpw model wearing a 3-bit label.

Extended Metrics

Prompt Processing + Text Generation Benchmarks

Actual File Structure

What each recipe actually wrote to disk, across the 122 quantizable tensors:

The Q2_K row is the clearest illustration of the problem: 74% of the tensors in a "2-bit" model are stored at 4.5 bpw. The Q4_K and Q5_K rows show the fallback resolves upward to Q5_0 (5.50 bpw) and Q5_1 (6.00 bpw) against a Q4_K/Q5_K nominal of 4.50/5.50 bpw, inflating those builds by roughly 1.4 and 0.9 bpw respectively.

4. Logs and Test Results