Mistral-Nemo-Instruct-2407-NVFP4-FP8

Mistral-Nemo-Instruct-2407-NVFP4-FP8

A version of Mistral-Nemo-Instruct-2407 created with llm-compressor 0.10.0 and compressed-tensors 0.14.0, mainly to test out a hybrid quantization format. The goal was to improve accuracy compared to regular NVFP4 with minimal impact on speed and VRAM usage, with the specific goal of remaining small enough to support a 32k-token context window in Aphrodite Engine on a RTX 5060 Ti 16GB.

This repository contains a quantized version of Mistral-Nemo-Instruct-2407

Quantization Format

The quant uses a mixed-precision approach, with different types of layers undergoing different amounts of quantization.

• Self-attention layers: FP8_DYNAMIC, a format that uses FP8 weights and activations, with per-channel BF16 scales for the weights and dynamic per-token scales for the activations. Self-attention tensors comprise slightly less than 20 percent of the weights in the linear layers, meaning that upgrading them from NVFP4 to FP8 has a very limited effect on model size and VRAM use, and the nature of attention means that as context length increases, the benefits of doing attention calculations in w8a8 rather than w4a4 become more pronounced.
• MLP layers: NVFP4 using the Four Over Six (4/6) quantization method described in the paper Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling. As this is purely a matter of how weights are selected rather than a change in format, it increases accuracy compared to regular NVFP4 without impacting performance or requiring any changes at runtime.
  • Code: https://github.com/mit-han-lab/fouroversix
  • Paper: Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
• lm_head, embed_tokens, layernorms: left in BF16. These are also left alone by regular NVFP4, since they're relatively small and very sensitive to quantization error.

More about Four Over Six

One of the main downsides of using FP4 is the extreme sparsity of large values. At a base level, NVFP4 works by dividing the model into sixteen-element blocks, then assigning FP8 scale factors to each block (as well as a single FP32 scale factor for the tensor as a whole) such that the largest absolute value in the block maps to ±6. For example, if a block has the values {10,-20,40,-60}, the scale factor would be set to 10 and the FP4 values would be {1,-2,4,-6}. The problem is that the FP4 format only allows for a very limited set of values. In particular, it can't represent any number between 4 and 6, so anything in the block that maps to ±5 will be severely affected by rounding error. In the following two graphs, the x-axis represents the value a given weight would have with the corresponding scaling, and the y-axis represents how much proportional error would be introduced by rounding it to the nearest valid FP4 value:

However, while scaling to ±4 reduces worst-case rounding error for large values, it increases rounding error for smaller values, so simply scaling every block to ±4 would be a bad idea. The solution is to try scaling each block both ways, then keep whichever gives the lowest quantization MSE for that block. The memoryless_mse observer in llm-compressor is designed to work on a similar principle, calculating scale factors as though the weights were multiplied by different values of \(p\) and choosing the scale that minimizes quantization error for each block. While this is primarily intended for using \(p\le1\) to allow extra precision for small values at the cost of clipping large values, when used with NVFP4 it's mathematically equivalent to mapping the most extreme values in each block to \(±6/p\). Obviously, this can be used to implement Four Over Six by setting \(p\in{1,1.5}\). The key to doing this is the following code from mse.py:

for i in range(int(maxshrink * grid)):
    p = 1 - i / grid

With maxshrink set to -1 and grid set to -2, this loop will run twice: once with \(p=1\) and a second time with \(p=1.5\). The comparisons are done by taking the absolute value of the difference between original and quantized values and raising it to the power of norm, then summing that for each block, so setting norm to 2 is equivalent to using quantization MSE.

Benchmarks

To test the quantization format, I compared this model to a version quantized with regular NVFP4 (available here) in a variety of benchmarks using lm_eval. For benchmarks that provided standard error in addition to the score, I also computed a \(z\) score by dividing the difference in scores by \(\sqrt{s_1^2+s_2^2}\) and used that to test for statistical significance. The results of the benchmarks are in the table below, with the cases where the difference was statistically significant \((\left|z\right|\ge1.96)\) in bold. Even for the tests where the results weren't individually significant, the fact that the hybrid got the better score on all of them except winogrande seems rather telling.

I also tested versions with just Four Over Six block scaling (here) and just FP8 self-attention (here) for the sake of thoroughness. Both were generally better than the NVFP4 baseline, and the hybrid quant was generally better than either separately. Detailed results are included in json format here, csv format here and md format here.

VRAM Usage

In Aphrodite Engine with the --single-user-mode flag, running the model with a KV cache size of 32,768 tokens used 15,157 MiB of VRAM.

In vLLM 0.19.0, which is somewhat less efficient with its VRAM use, using --gpu-memory-utilization 0.9 and --max-model-len auto on a RTX 5060 Ti 16GB allowed a KV cache size of up to 30,304 tokens and used 15,415 MiB of VRAM.

Prefill speed

I benchmarked model speed by splitting sample texts into \(n\)-token chunks and running requests for the next token (thus forcing the model to prefill the same number of tokens repeatedly), then recording Aphrodite Engine's average prefill speed at different values of \(n\) for both the NVFP4 and hybrid quants (as Four Over Six has no impact on performance, I only needed to compare the two models this time).

As shown in the following graph, the difference in speed shrinks as context length grows and memory bandwidth overtakes compute as the limiting factor.

Long-context Perplexity

For this test, I split sample texts into \(n\)-token chunks and computed perplexity scores for each chunk in all four quantized models: the NVFP4 baseline, one with Four Over Six weight selection, one with the attention tensors in FP8, and the hybrid format. I then recorded the average perplexity score for each model at each chunk size. Sample texts for this step were UTF-8 text files taken from Project Gutenberg, listed below.

<details> <summary>Sample texts used</summary>

• Pride and Prejudice
• Frankenstein
• Wuthering Heights
• Dracula

</details>

While perplexity for all quants increases with context length past 8192, the chart is very different from the one for performance and rather informative. Changing between Four Over Six and the default NVFP4 weight selection was a linear change in both the pure NVFP4 model and the one with FP8 attention. The two models with FP8 attention diverge from the two without as context length changes, however, indicating that as the number of tokens attending to each other increases, the benefits of doing attention calculations in higher precision become more pronounced.

Inference

This model requires compressed-tensors 0.14.0 or later and has been tested on both vLLM and Aphrodite Engine.

Credits

Mistral-Nemo-Instruct-2407 was made by Mistral AI and nVidia

Four Over Six was discovered by Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, and Song Han

Citation

@misc{cook2025sixaccuratenvfp4quantization,
      title={Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling},
      author={Jack Cook and Junxian Guo and Guangxuan Xiao and Yujun Lin and Song Han},
      year={2025},
      eprint={2512.02010},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2512.02010},
}