R1-Llama-8B-ReQAT-nvfp4-w4a4kv4-fake

R1-Llama-8B ReQAT NVFP4 W4A4KV4

This repository contains the NVFP4 W4A4KV4 ReQAT checkpoint derived from deepseek-ai/DeepSeek-R1-Distill-Llama-8B for research on low-precision reasoning.

Model and training

ReQAT combines Trace-Aligned QAT, Selective Entropy Minimization, and Quantization-Friendly Initialization via Transformation to preserve reasoning under FP4 quantization. The released models use the mathematics subset of OpenThoughts3-1.2M.

Checkpoint format

Weights are stored as BF16 safetensors. The configuration enables cfg=nvfp4, kv_cfg=nvfp4_e1m2, kv_quant=true, and shift_k=true. The ReQAT runtime applies the configured fake quantization and post-RoPE key shift.

Use the ReQAT repository's vllm_custom model implementation. inference.py registers the custom architecture before loading; a stock Transformers pipeline or unmodified vLLM loader does not implement this configuration.

Usage

Use a working ReQAT environment with the repository's ModelOpt, vLLM and lighteval components. The project targets Hopper-generation NVIDIA GPUs (H100/H200) or newer, including B200. Fake-quantized storage retains BF16 tensors, so size GPU memory for the stored model, KV cache and runtime buffers.

git clone https://github.com/aiha-lab/ReQAT.git
cd ReQAT
git checkout 55038ae6ac807a33f9192a25a03f54a03789d21e

CUDA_VISIBLE_DEVICES=0 ENABLE_THINKING=true python -m inference \
  --model superdocker/R1-Llama-8B-ReQAT-nvfp4-w4a4kv4-fake \
  --dataset MATH-500 \
  --temperature 0.6 \
  --top_p 0.95 \
  --seed 42

The command assumes the project dependencies have been installed. The visible GPU count determines tensor parallelism; adjust CUDA_VISIBLE_DEVICES to your hardware. The repository also supports AIME-2024, AIME-2025, AIME-90, GSM8K, GPQA-Diamond and LiveCodeBench. See inference.py for available arguments.

Evaluation

The paper reports 41.85% AIME-120 accuracy for R1-Llama-8B with NVFP4 W4A4KV4 (full TAQ + Q-FIT + SEM, 350M training tokens; Table 2). The reported value is for the paper configuration; it has not been independently verified for this exact released checkpoint.

Intended use and limitations

Use this checkpoint for reasoning research, quantization experiments and benchmark evaluation. Generated answers can be incorrect, biased or misleading and require verification. The fake-quantized release does not itself establish native 4-bit memory or throughput gains.

For W4A4KV4 checkpoints, the project's deploy/trtllm/ path separately exports native NVFP4 for Blackwell throughput evaluation. Stock TensorRT-LLM does not reproduce ReQAT's E1M2 KV-cache and K-shift behavior, so accuracy can differ from the research runtime.

License

This release uses the MIT license. The ReQAT code is also MIT-licensed. The base model is distributed by DeepSeek under MIT and derives from Meta Llama 3.1 8B Base. The upstream Llama 3.1 Community License and associated usage conditions also apply. See the base model license section.

Citation

ICML 2026 paper | arXiv | Code

@InProceedings{pmlr-v306-lee26an,
  title = {{R}e{QAT}: Achieving Full-Precision Reasoning Accuracy with
           4-bit Floating-Point Quantization-Aware Training},
  author = {Lee, Janghwan and Lee, Sihwa and Kim, Jinseok and Kim, Yongjik
            and Lim, Jieun and Oh, Jinwook and Choi, Jungwook},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  pages = {64525--64546},
  year = {2026},
  volume = {306},
  series = {Proceedings of Machine Learning Research},
  publisher = {PMLR},
  url = {https://proceedings.mlr.press/v306/lee26an.html}
}