Falcon-H1R-7B-FP8

<img src="https://huggingface.co/datasets/tiiuae/reasoning-images/resolve/main/falcon-h1r-logo.png" alt="drawing" width="800"/>

Falcon-H1R-7B-FP8

This repository presents post FP8 quantized Falcon-H1R-7B-FP8 via NVIDIA Model Optimizer, enabling efficient inference while preserving the strong reasoning introduced in the paper Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling.

Falcon-H1R-7B was trained via cold-start supervised fine-tuning with long reasoning traces and further enhanced by scaling RL with GRPO. The model demonstrates outstanding performance across various benchmark evaluations, including mathematics, programming, instruction following, and general logic.

Model Description

• Developed by: Technology Innovation Institute
• Model type: Causal decoder-only
• Architecture: Hybrid (Transformers + Mamba2) architecture
• Language(s): English, Multilingual
• License: Falcon-LLM License

Details

For more details on FP8 post-quantization for this model, please refer to the Falcon-H1R-FP8 technical blogpost.

For more details about the training protocol of this model, please refer to the Falcon-H1R technical blogpost and Technical Report.

Usage

Currently to use this model, you can either rely on Hugging Face transformers, vLLM or SGLang library.

Inference

Make sure to install the latest version of transformers or vLLM or SGLang.

pip install transformers
pip install mamba-ssm[causal-conv1d]

For vLLM, make sure to install latest vllm:

pip install vllm --extra-index-url https://wheels.vllm.ai/nightly

• FP8 support (vLLM): FP8 enablement from vLLM PR #32728 has been merged: PR.
• Memory footprint: Model weight memory drops from 14.2 GB (BF16) to 7.9 GB (FP8).
• Throughput: Inference throughput improves from ~1.2× to up to 1.5×, depending on batch size, prompt length, and generation length, with minimal accuracy impact for this post-training FP8-quantized model.
• Tensor parallelism (quantized models): Tensor parallel enablement from vLLM PR #33257 has been merged: PR.

Sampling Parameters

We recommend using a temperature of 0.6 and top-p as 0.95 with max new tokens up to 65536. For supported frameworks, you can adjust the repetition_penalty and presence_penalty parameters to reduce endless repetitions.

vLLM

For vLLM, simply start a server by executing the command below:

<details> <summary> Click to expand </summary> <br>

vllm serve tiiuae/Falcon-H1R-7B-FP8 \
  --tensor-parallel-size 1 \
  --data-parallel-size 1 \
  --reasoning-parser deepseek_r1
  --quantization modelopt

<br> Additional flags:

• You can reduce --max-model-len to preserve memory. Default value is 262144 which is quite large but not necessary for most scenarios.
• For function calling, append --enable-auto-tool-choice and --tool-call-parser hermes to the vllm serve command.

vLLM client execution code:

from openai import OpenAI
import json

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

completion = client.chat.completions.create(
    model="tiiuae/Falcon-H1R-7B-FP8",
    messages=[
        {"role": "user", "content": "If the product of two numbers is 360 and their GCD is 6, what is their LCM?"},
    ],
    temperature=0.6,
    top_p=0.95,
    max_tokens=65536
)

msg = completion.choices[0].message

print(json.dumps({
    "reasoning": msg.reasoning_content,
    "answer": msg.content
}, indent=2))

</details> <br>

Evaluation

Falcon-H1R achieves state of art results in reasoning benchmarks. Please refer to the Falcon-H1R-7B BF16 model card for full benchmark details. The table below highlights accuracy comparisons on selected benchmarks between the Falcon-H1R BF16 checkpoint and the Falcon-H1R FP8 post-quantized checkpoint.

<table border="1" style="width: 100%; text-align: center; border-collapse: collapse;"> <colgroup> <col style="width: 6%;"> <col style="width: 6%;"> <col style="width: 6%;"> </colgroup> <thead> <tr> <th>Benchmark</th> <th>Falcon-H1R-7B BF16</th> <th>Falcon-H1R-7B FP8</th> </tr> </thead> <tbody> <tr> <td>AIME25</td> <td>83.1</td> <td>82.3</td> </tr> <tr> <td>LCBv5-v6</td> <td>68.6</td> <td>67.6</td> </tr> <tr> <td>GPQA-D</td> <td>61.3</td> <td>61.2</td> </tr> </tbody> </table>

Useful links

• View our FP8 release blogpost.
• View our release blogpost.
• View our technical report.
• Feel free to join our discord server if you have any questions or to interact with our researchers and developers.

Acknowledgements

We sincerely thank the NVIDIA team — Sergio Perez, Shengliang Xu, Vadim Gimpelson, Mireille Fares, Liana Mikaelyan, Amit Kushwaha, and Adam Czekalowski — for their valuable collaboration and support in post-quantizing Falcon-H1R-7B to FP8.

Citation

If the Falcon-H1R family of reasoning models is helpful to your work, feel free to give us a cite.

@misc{falcon-h1r,
      title={Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scaling}, 
      author={Falcon LLM Team and Iheb Chaabane and Puneesh Khanna and Suhail Mohmad and Slim Frikha and Shi Hu and Abdalgader Abubaker and Reda Alami and Mikhail Lubinets and Mohamed El Amine Seddik and Hakim Hacid},
      year={2026},
      eprint={2601.02346},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2601.02346}, 
}