Mistral-Nemo-Instruct-2407-FP8

Mistral-Nemo-Instruct-2407-FP8

This repository contains RedHatAI/Mistral-Nemo-Instruct-2407-FP8 together with a Furiosa Executable Bundle (FXB) for running it on FuriosaAI RNGD with Furiosa-LLM. The same model also runs on other frameworks (such as vLLM, SGLang, and Transformers); for usage with those, see the upstream RedHatAI/Mistral-Nemo-Instruct-2407-FP8 model card.

Overview

Mistral NeMo Instruct 2407 is a 12B instruction-tuned, auto-regressive dense transformer developed jointly by Mistral AI and NVIDIA for assistant-style text generation. This is the FP8-quantized variant published by RedHatAI, derived from mistralai/Mistral-Nemo-Instruct-2407. Its intended use is the same as the upstream RedHatAI/Mistral-Nemo-Instruct-2407-FP8.

• Architecture: Mistral NeMo (dense)
• Input / Output: Text / Text
• Supported Inference Engine: Furiosa LLM
• Supported Hardware: FuriosaAI RNGD

Quantization

The linear (Transformer block) weights are quantized to FP8 (static), and activations use dynamic FP8 quantization at runtime. Activations are quantized per token at inference time rather than with offline calibration. The KV cache stays in 16-bit precision.

Features

Mistral-Nemo-Instruct-2407-FP8 provides non-reasoning, assistant-style text generation. Its chat template supports system, user, and assistant messages.

Parallelism Strategy

On RNGD, Mistral-Nemo-Instruct-2407-FP8 runs with a tensor-parallel size of 16 PEs, which maps to two RNGD cards (8 PEs per card).

Usage

To run this model with Furiosa-LLM, follow the example commands below after installing Furiosa-LLM and its prerequisites.

Launch the server

The simplest way to serve the model is:

# Launch the server, listening on port 8000 by default
furiosa-llm serve furiosa-ai/Mistral-Nemo-Instruct-2407-FP8

When the server is ready, you will see:

INFO:     Started server process [27507]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Basic Usage

The server exposes an OpenAI-compatible API. You can send a request with curl:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "furiosa-ai/Mistral-Nemo-Instruct-2407-FP8",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
    }' \
    | python -m json.tool

Advanced Usage

Mistral-Nemo-Instruct-2407-FP8 is a non-reasoning chat model and does not require a reasoning parser or model-specific chat-template flags. For general serving options, see the Furiosa-LLM Server guide.

Learn more

• Tool Calling — supported parsers, tool-choice options, and examples
• Furiosa-LLM Server (furiosa-llm serve) — full OpenAI-compatible API reference and serving options
• RedHatAI/Mistral-Nemo-Instruct-2407-FP8 — upstream model card