Devstral-Small-2507-FP8-Dynamic

Devstral-Small-2507-FP8-Dynamic

Model Overview

• Model Architecture: MistralForCausalLM
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Activation quantization: FP8
  • Weight quantization: FP8
• Release Date: 08/28/2025
• Version: 1.0
• Model Developers: Red Hat (Neural Magic)

Model Optimizations

This model was obtained by quantizing weights and activations of Devstral-Small-2507 to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%). Weight quantization also reduces disk size requirements by approximately 50%.

Creation

<details> This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below.

from transformers import AutoModelForCausalLM
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier


MODEL_ID = "mistralai/Devstral-Small-2507"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto")
recipe = QuantizationModifier(
    targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]
)
oneshot(model=model, recipe=recipe)
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-Dynamic"
model.save_pretrained(SAVE_DIR)

</details>

Deployment

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

vllm serve RedHatAI/Devstral-Small-2507-FP8-Dynamic --tensor-parallel-size 1 --tokenizer_mode mistral

Evaluation

The model was evaluated on popular coding tasks (HumanEval, HumanEval+, MBPP, MBPP+) via EvalPlus and vllm backend (v0.10.1.1). For evaluations, we run greedy sampling and report pass@1. The command to reproduce evals:

evalplus.evaluate --model "RedHatAI/Devstral-Small-2507-FP8-Dynamic" \
                  --dataset [humaneval|mbpp] \
                  --base-url http://localhost:8000/v1 \
                  --backend openai --greedy

Accuracy