Pixtral-Large-Instruct-2411-hf-FP8-dynamic

Model Card for Pixtral-Large-Instruct-2411

Pixtral-Large-Instruct-2411 is a 124B multimodal model built on top of Mistral Large 2, i.e., Mistral-Large-Instruct-2407. Pixtral Large is the second model in our multimodal family and demonstrates frontier-level image understanding. Particularly, the model is able to understand documents, charts and natural images, while maintaining the leading text-only understanding of Mistral Large 2.

For more details about this model please refer to the Pixtral Large blog post and the Pixtral 12B blog post.

Key features

• Frontier-class multimodal performance
• State-of-the-art on MathVista, DocVQA, VQAv2
• Extends Mistral Large 2 without compromising text performance
• 123B multimodal decoder, 1B parameter vision encoder
• 128K context window: fits minimum of 30 high-resolution images

<!-- - **Multi-lingual by design:** Dozens of languages supported, including English, French, German, Spanish, Italian, Chinese, Japanese, Korean, Portuguese, Dutch and Polish. - **Proficient in coding:** Trained on 80+ coding languages such as Python, Java, C, C++, Javacsript, and Bash. Also trained on more specific languages such as Swift and Fortran. - **Agentic-centric:** Best-in-class agentic capabilities with native function calling and JSON outputting. - **Advanced Reasoning:** State-of-the-art mathematical and reasoning capabilities. - **Mistral Research License:** Allows usage and modification for research and non-commercial usages. - **Large Context:** A large 128k context window. - **System Prompt:** Maintains strong adherence and support for more reliable system prompts. - **Vision:** A 1B parameter Vision Encoder achieving SOTA vision capabilities. -->

System Prompt Handling

We appreciate the feedback received from our community regarding our system prompt handling.
In response, we have implemented stronger support for system prompts.
To achieve optimal results, we recommend always including a system prompt that clearly outlines the bot's purpose, even if it is minimal.

Basic Instruct Template (V7)

<s>[SYSTEM_PROMPT] <system prompt>[/SYSTEM_PROMPT][INST] <user message>[/INST] <assistant response></s>[INST] <user message>[/INST]

Be careful with subtle missing or trailing white spaces!

Please make sure to use mistral-common as the source of truth

Metrics

Specific model versions evaluated: Claude-3.5 Sonnet (new) [Oct 24], Gemini-1.5 Pro (002) [Sep 24], GPT-4o (2024-08-06) [Aug 24].

See mistral-evals for open-source MM MT-Bench evaluation scripts.

Usage

The model can be used with the following frameworks

• vllm: See here

vLLM

We recommend using Pixtral-Large-Instruct-2411 with the vLLM library to implement production-ready inference pipelines with Pixtral-Large-Instruct-2411.

Installation

Make sure you install vLLM >= v0.6.4.post1:

pip install --upgrade vllm

Also make sure you have mistral_common >= 1.5.0 installed:

pip install --upgrade mistral_common

You can also make use of a ready-to-go docker image or on the docker hub.

Server (Image)

We recommend to use Pixtral-Large-Instruct-2411 in a server/client setting.

1. Spin up a server:

vllm serve mistralai/Pixtral-Large-Instruct-2411 --config-format mistral --load-format mistral --tokenizer_mode mistral --limit_mm_per_prompt 'image=10' --tensor-parallel-size 8

1. And ping the client:

import requests
import json
from huggingface_hub import hf_hub_download
from datetime import datetime, timedelta

url = "http://<your-server-url>:8000/v1/chat/completions"
headers = {"Content-Type": "application/json", "Authorization": "Bearer token"}

model = "mistralai/Pixtral-Large-Instruct-2411"


def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")

image_url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/europe.png"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Which of the depicted countries has the best food? Which the second and third and fourth? Name the country, its color on the map and one its city that is visible on the map, but is not the capital. Make absolutely sure to only name a city that can be seen on the map.",
            },
            {"type": "image_url", "image_url": {"url": image_url}},
        ],
    },
]

data = {"model": model, "messages": messages}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json()["choices"][0]["message"]["content"])
# Determining which country has the "best" food can be subjective and depends on personal preferences. However, based on popular culinary reputations, here are some countries known for their cuisine:

#1. **Italy** (Brown) - Known for its pasta, pizza, and diverse regional dishes.
#   - City: Milan

#2. **France** (Dark Brown) - Renowned for its fine dining, pastries, and wine.
#   - City: Lyon

#3. **Spain** (Yellow) - Famous for tapas, paella, and a variety of seafood dishes.
#   - City: Barcelona

#4. **Greece** (Yellow) - Known for its Mediterranean cuisine, including moussaka, souvlaki, and fresh seafood.
#   - City: Thessaloniki

#These rankings are based on general culinary reputations and can vary widely depending on individual tastes.

Server (Text-only)

You can also ping the client with a text-only example. The following example shows how the system prompt can be used to make sure the model always knows the current date.

import requests
import json
from huggingface_hub import hf_hub_download
from datetime import datetime, timedelta

url = "http://<your-server-url>:8000/v1/chat/completions"
headers = {"Content-Type": "application/json", "Authorization": "Bearer token"}

model = "mistralai/Pixtral-Large-Instruct-2411"


def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")

image_url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/europe.png"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": "Without browsing the web, how many days ago was Mistral founded?"
    },
]

data = {"model": model, "messages": messages}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json()["choices"][0]["message"]["content"])
# Mistral AI was founded in April 2023. Since the current date is November 18, 2024, we can calculate the number of days between April 2023 and November 18, 2024.

#First, calculate the days from April 2023 to the end of 2023:
#- April: 27 days (30 - 3)
#- May: 31 days
#- June: 30 days
#- July: 31 days
#- August: 31 days
#- September: 30 days
#- October: 31 days
#- November: 30 days
#- December: 31 days

#Total days from April 2023 to December 31, 2023: 27 + 31 + 30 + 31 + 31 + 30 + 31 + 30 + 31 = 272 days

#Next, calculate the days from January 1, 2024, to November 18, 2024:
#- January: 31 days
#- February: 29 days (2024 is a leap year)
#- March: 31 days
#- April: 30 days
#- May: 31 days
#- June: 30 days
#- July: 31 days
#- August: 31 days
#- September: 30 days
#- October: 31 days
#- November: 18 days

#Total days from January 1, 2024, to November 18, 2024: 31 + 29 + 31 + 30 + 31 + 30 + 31 + 31 + 30 + 31 + 18 = 323 days

#Adding the two periods together:
#272 days (from April 2023 to December 2023) + 323 days (from January 2024 to November 18, 2024) = 595 days

#Therefore, Mistral AI was founded 595 days ago from November 18, 2024.

Offline Example

from vllm import LLM
from vllm.sampling_params import SamplingParams
from huggingface_hub import hf_hub_download
from datetime import datetime, timedelta

model_name = "mistralai/Pixtral-Large-Instruct-2411"

def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, 'r') as file:
        system_prompt = file.read()
    today = datetime.today().strftime('%Y-%m-%d')
    yesterday = (datetime.today() - timedelta(days=1)).strftime('%Y-%m-%d')
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)

SYSTEM_PROMPT = load_system_prompt(model_name, "SYSTEM_PROMPT.txt")

image_url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/europe.png"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Which of the depicted countries has the best food? Which the second and third and fourth? Name the country, its color on the map and one its city that is visible on the map, but is not the capital. Make absolutely sure to only name a city that can be seen on the map.",
            },
            {"type": "image_url", "image_url": {"url": image_url}},
        ],
    },
]

sampling_params = SamplingParams(max_tokens=512)

# note that running this model on GPU requires over 300 GB of GPU RAM
llm = LLM(model=model_name, config_format="mistral", load_format="mistral", tokenizer_mode="mistral", tensor_parallel_size=8, limit_mm_per_prompt={"image": 4})

outputs = llm.chat(messages, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)

Evaluation

The model was evaluated using mistral-evals for vision-related tasks and using lm_evaluation_harness for select text-based benchmarks. The evaluations were conducted using the following commands:

<details> <summary>Evaluation Commands</summary>

Vision Tasks

• vqav2
• docvqa
• mathvista
• mmmu
• chartqa

vllm serve neuralmagic/pixtral-12b-quantized.w8a8 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7

python -m eval.run eval_vllm \
        --model_name neuralmagic/pixtral-12b-quantized.w8a8 \
        --url http://0.0.0.0:8000 \
        --output_dir ~/tmp \
        --eval_name <vision_task_name>

Text-based Tasks

MMLU

lm_eval \
  --model vllm \
  --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
  --tasks mmlu \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path output_dir

MGSM

lm_eval \
  --model vllm \
  --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
  --tasks mgsm_cot_native \
  --apply_chat_template \
  --num_fewshot 0 \
  --batch_size auto \
  --output_path output_dir

</details>

Accuracy

<table> <thead> <tr> <th>Category</th> <th>Metric</th> <th>neuralmagic/Pixtral-Large-Instruct-2411-hf</th> <th>neuralmagic/Pixtral-Large-Instruct-2411-hf-FP8-dynamic</th> <th>Recovery (%)</th> </tr> </thead> <tbody> <tr> <td rowspan="6"><b>Vision</b></td> <td>MMMU (val, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td> <td>63.56</td> <td>63.44</td> <td>99.81%</td> </tr> <tr> <td>VQAv2 (val)<br><i>vqa_match</i></td> <td>79.03</td> <td>79.06</td> <td>100.04%</td> </tr> <tr> <td>DocVQA (val)<br><i>anls</i></td> <td>89.55</td> <td>89.63</td> <td>100.09%</td> </tr> <tr> <td>ChartQA (test, CoT)<br><i>anywhere_in_answer_relaxed_correctness</i></td> <td>82.24</td> <td>82.80</td> <td>100.68%</td> </tr> <tr> <td>Mathvista (testmini, CoT)<br><i>explicit_prompt_relaxed_correctness</i></td> <td>67.3</td> <td>66.50</td> <td>98.81%</td> </tr> <tr> <td><b>Average Score</b></td> <td><b>76.34</b></td> <td><b>76.29</b></td> <td><b>99.93%</b></td> </tr> <tr> <td rowspan="2"><b>Text</b></td> <td>MGSM (CoT)</td> <td>76.05</td> <td>75.58</td> <td>99.38%</td> </tr> <tr> <td>MMLU (5-shot)</td> <td>82.8</td> <td>82.74</td> <td>99.93%</td> </tr> </tbody> </table>

Inference Performance

This model achieves up to 1.9x speedup in single-stream deployment and up to 1.78x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario. The following performance benchmarks were conducted with vLLM version 0.7.2, and GuideLLM.

<details> <summary>Benchmarking Command</summary> ``` guidellm --model neuralmagic/Pixtral-Large-Instruct-2411-hf-FP8-dynamic --target "http://localhost:8000/v1" --data-type emulated --data prompt_tokens=<prompt_tokens>,generated_tokens=<generated_tokens>,images=<num_images>,width=<image_width>,height=<image_height> --max seconds 120 --backend aiohttp_server ``` </details>

Single-stream performance (measured with vLLM version 0.7.2)

<table border="1" class="dataframe"> <thead> <tr> <th></th> <th></th> <th></th> <th></th> <th style="text-align: center;" colspan="2" >Document Visual Question Answering<br>1680W x 2240H<br>64/128</th> <th style="text-align: center;" colspan="2" >Visual Reasoning <br>640W x 480H<br>128/128</th> <th style="text-align: center;" colspan="2" >Image Captioning<br>480W x 360H<br>0/128</th> </tr> <tr> <th>Hardware</th> <th>Number of GPUs</th> <th>Model</th> <th>Average Cost Reduction</th> <th>Latency (s)</th> <th>Queries Per Dollar</th> <th>Latency (s)</th> <th>Queries Per Dollar</th> <th>Latency (s)</th> <th>Queries Per Dollar</th> </tr> </thead> <tbody style="text-align: center"> <tr> <th rowspan="3" valign="top">A100</th> <td>4</td> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf</td> <td></td> <td>7.5</td> <td>67</td> <td>6.5</td> <td>77</td> <td>6.4</td> <td>79</td> </tr> <tr> <td>2</td> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w8a8</td> <td>1.86</td> <td>8.1</td> <td>124</td> <td>7.1</td> <td>142</td> <td>6.8</td> <td>148</td> </tr> <tr> <td>2</td> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td> <td>2.52</td> <td>6.9</td> <td>147</td> <td>5.1</td> <td>199</td> <td>4.5</td> <td>221</td> </tr> <tr> <th rowspan="3" valign="top">H100</th> <td>4</td> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf</td> <td></td> <td>4.4</td> <td>67</td> <td>3.9</td> <td>74</td> <td>3.7</td> <td>79</td> </tr> <tr> <td>2</td> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-FP8-Dynamic</td> <td>1.82</td> <td>4.7</td> <td>120</td> <td>4.1</td> <td>137</td> <td>3.9</td> <td>145</td> </tr> <tr> <td>2</td> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td> <td>1.87</td> <td>4.7</td> <td>120</td> <td>3.9</td> <td>144</td> <td>3.8</td> <td>149</td> </tr> </tbody> </table>

**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens

**QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025).

Multi-stream asynchronous performance (measured with vLLM version 0.7.2)

<table border="1" class="dataframe"> <thead> <tr> <th></th> <th></th> <th></th> <th style="text-align: center;" colspan="2" >Document Visual Question Answering<br>1680W x 2240H<br>64/128</th> <th style="text-align: center;" colspan="2" >Visual Reasoning <br>640W x 480H<br>128/128</th> <th style="text-align: center;" colspan="2" >Image Captioning<br>480W x 360H<br>0/128</th> </tr> <tr> <th>Hardware</th> <th>Model</th> <th>Average Cost Reduction</th> <th>Maximum throughput (QPS)</th> <th>Queries Per Dollar</th> <th>Maximum throughput (QPS)</th> <th>Queries Per Dollar</th> <th>Maximum throughput (QPS)</th> <th>Queries Per Dollar</th> </tr> </thead> <tbody style="text-align: center"> <tr> <th rowspan="3" valign="top">A100x4</th> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf</td> <td></td> <td>0.4</td> <td>222</td> <td>0.7</td> <td>341</td> <td>0.8</td> <td>399</td> </tr> <tr> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w8a8</td> <td>1.70</td> <td>0.8</td> <td>383</td> <td>1.1</td> <td>571</td> <td>1.3</td> <td>674</td> </tr> <tr> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td> <td>1.48</td> <td>0.5</td> <td>276</td> <td>1.0</td> <td>505</td> <td>1.4</td> <td>680</td> </tr> <tr> <<th rowspan="3" valign="top">H100x4</th> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf</td> <td></td> <td>1.0</td> <td>284</td> <td>1.6</td> <td>465</td> <td>1.8</td> <td>511</td> </tr> <tr> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-FP8-Dynamic</td> <td>1.61</td> <td>1.7</td> <td>467</td> <td>2.6</td> <td>726</td> <td>3.2</td> <td>908</td> </tr> <tr> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td> <td>1.33</td> <td>1.4</td> <td>393</td> <td>2.2</td> <td>726</td> <td>2.7</td> <td>764</td> </tr> </tbody> </table>

**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens

**QPS: Queries per second.

**QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025).

The Mistral AI Team

Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Alok Kothari, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Augustin Garreau, Austin Birky, Bam4d, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Carole Rambaud, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Diogo Costa, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gaspard Blanchet, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Henri Roussez, Hichem Sattouf, Ian Mack, Jean-Malo Delignon, Jessica Chudnovsky, Justus Murke, Kartik Khandelwal, Lawrence Stewart, Louis Martin, Louis Ternon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Marjorie Janiewicz, Mickaël Seznec, Nicolas Schuhl, Niklas Muhs, Olivier de Garrigues, Patrick von Platen, Paul Jacob, Pauline Buche, Pavan Kumar Reddy, Perry Savas, Pierre Stock, Romain Sauvestre, Sagar Vaze, Sandeep Subramanian, Saurabh Garg, Sophia Yang, Szymon Antoniak, Teven Le Scao, Thibault Schueller, Thibaut Lavril, Thomas Wang, Théophile Gervet, Timothée Lacroix, Valera Nemychnikova, Wendy Shang, William El Sayed, William Marshall### Multi-stream asynchronous performance (measured with vLLM version 0.7.2)

<table border="1" class="dataframe"> <thead> <tr> <th></th> <th></th> <th></th> <th style="text-align: center;" colspan="2" >Document Visual Question Answering<br>1680W x 2240H<br>64/128</th> <th style="text-align: center;" colspan="2" >Visual Reasoning <br>640W x 480H<br>128/128</th> <th style="text-align: center;" colspan="2" >Image Captioning<br>480W x 360H<br>0/128</th> </tr> <tr> <th>Hardware</th> <th>Model</th> <th>Average Cost Reduction</th> <th>Maximum throughput (QPS)</th> <th>Queries Per Dollar</th> <th>Maximum throughput (QPS)</th> <th>Queries Per Dollar</th> <th>Maximum throughput (QPS)</th> <th>Queries Per Dollar</th> </tr> </thead> <tbody style="text-align: center"> <tr> <th rowspan="3" valign="top">A100x4</th> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf</td> <td></td> <td>0.4</td> <td>222</td> <td>0.7</td> <td>341</td> <td>0.8</td> <td>399</td> </tr> <tr> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w8a8</td> <td>1.70</td> <td>0.8</td> <td>766</td> <td>1.1</td> <td>1142</td> <td>1.3</td> <td>1348</td> </tr> <tr> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td> <td>1.48</td> <td>0.5</td> <td>552</td> <td>1.0</td> <td>1010</td> <td>1.4</td> <td>1360</td> </tr> <tr> <<th rowspan="3" valign="top">H100x4</th> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf</td> <td></td> <td>1.0</td> <td>284</td> <td>1.6</td> <td>465</td> <td>1.8</td> <td>511</td> </tr> <tr> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-FP8-Dynamic</td> <td>1.61</td> <td>1.7</td> <td>905</td> <td>2.6</td> <td>1406</td> <td>3.2</td> <td>1759</td> </tr> <tr> <td>neuralmagic/Pixtral-Large-Instruct-2411-hf-quantized.w4a16</td> <td>1.33</td> <td>1.4</td> <td>761</td> <td>2.2</td> <td>1228</td> <td>2.7</td> <td>1480</td> </tr> </tbody> </table>

**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens

**QPS: Queries per second.

**QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025).