Llama-4-Maverick-17B-128E-Instruct-speculator.eagle3

Llama-4-Maverick-17B-128E-Instruct-speculators.eagle3

Model Overview

• Verifier: meta-llama/Llama-4-Maverick-17B-128E-Instruct
• Speculative Decoding Algorithm: EAGLE-3
• Model Architecture: Eagle3Speculator
• Release Date: 09/17/2025
• Version: 1.0
• Model Developers: RedHat

This is a speculator model designed for use with meta-llama/Llama-4-Maverick-17B-128E-Instruct, based on the EAGLE-3 speculative decoding algorithm. It was converted into the speculators format from the model nvidia/Llama-4-Maverick-17B-128E-Eagle3. This model should be used with the meta-llama/Llama-4-Maverick-17B-128E-Instruct chat template, specifically through the /chat/completions endpoint.

Use with vLLM

vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  -tp 8 \
  --speculative-config '{
    "model": "RedHatAI/Llama-4-Maverick-17B-128E-Instruct-speculators.eagle3",
    "num_speculative_tokens": 3,
    "method": "eagle3"
  }'

Evaluations

<h3>Use cases</h3> <table> <thead> <tr> <th>Use Case</th> <th>Dataset</th> <th>Number of Samples</th> </tr> </thead> <tbody> <tr> <td>Coding</td> <td>HumanEval</td> <td>168</td> </tr> <tr> <td>Math Reasoning</td> <td>gsm8k</td> <td>80</td> </tr> <tr> <td>Text Summarization</td> <td>CNN/Daily Mail</td> <td>80</td> </tr> </tbody> </table> <h3>Acceptance lengths</h3> <table> <thead> <tr> <th>Use Case</th> <th>k=1</th> <th>k=2</th> <th>k=3</th> <th>k=4</th> <th>k=5</th> <th>k=6</th> <th>k=7</th> </tr> </thead> <tbody> <tr> <td>Coding</td> <td>1.83</td> <td>2.45</td> <td>2.94</td> <td>3.26</td> <td>3.47</td> <td>3.57</td> <td>3.62</td> </tr> <tr> <td>Math Reasoning</td> <td>1.86</td> <td>2.56</td> <td>3.08</td> <td>3.53</td> <td>3.73</td> <td>3.91</td> <td>4.02</td> </tr> <tr> <td>Text Summarization</td> <td>1.69</td> <td>2.12</td> <td>2.37</td> <td>2.52</td> <td>2.60</td> <td>2.63</td> <td>2.63</td> </tr> </tbody> </table> <h3>Performance benchmarking (2xA100)</h3> <div style="display: flex; justify-content: center; gap: 20px;"> <figure style="text-align: center;"> <img src="assets/Llama-4-Maverick-17B-128E-Instruct-coding.png" alt="Coding" width="100%"> </figure> <figure style="text-align: center;"> <img src="assets/Llama-4-Maverick-17B-128E-Instruct-math_reasoning.png" alt="Math Reasoning" width="100%"> </figure> <figure style="text-align: center;"> <img src="assets/Llama-4-Maverick-17B-128E-Instruct-summarization.png" alt="Text Summarization" width="100%"> </figure> </div> <details> <summary>Details</summary> <strong>Configuration</strong>

• temperature: 0.6
• top_p: 0.9
• repetitions: 3
• time per experiment: 3min
• hardware: 8xB200
• vLLM version: 0.11.0
• GuideLLM version: 0.3.0

<strong>Command</strong>

GUIDELLM__PREFERRED_ROUTE="chat_completions" \
guidellm benchmark \
  --target "http://localhost:8000/v1" \
  --data "RedHatAI/speculator_benchmarks" \
  --data-args '{"data_files": "HumanEval.jsonl"}' \
  --rate-type sweep \
  --max-seconds 180 \
  --output-path "Llama-4-Maverick-HumanEval.json" \
  --backend-args '{"extra_body": {"chat_completions": {"temperature":0.6, "top_p":0.9}}}'

GuideLLM interface changed, so for compatibility with the latest version (v0.6.0), please use the following command:

GUIDELLM__PREFERRED_ROUTE="chat_completions" \
guidellm benchmark \
  --target "http://localhost:8000/v1" \
  --data "RedHatAI/speculator_benchmarks" \
  --data-args '{"data_files": "HumanEval.jsonl"}' \
  --profile sweep \
  --max-seconds 1800 \
  --output-path "my_output.json" \
  --backend-args '{"extras": {"body": {"temperature":0.6, "top_p":0.95, "top_k":20}}}'

</details>

Citation

If you use this model, please cite both the original NVIDIA model and the Speculators library:

@misc{nvidia2025llama4maverick,
  title={Llama 4 Maverick 17B Eagle3},
  author={NVIDIA Corporation},
  year={2025},
  publisher={Hugging Face}
}

@misc{speculators2024,
  title={Speculators: A Unified Library for Speculative Decoding},
  author={Neural Magic},
  year={2024},
  url={https://github.com/neuralmagic/speculators}
}

Acknowledgments

• Original model by NVIDIA Corporation
• Conversion and formatting for Speculators/vLLM compatibility
• Based on Eagle3 architecture with Llama3 draft head targeting Llama4 verifier