Mellum2-12B-A2.5B-Thinking-speculator.dflash

RedHatAI/Mellum2-12B-A2.5B-Thinking-speculator.dflash

Mellum2-12B-A2.5B-Thinking-speculator.dflash is a machine learning model from RedHatAI. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for speculators. The card lists the license as apache-2.0.

This is a DFlash speculator model for JetBrains/Mellum2-12B-A2.5B-Thinking with sliding window attention.

Downloads · 30 days

166

34% of all-time downloads

All-time downloads

495

Public

Parameters

681M

2.7 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors1.4 GB · 100%

Parameter types

How the weights are stored.

BF16681M · 100%

Base models

JetBrains/Mellum2-12B-A2.5B-Thinking

Model card

From the Hugging Face model README

RedHatAI/Mellum2-12B-A2.5B-Thinking-Dflash

This is a DFlash speculator model for JetBrains/Mellum2-12B-A2.5B-Thinking with sliding window attention.

It was trained using the Speculators library on a combination of the Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered dataset and the train_sft split of the HuggingFaceH4/ultrachat_200k dataset. Responses were regenerated by JetBrains/Mellum2-12B-A2.5B-Thinking.

Model Specifications


Base Model	JetBrains/Mellum2-12B-A2.5B-Thinking
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia H100

Training Details

This model was trained using the Speculators library and the helper scripts provided in the repo.

Launch vLLM

python3 launch_vllm.py JetBrains/Mellum2-12B-A2.5B-Thinking \
  --hidden-states-path /tmp \
  --target-layer-ids 1 6 12 17 23 28 \
  -- \
  --max-model-len 16384 \
  -dp 2 \
  --max_num_batched_tokens 16384 \
  --max_num_seqs 4096 \
  --gpu-memory-utilization 0.9 \
  --no-enable-chunked-prefill

Launch Training

Must be run once vLLM has finished launching and is running in the background.

torchrun \
  --standalone \
  --nproc_per_node=2 scripts/train.py \
  --verifier-name-or-path JetBrains/Mellum2-12B-A2.5B-Thinking \
  --data-path "prepared_data" \
  --on-missing generate \
  --on-generate delete \
  --scheduler-type cosine \
  --draft-vocab-size 32000 \
  --max-anchors 3072 \
  --target-layer-ids 1 6 12 17 23 \
  --speculator-type dflash \
  --num-layers 5 \
  --logger trackio \
  --run-name demo_qwen3_9b \
  --lr 0.0006 \
  --epochs 10 \
  --num-workers 16 \
  --sliding-window-indices 0 1 2 3 4

Deployment

Deploy with vLLM using the speculator as a draft model.

First install vLLM nightly:

uv pip install -U vllm \
    --torch-backend=auto \
    --extra-index-url https://wheels.vllm.ai/nightly

Then run:

vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \
    --speculative-config '{ \
        "model": "RedHatAI/Mellum2-12B-A2.5B-Thinking-Dflash", \
        "num_speculative_tokens": 16, \
        "method": "dflash" \
    }'

Preliminary Evaluations

Per-Position Acceptance Rate

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Avg. Length
HumanEval	72.5%	50.9%	34.3%	23.4%	16.4%	11.4%	7.6%	3.17
math_reasoning	80.9%	63.3%	48.5%	36.8%	26.7%	19.1%	12.9%	3.88
qa	64.1%	38.4%	23.0%	13.7%	8.3%	5.0%	2.4%	2.55
question	64.9%	40.1%	24.3%	15.1%	9.3%	5.8%	3.4%	2.63
rag	64.9%	39.4%	24.0%	13.6%	7.8%	4.3%	2.3%	2.56
summarization	58.5%	31.9%	17.3%	9.3%	5.2%	3.0%	1.7%	2.27
tool_call	68.8%	43.1%	26.5%	15.5%	9.1%	5.4%	3.1%	2.71
translation	58.4%	33.6%	18.9%	10.0%	5.5%	2.9%	1.5%	2.31
writing	66.2%	40.9%	25.3%	15.8%	10.2%	6.3%	3.8%	2.68

Latency Speedup

Speedup comparisons of DFlash speculative decoding vs. baseline (no speculation) at varying request rates on Nvidia H100:

References

Paper: DFlash: Block Diffusion for Flash Speculative Decoding

Library: Speculators

Library: speculators
License: apache-2.0
Created: Jun 12, 2026
Updated: Jul 25, 2026