MiniMax-M3-MXFP4-DSpark

DSpark Speculator for MiniMax-M3-MXFP4

A DSpark speculative-decoding draft model for olka-fi/MiniMax-M3-MXFP4, trained with vLLM Speculators.

Results

Acceptance length (mean tokens accepted per verifier step) measured in vLLM at 4 speculative tokens, over the RedHatAI/speculator_benchmarks subsets:

The head is strongest on code, math, and agentic/tool-use traffic (its training emphasis), translating to roughly a 2.3–3.4× decode speedup depending on domain. Open-ended chat/writing is weaker (~2.0), as high-entropy natural language is intrinsically harder to draft.

Usage (vLLM)

vllm serve olka-fi/MiniMax-M3-MXFP4 \
  --trust-remote-code --block-size 128 \
  --speculative-config '{"model": "olka-fi/MiniMax-M3-MXFP4-DSpark", "num_speculative_tokens": 4}'

Serving MiniMax-M3-MXFP4 also requires the checkpoint's shipped vllm_patch/ (SwiGLU-OAI clamp fix; force the Marlin MXFP4 MoE path with VLLM_MXFP4_USE_MARLIN=1). The draft was trained with block_size=8; serving with num_speculative_tokens=4 targets the high-acceptance early positions (per-position accuracy decays across the block).

Training

• Algorithm: DSpark (block size 8, 3 draft layers, Markov head rank 256 vanilla, confidence head enabled, sliding-window attention window 8192 on all draft layers).
• Auxiliary layers: verifier layers 2, 30, 57 (of 60) + final hidden state.
• Draft vocab: 32,000 (reduced from 200,064; top-32k tokens cover 97.6% of training tokens).
• Data (~24k samples): a mix chosen for general-assistant + agentic + code traffic —
  • on-policy open-perfectblend regenerated by MiniMax-M3 itself (temp 1.0, top_p 0.95),
  • off-policy open-perfectblend, source-filtered to instruction/reasoning/code (ultrachat, ultrafeedback, AutoIF, evol-codealpaca, UltraInteract, lmsys; math dropped),
  • hermes-function-calling-v1 (agentic tool-use),
  • Agentic-Coding-Tessa (opencode-style coding agent, on- and off-policy).
• Recipe: offline training on pre-extracted hidden states, 9 epochs, Muon optimizer, cosine LR (peak 3e-4), loss {"ce": 0.1, "tv": 0.9} + confidence-head BCE.

Limitations

• Trained mix is weighted toward code / agentic / instruction; open-ended chat and creative writing accept fewer tokens (~2.0). Adding diverse chat data would lift these.
• Acceptance is lossless (output distribution is unchanged); only latency is affected.