Quatfit-Mini-MTP

<p align="center"> <img src="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/banner.png" alt="Quatfit Mini MTP Banner" width="100%"> </p> <h1 align="center">Quatfit Mini — MTP</h1> <h3 align="center">Standalone Multi-Token-Prediction Drafter for Speculative Decoding</h3> <p align="center"> <a href="https://huggingface.co/Quatfit/Quatfit-Mini-MTP"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face-Model%20Hub-FFD21E?style=flat-square" alt="Hugging Face"></a> <a href="https://huggingface.co/Quatfit/Quatfit-Mini"><img src="https://img.shields.io/badge/🧠%20Target%20Model-Quatfit%20Mini-4052F5?style=flat-square" alt="Target Model"></a> <a href="https://huggingface.co/Quatfit/Quatfit-Mini-FP8"><img src="https://img.shields.io/badge/⚡%20Serving-FP8%20%7C%20GGUF-00C853?style=flat-square" alt="Serving"></a> <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/📜%20License-Apache%202.0-E53935?style=flat-square" alt="License"></a> </p>

---

Overview

Quatfit Mini MTP is the standalone multi-token-prediction (MTP) drafter used for speculative decoding with Quatfit Mini. Rather than a separate small language model, it is a lightweight autoregressive head that cross-attends directly to the target model's own KV cache — it has its own token embedder and a compact 4-layer transformer block (3 local sliding-window layers + 1 global layer), but no separate prefill pass and no independent context of its own to maintain.

This repository packages that drafter as a standalone download so it can be attached to any Quatfit Mini serving deployment (FP32, BF16, FP8, or GGUF) to accelerate decoding.

This is not a general-purpose chat model — it drafts candidate continuations that the target model verifies. Use it together with Quatfit/Quatfit-Mini, Quatfit/Quatfit-Mini-FP8, or Quatfit/Quatfit-Mini-GGUF.

---

Headline Results

See Methodology below for exactly what these numbers measure and the conditions they were measured under — read that section before quoting these figures in your own comparisons.

---

Why It's Faster and More Accurate Than a Standalone Draft Model

Most speculative-decoding setups pair a large target model with a small, independently-trained draft model that has to run its own forward pass and maintain its own context. Quatfit Mini MTP takes a different approach:

1. Shared KV cache, no separate prefill. The drafter cross-attends to the target model's existing KV cache instead of recomputing its own — it never needs to catch up on context the target model has already processed. This removes the drafter-prefill overhead that standalone draft models pay on every request.
2. Trained jointly with the target model's representations. Because the drafter consumes the target model's own last-layer activations and embeddings rather than approximating them from scratch, its predictions track the target model's actual output distribution more closely than an independently trained model of similar size — this is the main driver of the higher acceptance rate.
3. Efficient decoding via top-k clustering. Instead of projecting over the full 262,144-token vocabulary at every drafted step, the drafter's output head uses a top-k clustering operation, shrinking the final matrix multiplication from $d \times 262{,}144$ to $d \times 4{,}096$ — this keeps the drafter's own per-token cost low enough that its overhead doesn't eat into the speedup from higher acceptance.
4. Arbitrary draft length. Because there's no separate prefill to amortize, the drafter can propose a variable number of tokens per step rather than being locked to a fixed draft length, letting the serving stack tune the depth/acceptance trade-off per workload.

---

Architecture

---

Usage

vLLM (speculative decoding)

vllm serve Quatfit/Quatfit-Mini \
  --speculative-model Quatfit/Quatfit-Mini-MTP \
  --num-speculative-tokens 5 \
  --max-model-len 131072

from vllm import LLM, SamplingParams

llm = LLM(
    model="Quatfit/Quatfit-Mini",
    speculative_model="Quatfit/Quatfit-Mini-MTP",
    num_speculative_tokens=5,
    max_model_len=131072,
)
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

outputs = llm.generate(["Explain the difference between GQA and MHA."], sampling_params)
print(outputs[0].outputs[0].text)

With the FP8-served target model

vllm serve Quatfit/Quatfit-Mini-FP8 \
  --quantization fp8 \
  --speculative-model Quatfit/Quatfit-Mini-MTP \
  --num-speculative-tokens 5

llama.cpp

./llama-server \
  -hf Quatfit/Quatfit-Mini-GGUF:Q4_K_M \
  --model-draft Quatfit-Mini-MTP-Q4_K_M.gguf \
  --draft-max 5 \
  -c 131072

🤗 Transformers (assisted generation)

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

target = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini", torch_dtype="auto", device_map="auto"
)
drafter = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini-MTP", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Quatfit/Quatfit-Mini")

messages = [{"role": "user", "content": "Write a Python implementation of binary search."}]
inputs = processor.apply_chat_template(messages, tokenize=True, return_tensors="pt").to(target.device)

outputs = target.generate(**inputs, assistant_model=drafter, max_new_tokens=512)
print(processor.decode(outputs[0]))

---

Methodology

The headline numbers above were measured as follows — please validate against your own workload before relying on them for capacity planning.

Decode throughput (2.4×): measured as end-to-end tokens/second during decode (post-prefill), comparing standard single-token autoregressive decoding against speculative decoding with this drafter, both targeting Quatfit Mini at the same precision and batch size, on a single H100. Throughput gain varies with batch size, sequence length, and task mix — larger batches and less-predictable generation (e.g. open-ended creative writing vs. structured code) typically see a smaller multiple than the figure quoted.

Draft acceptance rate (+97% relative): measured as the fraction of drafted tokens accepted by the target model's verification step, averaged across a mixed evaluation set of code, chat, reasoning, and tool-use prompts, at draft length 5. The "baseline" is an independently trained standalone draft model of comparable parameter count (~180M) with its own embedder, own KV cache, and no cross-attention into the target model — i.e., the conventional speculative-decoding setup this drafter is designed to improve on, not a specific named third-party product.

---

Relationship to Other Quatfit Mini Repositories

Full architecture details for both the target model and this drafter are documented in the Quatfit Mini Technical Report.

---

Responsible AI

This drafter does not generate final output on its own — all drafted tokens are verified by the target model before being returned, so speculative decoding does not change what the target model would otherwise produce; it only changes how fast those tokens are produced. Standard responsible-use guidance for Quatfit Mini itself still applies:

• Verify critical information
• Apply RAG for factual grounding
• Use application-level safety filters
• Keep human oversight for high-risk domains

---

Citation

@article{quatfitminimtp2026,
    title={Quatfit Mini MTP: A Cross-Attending Multi-Token-Prediction Drafter for Speculative Decoding},
    author={Quatfit AI Research},
    year={2026}
}

---

License

Apache License 2.0. See LICENSE for details.

---

<p align="center"> <a href="https://huggingface.co/Quatfit/Quatfit-Mini-MTP">🤗 MTP Repository</a> • <a href="https://huggingface.co/Quatfit/Quatfit-Mini">🧠 Base Model (FP32)</a> • <a href="https://huggingface.co/Quatfit/Quatfit-Mini-FP8">⚡ FP8 Build</a> • <a href="https://huggingface.co/Quatfit/Quatfit-Mini-GGUF">⚙️ GGUF Builds</a> • <a href="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/Quatfit-Mini_Technical_Report.pdf">📄 Technical Report</a> </p>