qwen1.7b-dart

<div align="center"> <img src="logo.png" alt="DART" width="220">

</div> <h3 align="center"> DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference </h3> <p align="center"> <a href="#">[Paper will come soon]</a> </p>

---

Overview

DART is a new speculative decoding approach for Large Language Models (LLMs) inference, which is inspired by diffusion-based Large Language Models (dLLMs). DART surpasses EAGLE3 by 30% on average, achieving up to 65% improvement on certain code-centric workloads.

DART's drafting requires only a single forward pass of a single transformer layer and a fast cpp-based tree search to build the draft token tree, resulting in extremely low drafting cost while preserving relatively high $\tau$ (Average Acceptance Length).

Key Features

• Fast drafting Forward: Produces multiple logits simultaneously with 1 forward of 1 layer.
• Fast Tree Search: Uses n-gram–based tree search to build the final draft tree (C++ based).
• Low Drafting Cost: Results in extremely low drafting cost for efficient inference.
• relatively high $\tau$: Average Acceptance Length is competitive with EAGLE3.

Quick Start

Installation

git clone https://github.com/fvliang/DART.git
cd DART
curl -LsSf https://astral.sh/uv/install.sh | sh (optimal, if you don't have a uv)
uv sync
uv pip install -e .

Model Weights (HuggingFace)

<div align="center"> <table> <thead> <tr> <th>Base Model</th> <th>DART Weights</th> <th>N-gram Model</th> </tr> </thead> <tbody> <tr> <td><a href="https://huggingface.co/Qwen/Qwen3-1.7B">Qwen3-1.7B</a></td> <td><a href="https://huggingface.co/fvliang/qwen1.7b-dart">fvliang/qwen1.7b-dart</a></td> <td><a href="https://huggingface.co/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> <tr> <td><a href="https://huggingface.co/Qwen/Qwen3-4B">Qwen3-4B</a></td> <td><a href="https://huggingface.co/fvliang/qwen4b-dart">fvliang/qwen4b-dart</a></td> <td><a href="https://huggingface.co/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> <tr> <td><a href="https://huggingface.co/Qwen/Qwen3-8B">Qwen3-8B</a></td> <td><a href="https://huggingface.co/fvliang/qwen8b-dart">fvliang/qwen8b-dart</a></td> <td><a href="https://huggingface.co/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> <tr> <td><a href="https://huggingface.co/Qwen/Qwen3-14B">Qwen3-14B</a></td> <td><a href="https://huggingface.co/fvliang/qwen14b-dart">fvliang/qwen14b-dart</a></td> <td><a href="https://huggingface.co/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> <tr> <td><a href="https://huggingface.co/Qwen/Qwen3-32B">Qwen3-32B</a></td> <td><a href="https://huggingface.co/fvliang/qwen32b-dart">fvliang/qwen32b-dart</a></td> <td><a href="https://huggingface.co/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> </tbody> </table> </div>

Model Weights (ModelScope)

<div align="center"> <table> <thead> <tr> <th>Base Model</th> <th>DART Weights</th> <th>N-gram Model</th> </tr> </thead> <tbody> <tr> <td><a href="https://modelscope.cn/models/Qwen/Qwen3-1.7B">Qwen3-1.7B</a></td> <td><a href="https://modelscope.cn/models/fvliang/Qwen3-1.7B-dart">fvliang/Qwen3-1.7B-dart</a></td> <td><a href="https://modelscope.cn/models/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> <tr> <td><a href="https://modelscope.cn/models/Qwen/Qwen3-4B">Qwen3-4B</a></td> <td><a href="https://modelscope.cn/models/fvliang/Qwen3-4B-dart">fvliang/Qwen3-4B-dart</a></td> <td><a href="https://modelscope.cn/models/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> <tr> <td><a href="https://modelscope.cn/models/Qwen/Qwen3-8B">Qwen3-8B</a></td> <td><a href="https://modelscope.cn/models/fvliang/Qwen3-8B-dart">fvliang/Qwen3-8B-dart</a></td> <td><a href="https://modelscope.cn/models/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> <tr> <td><a href="https://modelscope.cn/models/Qwen/Qwen3-14B">Qwen3-14B</a></td> <td><a href="https://modelscope.cn/models/fvliang/Qwen3-14B-dart">fvliang/Qwen3-14B-dart</a></td> <td><a href="https://modelscope.cn/models/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> <tr> <td><a href="https://modelscope.cn/models/Qwen/Qwen3-32B">Qwen3-32B</a></td> <td><a href="https://modelscope.cn/models/fvliang/Qwen3-32B-dart">fvliang/Qwen3-32B-dart</a></td> <td><a href="https://modelscope.cn/models/fvliang/dart-qwen3-ngram">fvliang/dart-qwen3-ngram</a></td> </tr> </tbody> </table> </div>