L-MTP

GitHub

<div align=center> <h1><a color="red" href="https://arxiv.org/pdf/2505.17505">L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models</a></h1>

</div>

<h4 align="center">An extendable MTP paradigm with <i>leaping</i> capability for both training and inference.</h4>

Large Language Models (LLMs) are typically trained and deployed using Next-Token Prediction (NTP), an inherently sequential process that limits contextual coverage and inference efficiency. To address this, we propose Leap Multi-Token Prediction (L-MTP), an innovative method that extends Multi-Token Prediction (MTP) by strategically skipping intermediate tokens to predict non-adjacent future tokens in a single forward pass. L-MTP enhances the model's ability to capture long-range dependencies and enables a specialized decoding strategy that significantly accelerates inference.

Models & Datasets

The paper evaluates L-MTP on a diverse set of base LLMs: Qwen 2.5 (3B, 7B), Llama 3.2 (3B), Llama 3.1 (8B), and Gemma 3 (4B, 12B).

Training datasets are curated from: Math (Hendrycks et al.), Evol-Instruct-Code (Luo et al., Chaudhary) and Alpaca-GPT4 (Peng et al.). You can go through our dataset in LLaMA-Factory/data:dataset_info.json and link.

---

Citation

If you find this work useful, please cite our paper:

@article{lmtp,
  title={L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models},
  author={Liu, Xiaohao and Xia, Xiaobo and Zhao, Weixiang and Zhang, Manyi and Yu, Xianzhi and Su, Xiu and Yang, Shuo and Ng, See-Kiong and Chua, Tat-Seng},
  journal={arXiv preprint arXiv:2505.17505},
  year={2025}
}

<div align=center> <h3>🌟 Empower your LLMs to predict farther and generate faster with L-MTP! 🌟</h3> </div>