MARS-Qwen2.5-7B-AR-SFT

MARS-Qwen2.5-7B-AR-SFT

Qwen2.5-7B-Instruct after stage 1 of the MARS recipe (MARS: Enabling Autoregressive Models Multi-Token Generation; Jin, Wang, Luo, Sun, 2026): 5 epochs of ordinary autoregressive (AR) supervised fine-tuning on allenai/Dolci-Instruct-SFT (about 2M rows; learning rate 5e-6, warmup ratio 0.03, effective batch 384, sequence length 512). A plain Qwen2ForCausalLM; nothing about its architecture or decoding is changed.

It is the matched AR control for Xalphinions/MARS-Qwen2.5-7B-blk16, which continues from this checkpoint with the MARS block-prediction objective on the same data, so that the effect of block training can be separated from the effect of seeing the data. In the ProVe: Proposer-Verifier Decoding repository (Jin, Xu, Li, Zhong, She, Sun, 2026) it is the AR baseline and the STAGE2_INIT for training a proposer.

• ProVe project page: https://xalp.net/prove/
• ProVe code (serving stack, benchmarks, training): https://github.com/Xalp/ProVe
• MARS code and training recipe: https://github.com/Xalp/MARS · paper: https://arxiv.org/abs/2604.07023
• Stage-2 checkpoint (the block-16 proposer): Xalphinions/MARS-Qwen2.5-7B-blk16

Model details

How to use

An ordinary causal LM:

from transformers import AutoModelForCausalLM, AutoTokenizer
path = "Xalphinions/MARS-Qwen2.5-7B-AR-SFT"
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype="bfloat16", device_map="auto")
tok = AutoTokenizer.from_pretrained(path)

To train a block proposer from it, follow train/README.md in https://github.com/Xalp/ProVe (stage 2 of train/train_proposer.sh with STAGE2_INIT pointing at this checkpoint) or the MARS repository.

Intended use and limitations

• The AR reference point for MARS / ProVe experiments, and a starting point for block training.
• Fine-tuned on English instruction data; behaviour on other languages follows the base model but was not evaluated.
• Safety behaviour is inherited from Qwen2.5-7B-Instruct and the public instruction data; no additional alignment or red-teaming was done. Apply the same safeguards you would apply to the base model.

Citation

@misc{jin2026mars,
  title         = {MARS: Enabling Autoregressive Models Multi-Token Generation},
  author        = {Ziqi Jin and Lei Wang and Ziwei Luo and Aixin Sun},
  year          = {2026},
  eprint        = {2604.07023},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2604.07023}
}

@article{jin2026prove,
  title   = {ProVe: Proposer-Verifier Decoding},
  author  = {Jin, Ziqi and Xu, Weiwen and Li, Jiaxi and Zhong, Yujie and She, Qi and Sun, Aixin},
  journal = {arXiv preprint},
  year    = {2026}
}