SemCoT-mistral-1.1b-gsm8k

SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens

🚀 Overview

SemCoT is a framework that improves the efficiency of Chain-of-Thought (CoT) reasoning by encoding reasoning steps inside hidden representations (termed "implicit reasoning") rather than explicit tokens. This approach accelerates CoT by reducing the reasoning length and bypassing some LLM components while preserving semantic alignment with ground-truth reasoning.

This specific checkpoint is fine-tuned from optimum/mistral-1.1b-testing using the SemCoT framework on the GSM8K dataset.

• Paper: SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens
• Code: GitHub Repository

🎯 Key Features

• 🗣️ Semantic Alignment: Uses a contrastively trained sentence transformer to ensure that implicit reasoning remains semantically consistent with human-readable CoT explanations.
• ⚡ Efficiency Optimization: Introduces a lightweight implicit reasoning generator, fine-tuned via knowledge distillation, to reduce token generation time and enhance inference speed.
• 🧩 Joint Optimization: SemCoT simultaneously optimizes for reasoning speed and semantic alignment.

💻 Usage

To use this model, please clone the official repository and install the requirements. You can then run the evaluation on GSM8K using the following command:

python main.py \
  --mode semcot \
  --dataset gsm8k \
  --config mistral \
  --use_best_params \
  --num_exps 3 \
  --device 0

Citation

@inproceedings{he2025semcot,
  title={SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens},
  author={He, Yinhan and Zheng, Wendy and Zhu, Yaochen and Zheng, Zaiyi and Su, Lin and Vasudevan, Sriram and Guo, Qi and Hong, Liangjie and Li, Jundong},
  booktitle={39th Conference on Neural Information Processing Systems (NeurIPS 2025)},
  year={2025}
}