TASTE2-8B-EN-IF-TaskVector-Merged

TASTE2-8B-EN with Instruction-Following Task Vector

This repository contains slm.pt — the speech language model (SLM) backbone of TASTE2-8B-EN, merged with an instruction-following (IF) task vector derived from Qwen2-7B-Instruct.

Model Overview

TASTE2-8B-EN is an English speech language model that generates speech tokens auto-regressively, conditioned on text instructions. Its SLM backbone is a Qwen2-7B-based language model trained on English speech data.

This model merges IF capability into the TASTE-EN backbone without any SFT, using task vector arithmetic.

Merge Method

Let:

• $\theta_{\text{TASTE-EN}}$ — dense LLM weights extracted from the original TASTE2-8B-EN slm.pt
• $\theta_{\text{Qwen2-7B}}$ — original Qwen2-7B base weights
• $\theta_{\text{Instruct}}$ — Qwen2-7B-Instruct weights

Step 1: Compute TIES-sparsified IF task vector

$$\tau = \theta_{\text{Instruct}} - \theta_{\text{Qwen2-7B}}$$

$$\hat{\tau} = \text{TopK}_{30%}(\tau) \quad \text{(keep top 30% by magnitude, zero out the rest)}$$

$$\delta_{\text{IF}} = 0.5 \times \hat{\tau}$$

Step 2: Apply task vector to TASTE-EN backbone

$$\theta_{\text{merged}} = \theta_{\text{TASTE-EN}} + \delta_{\text{IF}}$$

Files

To run inference, place this slm.pt alongside the other TASTE2-8B-EN model files (flow.pt, hift.pt, llm.pt, speech_tokenizer_v2.onnx, campplus.onnx, etc.) and use the standard TASTE inference pipeline.

Citation

If you use this model, please cite the original TASTE paper:

@article{taste,
  title={TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling},
  author={Tseng, Liang-Hsuan and Chen, Yi-Chang and Lee, Kuan-Yi and Shiu, Da-Shan and Lee, Hung-yi},
  journal={arXiv preprint arXiv:2504.07053},
  year={2025},
  url={https://arxiv.org/abs/2504.07053}
}