ps-slm

TASU: Text-only Alignment for Speech Understanding

TASU (Text-only Alignment for Speech Understanding) is a newly proposed training paradigm for Speech Large Language Models (Speech LLMs), with a primary focus on semantic speech understanding.

Overview

TASU enables robust cross-modal alignment between speech and text without relying on paired audio-text data. By leveraging only unpaired text data, TASU achieves competitive zero-shot speech recognition and strong generalization across diverse speech understanding tasks.

Key Innovations

• Label-Synchronous Decoding (LSD): Compresses CTC posteriors by 6× while preserving semantic information
• CTC Posterior Simulation (CPS): Generates realistic pseudo-posteriors from text-only data
• Zero-shot Transfer: Train on text, infer on speech seamlessly

Model Architecture

This checkpoint integrates:

• Speech Encoder: SenseVoice-Small (for CTC posterior extraction)
• LLM Backbone: Qwen2.5-1.5B (frozen during alignment)
• Projector: Linear–SiLU–Linear trainable adapter