Temporal-Coupled-Flow-Matching-for-Speech-Continuation

Temporal-Coupled Flow Matching for Speech Continuation

This repository contains the three formally evaluated Temporal-Coupled Flow Matching (TCFM) FlowHead adapters for speech continuation. TCFM preserves the Flow-SLM sampler, Mimi codec, Transformer, and semantic pathway. It fine-tunes only 488,448 LoRA parameters in the FlowHead by adding speech-time endpoint-trajectory supervision to conditional flow matching.

Code and complete reproduction instructions: https://github.com/JoyBoyuuu/TCFM

Required base model

These files are adapters, not standalone 1.3B-parameter models. Inference requires the Flow-SLM 1B-extended base checkpoint distributed by the Flow-SLM authors. Download it with:

git clone https://github.com/JoyBoyuuu/TCFM.git
cd TCFM
conda env create -f environment.yml
conda activate tcfm
python scripts/download_assets.py --checkpoint-dir checkpoints \
  --cache-dir cache/huggingface

Released checkpoints

The three training runs all completed 2,000 optimizer steps. The released checkpoint for each seed is the one with the lowest validation objective. The files are path-free, inference-only exports: optimizer state, random number generator state, absolute cluster paths, and duplicated LoRA state are excluded. manifest.json records SHA-256 hashes and adapter settings.

Training data and objective

• Base model: Flow-SLM 1B-extended.
• Adaptation data: 4,096 eligible utterances from LibriSpeech train-clean-100.
• Validation data: 256 eligible utterances from LibriSpeech dev-clean.
• Each example uses a 3-second prefix and a 5-second future segment.
• LoRA rank 4, alpha 8, dropout 0.05.
• Batch size 8, learning rate 5e-5, weight decay 1e-4.
• TCFM transition weight 0.1 and prefix-boundary weight 0.1.
• Curvature weight 0.5, direction weight 0.1, boundary window 16 frames.

TCFM reconstructs the clean endpoint implied by the predicted FlowHead velocity and regularizes endpoint displacement, curvature, movement direction, and prefix-continuation boundary behavior along speech time. See the code repository for the complete mathematical objective.

Inference

Download one adapter:

from huggingface_hub import hf_hub_download

adapter = hf_hub_download(
    repo_id="JoyBoyuuu/Temporal-Coupled-Flow-Matching-for-Speech-Continuation",
    filename="tcfm_adapter_seed_42.pt",
)
print(adapter)

Then run a smoke evaluation on a GPU/Slurm compute node:

export AUDIO_ROOT=/path/to/LibriSpeech

sbatch --account=<account> --partition=<gpu-partition> \
  --export=ALL,PROJECT_DIR="$PWD",AUDIO_ROOT="$AUDIO_ROOT",\
METHOD=tcfm_adapter,TRAINING_SEED=42,LIMIT=10,\
ADAPTATION_CKPT=/path/to/tcfm_adapter_seed_42.pt,PROTOCOL=open_ended \
  scripts/slurm/infer_1gpu.sbatch

The reported decoder uses 32 Euler flow steps, 16 Mimi RVQ stages, acoustic and semantic temperatures 0.8, top-p 0.95, and CFG scale 0.3.

Evaluation summary

The formal campaign evaluates all three training seeds on LibriSpeech test-clean and test-other with three matched inference seeds. Selected adapter-level results relative to Base Flow-SLM are:

• open-ended speaker similarity: 0.909415 -> 0.912781, paired effect +0.003367 with 95% CI [+0.002298, +0.004455];
• open-ended natural-calibrated 200-ms boundary log-mel distance: 8.5462 -> 8.4945, effect -0.05175 with 95% CI [-0.07862, -0.02511];
• open-ended WavLM FSD: 3.653 -> 2.919, about 20% lower;
• oracle-semantic Mimi q0 frame agreement: 0.65553 -> 0.66977, effect +0.01424 with 95% CI [+0.01265, +0.01576];
• oracle-semantic WavLM FSD: 0.590 -> 0.533, about 9.8% lower.

These results support improved continuation stability, speaker persistence, and distributional acoustic match. They do not establish universal boundary improvement, human preference, or broad lexical/syntactic improvement. Only three training seeds were evaluated.

Intended use and limitations

The adapters are research artifacts for English speech continuation and for reproducing the TCFM experiments. They should not be treated as a speech recognizer, a semantic dialogue model, or a production voice-cloning system. Generated speech may contain incorrect, unstable, or biased content. Obtain consent before processing or imitating a person's voice.

Attribution and licensing

TCFM is derived from Flow-SLM. The upstream Flow-SLM revision used for this work did not include a license file, so this release does not assert a new license over inherited architecture or base-model assets. The Flow-SLM base checkpoint and third-party datasets are not redistributed here and remain subject to their original terms. See the repository NOTICE.md for details.