tempo-rl-multitask

TEMPO

Temporally-grounded Multi-task Post-training for Large Audio-Language Models

Paper · All artifacts · Dataset

Description

The main TEMPO model. GRPO with verifiable rewards over all five tasks jointly, 1,000 steps from the Stage 1+2 SFT checkpoint. The headline TEMPO model.

TEMPO is the first unified large audio-language model for timestamping across speech, sound and music. A single decoder handles all five tasks, emitting text interleaved with timestamp tokens and distinguished only by a task tag in the prompt. Three components make it work: atomic timestamp tokens (~601 dedicated tokens at 0.1 s resolution, so a timestamp is one categorical decision rather than several BPE fragments), a time-aware multi-modal projector injecting sinusoidal wall-clock encodings into frame embeddings, and a distance-aware Gaussian loss that gives partial credit to near misses instead of penalising every wrong timestamp equally.

Best for

• Assigning who spoke what, and when in meetings and conversational audio
• Localising a natural-language query to a time interval in an audio stream
• Dense captioning — segmenting a recording and describing each event with boundaries
• Timestamped music analysis — chord spans, tempo changes, instrument entries/exits

This checkpoint

Table 2 — SFT + Multi-task RL — reported results:

ASR MAE 0.76 / mIoU 65.8 / WER 43.5 · DER 25.4 · dense-cap eF1 59.3 · grounding F1 46.5

This variant uses the time-aware projector, so time_proj.pt is included and must be loaded.

TEMPO variants

Plus the training and evaluation data: Kaousheik/tempo — five task configs with splits synthetic_stage1 / sft_stage2 / rl / val / evaluation.

Usage

import torch
from transformers import AutoProcessor, AutoModelForCausalLM

repo = "Kaousheik/tempo-rl-multitask"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

# REQUIRED: the time-aware projector lives outside the transformer checkpoint.
# Skipping this silently falls back to the stock AF3 projector and degrades timestamps.
from huggingface_hub import hf_hub_download
sd = torch.load(hf_hub_download(repo, "time_proj.pt"), map_location="cpu")
model.load_time_projector(sd)   # adapt to your loading code

Prompt with the task tag for the behaviour you want:

Timestamps are emitted as atomic tokens <|0.0|> … <|60.0|> at 0.1 s resolution.

Evaluation

Held-out benchmark of 10,521 question–answer pairs over six corpora. WER and DER are percentages (lower is better); mIoU is the symmetric harmonic mean of recall- and precision-side mean-best-IoU, so over-prediction cannot inflate it.

Training summary

Built on Audio Flamingo 3 — frozen Whisper-large audio encoder (~0.6B) + Qwen2-7B, joined by a two-layer MLP projector. Three stages:

Rewards for GRPO are the evaluation metrics themselves, bounded in [0, 1] and gated by a format check, so maximising reward and improving evaluation performance are aligned. Trained on NVIDIA A6000s; the audio encoder stays frozen throughout.

Limitations

• English only; timestamps are capped at 60 s by the token vocabulary, so longer audio must be chunked.
• Multi-speaker ASR remains hard in absolute terms (WER 43.5%) — useful for alignment, not a drop-in replacement for a dedicated ASR system.
• Chord F1 is low in absolute terms; music captioning is best treated as approximate.
• Speaker labels are consistent within a clip but arbitrary across clips.
• CHiME-6 is excluded from the released dataset (its licence forbids redistribution), so the public data is smaller than what the paper trained on.

Licence

Derived from Audio Flamingo 3, released by NVIDIA under a research-only licence, which governs this checkpoint. Non-commercial academic research only. Training corpora carry their own terms: CC BY 4.0 for AMI, ICSI, AudioSet Strong, TACOS, Slakh2100 and LibriSpeech; CC BY-NC 3.0 for ESC-50.

Citation

@inproceedings{tempo,
  title     = {TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models},
  author    = {TODO},
  booktitle = {TODO},
  year      = {2026}
}