seamless-translation

videoloc/seamless-translation

Model Description

This is a SeamlessTranslation model that processes audio and text inputs with translation awareness to predict Time To Edit (TTE) for subtitle segments. Given an audio segment and its corresponding subtitle text, the model predicts how much time (in seconds) would be required to edit/refine that subtitle segment, while taking into account whether the subtitle is a translation or original content.

The model extends the basic SeamlessM4T architecture with a translation feature that helps distinguish between original and translated subtitle content, improving TTE prediction accuracy across 5 languages: English, French, Spanish, Italian, and German with various translation pairs between them.

Key Features

• Translation-Aware Processing: Distinguishes between original and translated content
• Multimodal Processing: Simultaneously processes audio (16kHz) and text inputs
• Frozen Encoders: Uses pre-trained SeamlessM4T encoders (frozen for stability)
• Enhanced Architecture: Adds translation embedding to basic model
• TTE Prediction: Predicts editing time required for subtitle segments
• Direct Output: Raw time values in seconds for immediate use

Model Architecture

The model extends the basic SeamlessM4T architecture with translation awareness:

1. Audio Processing:

   • SeamlessM4T speech encoder (frozen) processes raw audio input
   • Audio projection layer maps speech encoder output to 1024 dimensions
   • Mean pooling over sequence length to get fixed-size audio embedding

2. Text Processing:

   • SeamlessM4T text encoder (frozen) processes tokenized text input
   • Text projection layer maps text encoder output to 1024 dimensions
   • Mean pooling over sequence length to get fixed-size text embedding

3. Translation Feature Processing:

   • Binary translation flag (0/1) indicating original vs translated content
   • Translation projection layer maps binary input to 64 dimensions
   • Learned embedding helps model distinguish translation effects

4. Feature Fusion:

   • Audio, text, and translation embeddings are concatenated (2112 total dimensions)
   • Simple concatenation without complex cross-modal interactions

5. Regression Head:

   • Multi-layer perceptron: 2112 → 1024 → 512 → 256 → 1
   • ReLU activations and dropout for regularization
   • Single output for TTE prediction (regression, in seconds)

Quick Start

Installation

pip install transformers torch torchaudio huggingface_hub

Basic Usage

from transformers import AutoModel, AutoConfig
from huggingface_hub import hf_hub_download
import torch
import numpy as np
import importlib.util

# Load model - custom architecture requires importing the model class
model_files = hf_hub_download(repo_id="videoloc/seamless-translation", filename="modeling_seamless_translation.py")
spec = importlib.util.spec_from_file_location("modeling_seamless_translation", model_files)
modeling_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(modeling_module)

# Now load the model using the custom class
config = modeling_module.SeamlessTranslationConfig.from_pretrained("videoloc/seamless-translation")
model = modeling_module.HFSeamlessTranslation.from_pretrained("videoloc/seamless-translation")

# Load the data collator (included in this repo)
collator_file = hf_hub_download(repo_id="videoloc/seamless-translation", filename="data_collator.py")
spec = importlib.util.spec_from_file_location("data_collator", collator_file)
collator_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(collator_module)

# Initialize data collator
data_collator = collator_module.DataCollatorSimpleSeamless(
    processor="facebook/hf-seamless-m4t-medium",
    max_audio_length_sec=8.0,
    max_text_length=256
)

# Prepare your data with translation information
your_data = [
    {
        'raw_audio': np.random.randn(16000 * 5),  # 5 seconds at 16kHz
        'raw_text': "Your subtitle text here",
        'is_translation': 1,  # 1 for translated content, 0 for original
    }
]

# Process and run inference
batch = data_collator(your_data)
model.eval()
with torch.no_grad():
    outputs = model(**batch)
    tte_prediction = outputs.logits.item()
    
print(f"Predicted Time To Edit (TTE): {tte_prediction:.2f} seconds")

Model Details

• Base Model: SeamlessM4T (facebook/hf-seamless-m4t-medium)
• Audio Encoder: Frozen SeamlessM4T speech encoder
• Text Encoder: Frozen SeamlessM4T text encoder
• Hidden Size: 1024
• Translation Embedding: 64 dimensions
• Audio Input: 16kHz
• Translation Input: Binary flag (0/1)
• Output: Single regression value (TTE in seconds)
• Task: Subtitle editing time prediction

Data Format

Your input data should be a list of dictionaries with:

• raw_audio: NumPy array of audio samples (16kHz sampling rate)
• raw_text: String of subtitle text
• is_translation: Binary flag (1 for translated, 0 for original content)
• labels: Target TTE values in seconds (optional, for training)

Example:

data = [
    {
        'raw_audio': audio_samples,  # shape: (num_samples,) at 16kHz
        'raw_text': "Subtitle text content",
        'is_translation': 1,  # 1 = translated, 0 = original
        'labels': 2.5  # optional TTE target value in seconds
    }
]

Performance Metrics

• Best Eval RMSE: 33.34

Training Details

• Base Model: facebook/hf-seamless-m4t-medium
• Model Type: seamless_with_translation
• Epochs: 10
• Batch Size (Train): 32
• Batch Size (Eval): 64
• Learning Rate: 1.2e-4
• LR Scheduler: cosine_with_restarts
• Warmup Ratio: 0.05
• Weight Decay: 0.001
• Optimizer: AdamW (torch)
• Max Grad Norm: 1.0
• FP16: True
• Early Stopping Patience: 5
• Audio Max Length: 8.0 seconds
• Text Max Length: 256 tokens
• Sample Rate: 16kHz
• Translation Feature: Binary flag (0/1)
• Normalization: None (raw values)
• Dataset Split: 80/20 train/test
• Random Seed: 42
• Metric: RMSE (lower is better)

Training Configuration

The model was trained with the following specifications:

• Dataset: Multimodal audio-subtitle pairs with translation annotations (5 languages: EN, FR, ES, IT, DE)
• Train/Test Split: 80/20 with random seed 42
• Audio Processing: 16kHz sampling, max 8.0 seconds, no offset
• Text Processing: Max 256 tokens
• Translation Feature: Binary flag indicating original vs translated content
• Normalization: None (raw TTE values in seconds)
• Caching: Audio segments cached and compressed for efficiency

Usage Notes

• This is the translation-aware variant - includes translation features
• For basic model without translation features, see seamless-basic
• For language pair embeddings, see seamless-langpairs
• Model expects 16kHz audio input (automatically resampled by data collator)
• Translation flag significantly impacts predictions
• No feature normalization applied - outputs raw TTE predictions in seconds
• Optimized for subtitle editing time estimation tasks

Limitations

• Requires translation annotation in training data
• Designed for TTE prediction, not general audio-text matching
• Performance may vary on out-of-domain content
• Requires specific data preprocessing (use included data collator)

Related Models

• seamless-basic: Basic audio+text model without translation features
• seamless-langpairs: Includes language pair embeddings for fine-grained multilingual control
• seamless-crossattention: Advanced cross-modal attention mechanisms for sophisticated audio-text interactions