whisper-twophase-multitask-80K-10WER-ct2

Whisper Bilingual German-English Multitask Model

CT2 INT8-FP16 | OpenAI Whisper-large-v3-turbo | Bidirectional De-En Speech Translation & Transcription

Developed at the Hasso Plattner Institute (HPI) with computational resources provided by the HPI High-Performance Computing (HPC) cluster. Fine-tuned from openai/whisper-large-v3-turbo on the aman4014/translated-german-english-asr dataset — a curated parallel German-English speech corpus totalling approximately 4.65 million samples across 12 training splits — using a two-phase multitask schedule (80,000 steps). Optimized with CTranslate2 INT8-FP16 quantization for efficient, real-time bidirectional speech translation and transcription between German and English.

• CTranslate2 Model (Quantized): aman4014/whisper-twophase-multitask-80K-10WER-ct2
• Transformers Model (FP16/PyTorch): aman4014/whisper-twophase-multitask-80K-10WER

---

Table of Contents

1. Model Overview
2. Key Features
3. Model Architecture
4. Training Strategy & Data
5. Performance Metrics
6. Usage
7. Supported Directions
8. Technical Specifications
9. Use Cases
10. Limitations
11. Citation
12. Acknowledgements
13. Funding Notice
14. License

---

Model Overview

This model is a CTranslate2-optimized fine-tune of OpenAI's Whisper Large V3 Turbo, fine-tuned for bidirectional German-English speech translation and bilingual transcription. Quantized using INT8 weights and FP16 activations, it achieves an optimal balance between translation quality, low latency, reduced memory footprint, and bidirectional capability (De $\leftrightarrow$ En).

The model extends Whisper's standard task vocabulary with a custom <|en_to_de|> task token, enabling English-to-German speech translation directly within Whisper's autoregressive decoder — a capability absent in vanilla Whisper models.

---

Key Features

• Bidirectional Translation: Supports both German-to-English (task="translate") and English-to-German (task="en_to_de").
• Accurate German Transcription: Delivers 10.32% WER and 5.23% CER on German speech.
• Turbo Architecture Efficiency: Built on the 4-layer decoder design of Whisper Large V3 Turbo, delivering 3.37x lower latency than vanilla Whisper Large V3.
• Real-Time Performance: Mean RTF of 0.030 on CUDA — 47.9x faster than real-time across 50 hours of audio.
• Best Quality-Efficiency Ratio: COMET/RTF of 20.3, ranking first among 12 benchmarked systems on LibriS2S.
• Efficient Footprint: Quantized size of ~814 MB (vs. ~3.24 GB in FP32); ~1.5 GB memory footprint during inference.
• Production-Ready: INT8-FP16 quantization optimized for deployment via faster-whisper and CTranslate2.
• Zero Hard Errors: Processes 100% of test samples with zero failures across 24,375 benchmark samples.
• Custom Task Token: Novel <|en_to_de|> task token (ID: 51866) initialized from a blend of <|translate|> and <|de|> embeddings.

---

Model Architecture

Custom Task Tokens

This model extends Whisper's vocabulary with:

---

Training Strategy & Data

Two-Phase Multitask Fine-Tuning

The model was trained on an NVIDIA H100 80GB HBM3 GPU (effective batch size: 32) using a two-phase schedule:

1. Phase 1: Decoder-Only Training (15,000 steps, LR $3 \times 10^{-4}$, Warmup 500):
   • The encoder is frozen to preserve the base model's acoustic representations.
   • Only the decoder is trained across all 4 tasks to adapt to the new <|en_to_de|> token and output text distributions.
2. Phase 2: End-to-End Joint Fine-Tuning (80,000 steps, LR $1 \times 10^{-5}$, Warmup 1000):
   • All parameters (encoder + decoder) are unfrozen for joint optimization.
   • Best checkpoint: checkpoint-74000 (evaluated at 80,000 steps).

Task Distribution

• German $\to$ English Translation: 25%
• English $\to$ German Translation (<|en_to_de|>): 35%
• German Transcription (ASR): 20%
• English Transcription (ASR): 20%

Training Corpus

Fine-tuned on the aman4014/translated-german-english-asr dataset, aggregating parallel German-English speech data:

---

Performance Metrics

1. In-Domain & Parallel Speech Test Evaluation

From the training evaluation (checkpoint-74000 at 80,000 steps):

---

2. LibriS2S Benchmark (Cross-Domain Test Set, German to English, 24,375 samples)

Benchmarked against 11 baseline systems spanning quality ceilings, size-matched peers, and cascade architectures:

Quality/Latency = COMET / RTF_mean. Higher is better. Measures semantic translation quality delivered per unit of compute time.

Latency Characteristics

---

Usage

1. Inference with faster-whisper (Recommended)

pip install faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel(
    "aman4014/whisper-twophase-multitask-80K-10WER-ct2",
    device="cuda",               # or "cpu"
    compute_type="int8_float16"  # use "int8" for CPU
)

# 1. German to English Speech Translation
segments, info = model.transcribe(
    "german_audio.wav",
    language="de",
    task="translate",
    beam_size=5
)
for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

# 2. English to German Speech Translation (Custom Task Token)
segments, info = model.transcribe(
    "english_audio.wav",
    language="de",
    task="en_to_de",
    beam_size=5
)
for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

# 3. German Speech-to-Text Transcription (ASR)
segments, info = model.transcribe(
    "german_audio.wav",
    language="de",
    task="transcribe",
    beam_size=5
)
for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

---

2. Inference with Hugging Face transformers

pip install transformers torch accelerate

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "aman4014/whisper-twophase-multitask-80K-10WER"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    device_map="cuda"
)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    torch_dtype=torch.float16,
    device="cuda",
)

# German transcription
result = pipe("german_audio.wav", generate_kwargs={"task": "transcribe", "language": "de"})
print(result["text"])

---

Supported Directions

---

Technical Specifications

Quantization Details

This model uses INT8-FP16 mixed precision quantization via CTranslate2:

• Weights: INT8 (8-bit integers) — 4x memory reduction
• Activations: FP16 (16-bit floats) — Preserves numerical stability
• Quantization Method: Symmetric per-tensor quantization
• Accuracy: ~98% of FP32 performance (minimal degradation)

Environment Variables

# Model configuration
export WHISPER_MODEL_REPO_ID="aman4014/whisper-twophase-multitask-80K-10WER-ct2"
export WHISPER_MODEL_REVISION="main"
export WHISPER_EN_DE_TASK_TOKEN="<|en_to_de|>"

# Decoding parameters
export WHISPER_FINAL_BEAM_SIZE=6
export WHISPER_FINAL_MAX_NEW_TOKENS=224
export WHISPER_DECODE_PATIENCE=1.15
export WHISPER_DECODE_LENGTH_PENALTY=1.0
export WHISPER_DECODE_REPETITION_PENALTY=1.08
export WHISPER_DECODE_NO_REPEAT_NGRAM_SIZE=3

# Audio preprocessing
export WHISPER_SILENCE_TRIM_ENABLED=1
export WHISPER_SILERO_VAD_ENABLED=1
export WHISPER_SILERO_VAD_THRESHOLD=0.44

---

Use Cases

• Live Conference Translation: German-English bidirectional interpretation with low latency.
• Podcast and Video Subtitling: High-throughput batch subtitle generation with near-zero failure rate.
• Bilingual Conversational AI: Real-time customer support for German and English speakers.
• Medical & Legal Documentation: Transcribe and translate clinical or official recordings.

---

Limitations

1. Output length compression on long segments: In zero-shot cross-domain tests with long audio (e.g. LibriS2S), long sentences without segmentation may exhibit length compression. Segmenting audio with VAD into chunks under 30s is strongly recommended.
2. Domain mismatch: Extreme accents or domain-specific jargon outside the training corpus may require domain-adapted prompts.
3. Short-clip overhead: For audio under 3 seconds, a fixed startup overhead (~0.114 s) dominates processing time; batching short clips is recommended.

---

Acknowledgements

• Hasso Plattner Institute (HPI): Research environment, infrastructure, and academic support.
• HPI High-Performance Computing (HPC) Cluster: Computational resources for model fine-tuning and evaluation on NVIDIA H100 GPUs.
• AI Service Center Berlin-Brandenburg (KI-Servicezentrum): Supporting research and deployment of speech and language technologies.

---

Funding Notice

German (Original)

Das zugrunde liegende Vorhaben wurde mit Mitteln des Bundesministeriums für Bildung und Forschung (BMBF) unter dem Förderkennzeichen "KI-Servicezentrum Berlin-Brandenburg" 16IS22092 gefördert. Die Verantwortung für den Inhalt dieser Veröffentlichung liegt beim Autor.

English Translation

This project was funded by the German Federal Ministry of Education and Research (BMBF) under the funding code "KI-Servicezentrum Berlin-Brandenburg" 16IS22092. Responsibility for the content of this publication remains with the author.

---

License

This model is released under the MIT License.

MIT License

Copyright (c) 2024 Aman Raut, Hasso Plattner Institute

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

---

Contact and Support