t5-speech-sanitizer

T5 Speech Sanitizer (ASR Post-Processing)

This model is a fine-tuned version of Google's t5-small, specifically designed to clean and sanitize raw Automatic Speech Recognition (ASR) transcripts. It identifies and removes conversational disfluencies—such as filler words ("um," "uh," "you know") and stuttering repetitions ("I- I went")—while strictly preserving the original semantic meaning and context of the sentence.

This model was developed as part of an Artificial Intelligence engineering initiative at the University of Technology Sydney (UTS).

Model Details

• Base Model: t5-small (60M parameters)
• Task: Sequence-to-Sequence / Text-to-Text Generation
• Language: English
• Optimizer: PyTorch (AdamW)

Intended Use

This model acts as the NLP post-processing module in a larger speech-to-text pipeline. It is designed to take the raw, messy string output from models like OpenAI's Whisper and output a clean, grammatically correct string suitable for presentation, closed captions, or UI interfaces.

Training Data & Augmentation

The model was fine-tuned using the industry-standard grammarly/coedit dataset.

Because high-quality datasets of exact acoustic disfluencies are scarce, a custom Data Augmentation algorithm was engineered during preprocessing. The baseline grammatically correct data was synthetically injected with randomized conversational fillers and simulated stuttering patterns (hyphenated word duplications). This allowed the model to learn the specific mathematical mapping of identifying and erasing disfluencies.

Training Hyperparameters

To prevent Catastrophic Forgetting of the base model's English proficiency, training was highly constrained:

• Epochs: 3
• Learning Rate: 2e-5
• Weight Decay: 0.01
• Batch Size: 16 (per device)
• Precision: Mixed Precision (fp16)

Usage & Inference (Crucial)

Note: To prevent model hallucination and recursive generation loops, you must use Beam Search decoding with n-gram repetition penalties. Furthermore, the model requires the task prefix "sanitize speech: " to trigger the correct attention weights.

Here is the exact Python implementation to use this model:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Load model and tokenizer
model_id = "[YOUR_USERNAME]/t5-speech-sanitizer" # Replace with your actual HF username
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

# The raw ASR output
raw_text = "I- I went to the store and um, bought some milk."

# 1. Add the mandatory prefix
formatted_input = "sanitize speech: " + raw_text

# 2. Tokenize
inputs = tokenizer(formatted_input, return_tensors="pt")

# 3. Generate using Beam Search (Prevents Hallucinations)
outputs = model.generate(
    **inputs, 
    max_length=128,
    num_beams=4,
    early_stopping=True,
    no_repeat_ngram_size=2
)

# 4. Decode
clean_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(clean_text)
# Output: "I went to the store and bought some milk."