mend-flan-t5-small

mend-flan-t5-small

Turns spoken dictation into written text: removes filled pauses, resolves self-repairs, restores casing and punctuation.

send it to john no wait sarah um by friday
-> Send it to Sarah by Friday.

A fine-tune of google/flan-t5-small (77M) on 50,000 synthetically disfluent sentences plus 25,000 real Switchboard utterances. Code, evaluation and the full write-up: https://github.com/daltonoscar0/mend

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("daltonoscar0/mend-flan-t5-small")
tok = AutoTokenizer.from_pretrained("daltonoscar0/mend-flan-t5-small")

text = "um send the the report to john no wait sarah by friday"
ids = tok("clean up: " + text, return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=160)[0], skip_special_tokens=True))
# Send the report to Sarah by Friday.

The clean up: prefix is required — it is the task prefix the model was trained with. Input should be lowercased with edge punctuation removed, which is the register the model saw in training.

Results

Scored on 293 held-out Switchboard conversations the model never trained on, and on a held-out split of the synthetic data. Brackets are 95% bootstrap intervals over 2000 resamples of sentences.

The headline finding of the project is the gap between the second and third rows. A model trained only on synthetically injected disfluencies resolves about a third of real human self-repairs; adding real speech to the training mix takes it to about a half. The two intervals do not overlap.

Limitations

Read these before using it for anything.

• Half of real self-repairs are still unresolved (0.503). This is not a solved problem.
• It has learned Switchboard's transcription conventions, including where that corpus puts commas and which discourse markers it treats as removable. Against a corpus that draws those lines elsewhere it will look worse.
• No real ASR output was involved. The disfluent side of every training pair is a transcript, never a recognition hypothesis, so the model has never seen a substitution error a real recogniser makes.
• Fragments are unhandled by construction. Utterances with cut-off words ("d-") were dropped from training, so it sure does ma make a difference keeps the stranded ma.
• Register mismatch. The target is dictation; the real training speech is telephone conversation.

Two real failures from the held-out set, for calibration:

in    and then i it's like well the thrill is gone
gold  And then it's like the thrill is gone.
out   And then I, it's like, the thrill is gone.     <- abandoned "i" kept

in    degree it's of course it's a whole lot easier
gold  Degree, of course, it's a whole lot easier.
out   Degree, it's a whole lot easier.               <- drops "of course"

Training

Synthetic pairs come from a disfluency injector structured after Shriberg's account of self-repair — reparandum, interregnum, repair — applied to WikiText sentences and dictation templates. Real pairs come from the Switchboard Dialog Act corpus, split by conversation so no conversation appears in both training and evaluation.

Licence

MIT. Switchboard Dialog Act corpus is freely redistributable; WikiText is CC BY-SA 3.0.