flan-t5-base-text-paraphraser

flan-t5-base-text-paraphraser

Text paraphraser: google/flan-t5-base fine-tuned (full fine-tuning, standard seq2seq cross-entropy) on the paranmt5m configuration of redis/langcache-sentencepairs-v3 (ParaNMT-5M sentence pairs). No other data was used. Training used a seeded random subset of 1,000,000 pairs out of 5,041,448 usable training pairs.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/flan-t5-base-text-paraphraser"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "paraphrase: the weather was extremely cold yesterday ."
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, num_beams=8, num_return_sequences=3, max_new_tokens=128)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

Input format: paraphrase: <sentence> (prefix is required; also stored in paraphraser_config.json). Default generation config: beam search, num_beams=4, max_new_tokens=128.

Training data

• Source: redis/langcache-sentencepairs-v3, config paranmt5m (single train split, all label=1).
• Direction: sentence1 → sentence2.
• Text is lowercased and PTB-tokenized in the source corpus; it was not re-cased or detokenized.
• Cleaning: null/empty removal, whitespace normalization, PTB quote tokens ( `` / '') mapped to " (the T5 vocabulary has no backtick), control-character / U+FFFD / no-letter filtering, removal of pairs identical after normalization or differing only in case/punctuation, exact pair deduplication, word-length-ratio filter (>3.0 when the longer side has ≥10 words), token-length filter (128 source incl. prefix / 128 target). No truncation.
• Split: deterministic, grouped by normalized source sentence (seed 42); validation/test pairs sharing any normalized sentence with train were moved to train, test pairs sharing a sentence with validation were dropped. Exact pair leakage: none; sentence-level (loose) leakage: none.

Training procedure

Evaluation

Generation with beam search (num_beams=4); held-out splits of the processed data. BLEU: sacreBLEU corpus BLEU (13a). ROUGE: rouge_score F-measure, no stemming. BERTScore: roberta-large, baseline-rescaled. Copy rate: output equal to the input after NFKC/whitespace normalization. Distinct-n: corpus unique/total n-grams. iBLEU = 0.8·BLEU(ref) − 0.2·BLEU(source). The copy-source baseline outputs the input unchanged.

Evaluation metrics not found.

Examples (random test samples)

Qualitative examples not found.

Limitations

• Trained on lowercased, PTB-tokenized text; outputs follow that format (lowercase, spaces around punctuation).
• ParaNMT pairs were produced by back-translation; some references are loose or change meaning, and the model inherits this noise.
• Sentence-level model: inputs up to 128 tokens; training sentences have at most ~30 words.
• English only. Not evaluated for factual consistency on long or domain-specific text.
• Automatic metrics against a single reference underestimate valid paraphrases.

Citation

ParaNMT-50M: Wieting & Gimpel, 2018, ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations. FLAN-T5: Chung et al., 2022, Scaling Instruction-Finetuned Language Models.