bangla-paraphrase-banglat5

Bengali Paraphrase Benchmark - BanglaT5

TL;DR: The strongest overall performer in this study — highest BLEU, cleanest LLM-judge error profile (zero meaning-reversal cases), at the cost of full-parameter fine-tuning (247.6M params) vs. the QLoRA models' ~13-18M.

A full fine-tune of csebuetnlp/banglat5 for Bengali paraphrase generation, trained on the BanglaParaphrase dataset.

This model is one of four architectures compared in a broader study, "Beyond BLEU: Evaluating Automated Metric Reliability for Bengali Paraphrase Generation" (GitHub repo), which benchmarks full fine-tuning vs. QLoRA and validates standard automated metrics (BLEU, ROUGE-L, BERTScore, PINC) against an LLM-as-a-judge. Across all four models, this one achieved the strongest overall results.

⚠️ License Notice

The base model (csebuetnlp/banglat5, BUET CSE NLP Group) does not display a clearly specified license on its Hugging Face repository or associated GitHub repo. This fine-tune is shared for research and educational purposes. Please cite the original BanglaT5 paper, and contact the BUET CSE NLP Group directly before any commercial use of this model or its base weights.

Model Details

Trained on a single free-tier Colab T4 GPU; exact carbon footprint not tracked, but compute was intentionally minimized to demonstrate feasibility for a low-resource-language task even at full fine-tuning scale.

Intended Use

Generating Bengali paraphrases of an input sentence — e.g., data augmentation for downstream Bengali NLP tasks, or research on low-resource paraphrase generation. Evaluated only on the BanglaParaphrase test distribution; not evaluated on other domains, dialects, or languages.

Out-of-Scope Use

• Not intended for languages other than Bengali, or for code-mixed/transliterated text.
• Not intended for high-stakes or unreviewed automated use — while this model had the cleanest error profile of the four studied (see Evaluation Results), it was still evaluated on only a 100-example judged subset, not exhaustively verified.
• Not a general-purpose Bengali text generation or summarization model — it is trained specifically for sentence-level paraphrasing and has not been evaluated for other generation tasks.

How to Use

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from normalizer import normalize  # pip install git+https://github.com/csebuetnlp/normalizer

model_name = "AvinabhDutta-Dev/bangla-paraphrase-banglat5"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

text = normalize("এই বাড়িতে সাধারণত কোনো ঝগড়াঝাঁটি হয় না।")
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=96)
output_ids = model.generate(**inputs, max_length=96)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Important: input text must be passed through csebuetnlp/normalizer's normalize() function before tokenizing — this matches the preprocessing used during training, and skipping it will degrade output quality.

Sample Outputs

Real outputs from this model on the held-out test set, judged 5/5 for both meaning preservation and fluency by the study's LLM-judge evaluation:

Evaluation Results

Evaluated on the full, held-out BanglaParaphrase test set (23,332 examples), using the same pipeline and test data across all four models in this study.

LLM-as-a-Judge Validation

A 400-pair, four-model symmetric subset (100 examples per model, identical sources) was independently scored by Gemini 3.5 Flash-Lite on semantic meaning preservation and fluency (1–5 scale), with a binary meaning-reversal flag.

Error breakdown (out of 100 judged examples):

BanglaT5 had the cleanest error profile of all four models in this study — zero meaning-reversal or disfluency cases, and the highest judge-rated meaning-preservation and fluency scores.

Related Models (this study)

• IndicBARTSS (full fine-tune)
• TigerLLM-1B (QLoRA)
• Qwen2.5-1.5B (QLoRA)

Limitations

• Trained on a 100K-example subset (not the full ~420K available training pairs) and evaluated with greedy decoding, for compute tractability. A directly comparable fine-tuned BanglaT5 trained on the full dataset has been reported to reach substantially higher BLEU in prior work — this gap likely reflects training-data volume and decoding strategy rather than a limitation of the architecture itself.

• Requires the csebuetnlp/normalizer preprocessing step; omitting it was not tested and is likely to degrade quality.

• BLEU/ROUGE-L are known to correlate only weakly with actual paraphrase quality (see the parent study's Notebook 07); BERTScore and the LLM-judge results are more informative for assessing this model.

• Reproducibility: Trained on a fixed 100,000-pair subset of BanglaParaphrase with a documented train/val/test split (see processed_data/metadata.json in the GitHub repo); decoding was fully deterministic (greedy decoding), so generation outputs are reproducible given the same checkpoint and inputs.

Citation

If you use this model, please cite both the original BanglaT5/BanglaParaphrase work and this study:

@article{bhattacharjee2022banglanlg,
  title={BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla},
  author={Bhattacharjee, Abhik and others},
  journal={arXiv preprint arXiv:2205.11081},
  year={2022}
}

@misc{banglaparaphrase2022,
  title={BanglaParaphrase: A High-Quality Bengali Paraphrase Dataset},
  author={Akil, Ajwad and others},
  year={2022}
}

See the project GitHub repository for the full paper, code, and citation for this study.

Framework Versions

• Transformers 5.13.1

---

license: other license_name: unspecified-base-model-license license_link: https://huggingface.co/csebuetnlp/banglat5