vietnamese-error-correction

BartPho-Syllable - Vietnamese Error Correction (LoRA)

Model Details

Model Description

This model is a Fine-tuned version of vinai/bartpho-syllable using LoRA (Low-Rank Adaptation). It is specifically designed for Vietnamese Error Correction (VEC) tasks.

Unlike simple diacritic restoration models, this model aims to correct:

1. Missing Diacritics: (e.g., "trang phuc" -> "trang phục")
2. Spelling Errors: (e.g., "bài toán" vs "bài toan")
3. Teencode & Informal Variants: Normalizing teencode, slang, and informal online writing into standard Vietnamese (e.g., "zui wa" -> "vui quá", "iu vk" -> "yêu vợ").
4. Basic Grammar/Contextual correction based on syllable-level understanding.

The model was trained on a dataset of approximately 70,000 sentences across the training, validation, and test splits, which were automatically labeled using a large language model from crawled Vietnamese social media comments. Due to the nature of social media data, the dataset may contain noise or labeling imperfections; however, it is not intended to include any offensive content or to target any individual or organization.

• Developed by: Thanh-Dan Bui
• Model type: Seq2Seq (Encoder-Decoder) with LoRA Adapter
• Language(s): Vietnamese
• License: MIT
• Finetuned from model: vinai/bartpho-syllable

Uses

Direct Use

The model is designed for Vietnamese text error correction. It takes noisy Vietnamese text as input, including missing diacritics, spelling mistakes, and informal or teencode expressions, and produces grammatically correct and orthographically normalized Vietnamese text as output.

Example:

• Input: "t đang xu ly 1 bai toán la sưa lỗi cho tieng viet"
• Output: "tôi đang xử lý 1 bài toán là sửa lỗi cho tiếng Việt"

Out-of-Scope Use

• Translation from other languages to Vietnamese.
• Generating text from scratch (Open-ended generation).
• Correcting highly specialized technical jargon not present in general Vietnamese corpora.

Bias, Risks, and Limitations

• Context Length: The model is optimized for sentence-level correction (max length ~256 tokens). Very long paragraphs should be split before processing.
• Ambiguity: In cases where a noisy or abbreviated form can correspond to multiple valid standard forms (for example, variants of “không” such as “k”, “ko”, “hong”, or “hông”), the model relies on contextual information to infer the most likely correction, which may occasionally result in incorrect predictions.
• Proper Nouns: The model might attempt to "correct" foreign names or uncommon proper nouns if they resemble Vietnamese words.

How to Get Started with the Model

You can use this model with the transformers libraries.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, pipeline

path = "yammdd/vietnamese-error-correction"

tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForSeq2SeqLM.from_pretrained(path)

pipe = pipeline("text2text-generation", model=model, tokenizer=tokenizer)

text = "hum ni a bùn wá bé iu ưi"
out = pipe(text, max_new_tokens=256)

print(out[0]["generated_text"])
# Output: hôm nay anh buồn quá bé yêu ơi

Training Details

Training Data

• Source: Aggregated Vietnamese text corpus.
• Task: Vietnamese text correction (diacritic restoration and error correction).
• Size: Approximately 70,000 sentence pairs (split into Train/Validation/Test sets).
• Data Format:
  • Input: Text with removed diacritics or synthetically induced spelling errors.
  • Target: Original, grammatically correct Vietnamese text.
• Sequence Length: Maximum input and output length of 256 tokens.

Training Procedure

• Base Model: vinai/bartpho-syllable
• Technique: Parameter-Efficient Fine-Tuning (PEFT) using LoRA (Low-Rank Adaptation).
• LoRA Configuration:
  • Target Modules: q_proj, v_proj, out_proj, fc1, fc2 (covering both attention and feed-forward layers).
  • Rank (r): 32
  • Alpha: 64
  • Dropout: 0.1
• Precision: FP16 (Mixed Precision) for optimized memory usage and speed.

Training Hyperparameters

• Optimizer: AdamW with weight decay of 0.01.
• Batch Size: 16 per device (Total effective batch size depends on GPU count, typically 32 on 2x T4).
• Learning Rate: 5e-4.
• Training Epochs: 5.
• Evaluation Strategy: Every 2,000 steps.
• Label Smoothing: Implicitly handled by DataCollatorForSeq2Seq with label_pad_token_id=-100.

Speeds, Sizes, Times

• Hardware: 2x NVIDIA T4 GPUs (Kaggle environment).
• Checkpoint Size: The adapter weights are lightweight (only several megabytes), significantly smaller than the full BARTpho base model.
• Training Dynamics: Managed via the Hugging Face Seq2SeqTrainer with predict_with_generate enabled for validation metrics.

Evaluation

Testing Data, Factors & Metrics

The model was evaluated on a held-out test set of 7,056 samples, covering a diverse range of Vietnamese sentence structures and lengths.

Metrics

• BLEU Score: Measures the n-gram overlap between the predicted and target text.
• Word Error Rate (WER): Measures the ratio of errors (substitutions, deletions, insertions) at the word level.
• Character Error Rate (CER): Measures accuracy at the character level, which is critical for verifying diacritic placement.
• Exact Match Accuracy: The percentage of sentences where every single character matches the ground truth.
• Word Accuracy: The percentage of individual words correctly predicted (excluding length mismatches).

Results

1. Overall Performance

Note: The Exact Match score reflects the inherent ambiguity in the Vietnamese language (e.g., "muon" could be "muốn", "mượn", or "muộn"), where multiple correct interpretations may exist without broader paragraph context.

2. Accuracy by Sentence Length

The model's performance varies based on the complexity and length of the input:

Analysis: The model performs exceptionally well on short to medium sentences. Accuracy declines on longer sequences (>30 words), likely due to the increased probability of cumulative errors and the 256-token limit.

---

Environmental Impact

• Hardware Type: 2 x NVIDIA Tesla T4 GPUs.
• Cloud Provider: Kaggle.
• Training Duration: ~4 hours.
• Carbon Emitted: Estimated based on the total GPU hours and the carbon intensity of the hosting region.

Framework Versions

• PEFT: 0.18.0
• Transformers: 4.57.3
• PyTorch: 2.9.0
• Datasets: 4.0.0

Note

• This model is intended for educational and research purposes only.