NusaMT-7B

Model Card for Model ID

NusaMT-7B is a large language model fine-tuned for machine translation of low-resource Indonesian languages, with a focus on Balinese and Minangkabau. Built on LLaMA2-7B and leveraging the Komodo-7B-base model, it incorporates continued pre-training on non-English monolingual data, supervised fine-tuning, data preprocessing for cleaning parallel sentences, and synthetic data generation.

Model Details

Model Description

• Developed by: William Tan
• Model type: Decoder-only Large Language Model
• Language(s) (NLP): Balinese, Minangkabau, Indonesian, English

<!-- - **License:** [More Information Needed] -->

• Finetuned from model: Yellow-AI-NLP/komodo-7b-base

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/williammtan/nusamt
• Paper: https://arxiv.org/abs/2410.07830
• Demo: https://indonesiaku.com/translate

Uses

The model is designed for:

• Bidirectional translation between English/Indonesian and low-resource Indonesian languages (currently Balinese and Minangkabau)
• Language preservation and documentation
• Cross-cultural communication
• Educational purposes and language learning

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

• Integrated into translation applications
• Used for data augmentation in low-resource language tasks
• Adapted for other Indonesian regional languages
• Used as a foundation for developing language learning tools

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

The model is not suitable for:

• Translation of languages outside its trained scope
• General text generation or chat functionality
• Real-time translation requiring minimal latency
• Critical applications where translation errors could cause harm

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

• Limited to specific language pairs (English/Indonesian ↔ Balinese/Minangkabau)
• Performance varies between translation directions, with better results for translations into low-resource languages
• Underperforms larger models (NLLB-3.3B) in translations into high-resource languages
• May not capture all dialectal variations or cultural nuances
• Uses significantly more parameters (7 billion) compared to traditional NMT models
• Limited by the quality and quantity of available training data

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

How to Get Started with the Model

Use the code below to get started with the model.

Training Details

Training Data

NusaMT: https://huggingface.co/datasets/williamhtan/NusaMT

Total parallel sentences after cleaning:

• Balinese ↔ English: 35.6k sentences
• Balinese ↔ Indonesian: 44.9k sentences
• Minangkabau ↔ English: 16.6k sentences
• Minangkabau ↔ Indonesian: 22.4k sentences

Data sources:

• NLLB Mined corpus (ODC-BY license)
• NLLB SEED dataset (CC-BY-SA license)
• BASAbaliWiki (CC-BY-SA license)
• Bible verses from Alkitab.mobi (for non-profit scholarly use)
• NusaX dataset (CC-BY-SA license)

Preprocessing

• Length filtering (15-500 characters)
• Word length ratio of 2
• Removal of sentences with words >20 characters
• Deduplication
• Language identification with GlotLid V3 (threshold: 0.9)
• LASER3 similarity scoring (threshold: 1.09)
• GPT-4o mini-based data cleaning

Training Hyperparameters

• Training regime: bfloat16 mixed precision
• LoRA rank: 16
• Learning rate: 0.002
• Batch size: 10 per device
• Epochs: 3
• Data splits: 90% training, 5% validation, 5% testing
• Loss: Causal Language Modeling (CLM)

<!-- #### Speeds, Sizes, Times [optional] --> <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. --> <!-- [More Information Needed] -->

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

• FLORES-200 multilingual translation benchmark
• Internal test set (5% of parallel data)

Metrics

• spBLEU (SentencePiece tokenized BLEU)

Results

Performance highlights:

• Outperforms SOTA models by up to +6.69 spBLEU in translations into Balinese
• Underperforms by up to -3.38 spBLEU in translations into higher-resource languages
• Consistently outperforms GPT-3.5, GPT-4, and GPT-4o in zero-shot translation

Table 2: spBLEU Score Comparison of the LLaMA2-7B SFT Model with Various Enhancements

This table presents spBLEU scores for various configurations of the LLaMA2-7B model, showing the impact of monolingual pre-training, backtranslation, and LLM cleaning on translation performance across different language pairs.

Table 3: spBLEU Scores of NusaMT-7B Compared Against SoTA Models and Large GPT Models

This table compares the performance of NusaMT-7B with state-of-the-art models and large GPT models in terms of spBLEU scores across multiple language pairs. NusaMT-7B shows significant improvements, particularly in translations into low-resource languages.

Environmental Impact

• Hardware Type: 2x NVIDIA RTX 4090
• Hours used: 1250
• Cloud Provider: Runpod.io
• Carbon Emitted: 210 kg CO2e

Citation

If you find this model useful, please cite the following works

@misc{tan2024nusamt7bmachinetranslationlowresource,
      title={NusaMT-7B: Machine Translation for Low-Resource Indonesian Languages with Large Language Models}, 
      author={William Tan and Kevin Zhu},
      year={2024},
      eprint={2410.07830},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.07830}, 
}