indic-bert-512

IndicBERT

IndicBERT is a multilingual ALBERT model pretrained exclusively on 12 major Indian languages. It is pre-trained on our novel monolingual corpus of around 9 billion tokens and subsequently evaluated on a set of diverse tasks. IndicBERT has much fewer parameters than other multilingual models (mBERT, XLM-R etc.) while it also achieves a performance on-par or better than these models.

The 12 languages covered by IndicBERT are: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu.

The code can be found here. For more information, checkout our project page or our paper.

Pretraining Corpus

We pre-trained indic-bert on AI4Bharat's monolingual corpus. The corpus has the following distribution of languages:

Evaluation Results

IndicBERT is evaluated on IndicGLUE and some additional tasks. The results are summarized below. For more details about the tasks, refer our official repo

IndicGLUE

Additional Tasks

* Note: all models have been restricted to a max_seq_length of 128.

Downloads

The model can be downloaded here. Both tf checkpoints and pytorch binaries are included in the archive. Alternatively, you can also download it from Huggingface.

Citing

If you are using any of the resources, please cite the following article:

@inproceedings{kakwani2020indicnlpsuite,
    title={{IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages}},
    author={Divyanshu Kakwani and Anoop Kunchukuttan and Satish Golla and Gokul N.C. and Avik Bhattacharyya and Mitesh M. Khapra and Pratyush Kumar},
    year={2020},
    booktitle={Findings of EMNLP},
}

We would like to hear from you if:

• You are using our resources. Please let us know how you are putting these resources to use.
• You have any feedback on these resources.

License

The IndicBERT code (and models) are released under the MIT License.

Contributors

• Divyanshu Kakwani
• Anoop Kunchukuttan
• Gokul NC
• Satish Golla
• Avik Bhattacharyya
• Mitesh Khapra
• Pratyush Kumar

This work is the outcome of a volunteer effort as part of AI4Bharat initiative.

Contact

• Anoop Kunchukuttan ([email protected])
• Mitesh Khapra ([email protected])
• Pratyush Kumar ([email protected])