bert-small-buddhist-nonbuddhist-sanskrit

bert-small-buddhist-nonbuddhist-sanskrit

BERT model trained on a lemmatized corpus containing Buddhist and non-Buddhist Sanskrit texts.

Model description

The model has the bert architecture and was pretrained from scratch as a masked language model on the lemmatized Sanskrit corpus. Due to lack of resources and to prevent overfitting, the model is smaller than bert-base, i.e. the number of attention heads and hidden layers have been reduced to 8 and the context has been reduced to 128 tokens. Vocabulary size is 10000 tokens.

How to use it

model = AutoModelForMaskedLM.from_pretrained("Matej/bert-small-buddhist-nonbuddhist-sanskrit")
tokenizer = AutoTokenizer.from_pretrained("Matej/bert-small-buddhist-nonbuddhist-sanskrit", use_fast=True)

Intended uses & limitations

MIT license, no limitations

Training and evaluation data

See the paper 'Embeddings models for Buddhist Sanskrit' for details on the corpora and the evaluation procedure.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 200

Framework versions

• Transformers 4.20.0
• Pytorch 1.9.0
• Datasets 2.3.2
• Tokenizers 0.12.1