bert-small-amharic

bert-small-amharic

This model has the same architecture as bert-small and was pretrained from scratch using the Amharic subsets of the oscar, mc4, and amharic-sentences-corpus datasets, on a total of 290 Million tokens. The tokenizer was trained from scratch on the same text corpus, and had a vocabulary size of 28k. It achieves the following results on the evaluation set:

• Loss: 2.77
• Perplexity: 15.96

Even though this model only has 27.8 Million parameters, its performance is comparable to the 10x larger 279 Million parameter xlm-roberta-base multilingual model on the same Amharic evaluation set.

How to use

You can use this model directly with a pipeline for masked language modeling:

>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='rasyosef/bert-small-amharic')
>>> unmasker("ከሀገራቸው ከኢትዮጵያ ከወጡ ግማሽ ምዕተ [MASK] ተቆጥሯል።")

[{'score': 0.5164287686347961,
  'token': 9345,
  'token_str': 'ዓመት',
  'sequence': 'ከሀገራቸው ከኢትዮጵያ ከወጡ ግማሽ ምዕተ ዓመት ተቆጥሯል ።'},
 {'score': 0.2229153960943222,
  'token': 9913,
  'token_str': 'አመት',
  'sequence': 'ከሀገራቸው ከኢትዮጵያ ከወጡ ግማሽ ምዕተ አመት ተቆጥሯል ።'},
 {'score': 0.1806153655052185,
  'token': 9617,
  'token_str': 'ዓመታት',
  'sequence': 'ከሀገራቸው ከኢትዮጵያ ከወጡ ግማሽ ምዕተ ዓመታት ተቆጥሯል ።'},
 {'score': 0.05486353859305382,
  'token': 10898,
  'token_str': 'አመታት',
  'sequence': 'ከሀገራቸው ከኢትዮጵያ ከወጡ ግማሽ ምዕተ አመታት ተቆጥሯል ።'},
 {'score': 0.014157092198729515,
  'token': 28157,
  'token_str': '##ዓመት',
  'sequence': 'ከሀገራቸው ከኢትዮጵያ ከወጡ ግማሽ ምዕተዓመት ተቆጥሯል ።'}]

Finetuning

This model was finetuned and evaluated on the following Amharic NLP tasks

• Sentiment Classification
  • Dataset: amharic-sentiment
  • Code: https://github.com/rasyosef/amharic-sentiment-classification
• Named Entity Recognition
  • Dataset: amharic-named-entity-recognition
  • Code: https://github.com/rasyosef/amharic-named-entity-recognition
• News Category Classification
  • Dataset: amharic-news-category-classification
  • Code: https://github.com/rasyosef/amharic-news-category-classification

Finetuned Model Performance

The reported F1 scores are macro averages.

Amharic News Category Classification