russian_text_normalizer

Russian text normalization

Finetuned version of FRED-T5 large 820M.

Code repo.

Trained on ficbook, librusec and pikabu sentences, inverse text normalized using modified version of word_to_number_ru package.

Also used Kaggle Text Normalization Challenge data to be able to normalize latin also. Became to perform worse on numbers but process latin succesfully. git checkout #8c2476b to choose previous number-only version.

Usage

Put numbers and latin words inside of squared brackets and follow with T5 tokens <extra_id_0>, <extra_id_1> and so on. One is advised to split numbers on groups by 3 digits always and do not put much of anything but numbers and latin inside of [ ] brackets.

import torch
from transformers import GPT2Tokenizer, T5ForConditionalGeneration 


device='cuda'
tokenizer = GPT2Tokenizer.from_pretrained('saarus72/russian_text_normalizer', eos_token='</s>')
model = T5ForConditionalGeneration.from_pretrained('saarus72/russian_text_normalizer').to(device)
lm_text = '<SC1>Было у отца [3]<extra_id_0> сына, но не было даже [2- 3]<extra_id_1> пиджаков с блёстками за [142 990]<extra_id_2> руб.'
input_ids = torch.tensor([tokenizer.encode(lm_text)]).to(device)
outputs = model.generate(input_ids, eos_token_id=tokenizer.eos_token_id, early_stopping=True)
print(tokenizer.decode(outputs[0][1:]))
# <extra_id_0>  три\n <extra_id_1>  двух- трёх\n <extra_id_2>  сто сорок две тысячи девятьсот девяносто\n</s>

More examples