rubert-finetune-goods_NER

Model Details

Model Description

BERT-base model for NER task in search field (Russian retail domain). Language - Russian (English not tested).

• Developed by: Evgen Novik
• Model type: BERT - encoder
• Language(s) (NLP): Russian
• License: Apache 2.0
• Finetuned from model: DeepPavlov-rubert

Uses

NER for TYPE, BRAND, O entities in search field (russian retail domain).

Direct Use

from transformers import pipeline, AutoTokenizer, AutoModelForTokenClassification

model_name = "MegaLDN/rubert-finetune-goods_NER"
tokenizer = AutoTokenizer.from_pretrained(model_name])
model = AutoModelForTokenClassification.from_pretrained(model_name)

ner_pipeline = pipeline(
"ner",
model=model ,
tokenizer=tokenizer ,
aggregation_strategy="none"
)

text = "Сок агуша 2л"
output = ner_pipeline(text)
# >> print(output)
# << [{'entity': 'B-TYPE', 'score': np.float32(0.99929225), 'index': 1, 'word': 'Сок', 'start': 0, 'end': 3}, {'entity': 'B-BRAND', 'score': np.float32(0.8768344), 'index': 2, 'word': 'аг', 'start': 4, 'end': 6}, {'entity': 'B-BRAND', 'score': np.float32(0.85818267), 'index': 3, 'word': '##уша', 'start': 6, 'end': 9}, {'entity': 'B-VOLUME', 'score': np.float32(0.99927515), 'index': 4, 'word': '2', 'start': 10, 'end': 11}, {'entity': 'B-VOLUME', 'score': np.float32(0.9982621), 'index': 5, 'word': '##л', 'start': 11, 'end': 12}]

Bias, Risks, and Limitations

The author is not responsible for any costs incurred as a result of using this model.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

Training Details

Training Data

Synthetic data (100k samples of spans)

Training Procedure

2 stages of tuning:

1. Pre-tune with synthetic data (large LR, bs = 32, clip_grad=1.0);
2. Fine - tune with non-synthetic data (lower LR, bs = 32, clip_grad=1.0)

Preprocessing [optional]

Data format - spans.

Training Hyperparameters

• First tune:

    eval_strategy="epoch",
    save_strategy="epoch",
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    #gradient_accumulation_steps=2,
    num_train_epochs=3,

    learning_rate=2e-5,
    weight_decay=0.01,
    warmup_ratio=0.06,
    lr_scheduler_type="linear",
    max_grad_norm=1.0,
    fp16=True,
    load_best_model_at_end=True,
    metric_for_best_model="f1",
    greater_is_better=True,

    save_total_limit=2,
    dataloader_drop_last=False,
    seed=42

• Second tune:

    eval_strategy="epoch",
    save_strategy="epoch",
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    #gradient_accumulation_steps=1,
    num_train_epochs=3,

    learning_rate=1e-5,
    weight_decay=0.01,
    warmup_ratio=0.1,
    lr_scheduler_type="linear",
    max_grad_norm=1.0,
    fp16=True,
    load_best_model_at_end=True,
    metric_for_best_model="f1",
    greater_is_better=True,

    save_total_limit=2,
    dataloader_drop_last=False,

Evaluation

F1 (test) = 0.6 (depend of domain)

Technical Specifications

Compute Infrastructure

Approximately 20 min in 16gb RAM (4060 ti 16gb)

Software

transformers 4.55 torch 2.7 cu118 python 3.12.9