Downloads · 30 days
217
1% of all-time downloads
ruBERT-ruLaw
TryDotAtwo/ruBERT-ruLaw
ruBERT-ruLaw is a fill-mask model from TryDotAtwo. Use it when you need the model to fill a missing word. The card lists the license as cc-by-4.0.
This model is a continued pretraining of DeepPavlov/rubert-base-cased on the RusLawOD dataset — a large corpus of Russian legal texts (court decisions and normative acts). The goal of this training is to improve RuBER…
All-time downloads
21.5K
Public
Parameters
178M
1.4 GB on disk
Likes
4
Public
Repo makeup
Click a slice to open those files.
.safetensors712 MB · 99%
At a glance
- Task
- Fill-Mask
- License
- cc-by-4.0
- Model type
- bert
- Access
- Public
- Created
- Oct 16, 2025
- Updated
- Oct 26, 2025
- SHA
- 67383a55
Try a prompt
Base models
Datasets
Model card
From the Hugging Face model README
RuBERT-ruLaw
This model is a continued pretraining of DeepPavlov/rubert-base-cased
on the RusLawOD dataset — a large corpus of Russian legal texts (court decisions and normative acts).
The goal of this training is to improve RuBERT’s performance on legal domain tasks such as classification, information extraction, and retrieval.
Repository: https://github.com/TryDotAtwo/ruBERT-ruLaw
Training Details
- Base model: DeepPavlov/rubert-base-cased
- Task: Masked Language Modeling (MLM)
- Max sequence length: 512 tokens (stride 128)
- Batch size: 160 per device
- Gradient accumulation: 1
- Epochs: 8 (3 in test mode)
- Max steps: 40,000
- Warmup steps: 2,000
- Mixed precision: BF16 (on A100/H100)
- Optimizer & scheduler: Default Hugging Face Trainer settings
- Evaluation metric:
eval_loss(best checkpoint loaded at end) - Hardware: 3× NVIDIA H200 GPUs
- Final eval loss: < 0.3
Dataset
We use the RusLawOD dataset.
Before tokenization, empty or None entries are filtered out. Tokenization is performed with the RuBERT tokenizer with truncation and sliding window (stride 128) to maximize coverage of long documents.
Usage
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_name = "TryDotAtwo/rubert-rulaw"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
Evaluation Overview
Models were tested on the sud-resh-benchmark legal texts using a masked language modeling setup. Tokens were randomly masked at varying probabilities (10–40%), and models predicted them using their pre-trained heads.
Note: The ruBERT-ruLaw model was pre-trained on legal texts such as laws and statutes, but not specifically on judicial decisions. The evaluation reflects how well it generalizes to predicting masked tokens in Russian court rulings.
- Top-1 Accuracy: fraction of masked tokens predicted exactly.
- Top-5 Accuracy: fraction of masked tokens predicted within the top 5 candidates.
Results reflect performance across all masked tokens, aggregated for the dataset.
MLM Accuracy Comparison
| MLM Probability | Metric | ruBERT-ruLaw | rubert-base-cased | legal-bert-base-uncased |
|---|---|---|---|---|
| 10% | Top-1 | 81.0% | 73.0% | 45.3% |
| 10% | Top-5 | 92.2% | 87.0% | 77.2% |
| 15% | Top-1 | 78.8% | 67.9% | 45.3% |
| 15% | Top-5 | 90.8% | 83.2% | 76.7% |
| 20% | Top-1 | 76.3% | 53.8% | 45.0% |
| 20% | Top-5 | 89.0% | 71.5% | 75.9% |
| 25% | Top-1 | 73.6% | 18.0% | 44.4% |
| 25% | Top-5 | 87.0% | 31.9% | 75.0% |
| 30% | Top-1 | 70.4% | 5.9% | 43.8% |
| 30% | Top-5 | 84.6% | 10.9% | 74.0% |
| 35% | Top-1 | 66.9% | 6.0% | 42.9% |
| 35% | Top-5 | 81.9% | 9.1% | 72.9% |
| 40% | Top-1 | 62.9% | 6.0% | 41.9% |
| 40% | Top-5 | 78.5% | 8.5% | 71.7% |
Citation
A paper describing the dataset and training process will be released on arXiv soon. [Link — coming soon]
- Task
- Fill-Mask
- Type
- bert
- License
- cc-by-4.0
- Languages
- ru
- Created
- Oct 16, 2025
- Updated
- Oct 26, 2025