RuBERT ruLaw This model is a continued pretraining of DeepPavlov/rubert base cased on the RusLawOD dataset — a large corpus of Russian legal texts (court decisions and normative acts). The goal of this training is to improve RuBERT’s performance on legal domain tasks such as classification, information extraction, and retrieval. Repository: https://github.com/TryDotAtwo/ruBERT ruLaw Training Details Base model: DeepPavlov/rubert base cased Task: Masked Language Modeling (MLM) Max sequence length: 512 tokens (stride 128) Batch size: 160 per device Gradient accumulation: 1 Epochs: 8 (3 in test mode) Max steps: 40,000 Warmup steps: 2,000 Mixed precision: BF16 (on A100/H100) Optimizer & scheduler: Default Hugging Face Trainer settings Evaluation metric: eval loss (best checkpoint loaded at end) Hardware: 3× NVIDIA H200 GPUs Final eval loss: Note: The ruBERT ruLaw model was pre trained on legal texts such as laws and statutes , but not specifically on judicial decisions . The evaluation reflects how well it generalizes to predicting masked tokens in Russian court rulings. Top 1 Accuracy: fraction of masked tokens predicted exactly. Top 5 Accuracy: fraction of masked tokens predicted wit…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy