nlp waseda/roberta base japanese Model description This is a Japanese RoBERTa base model pretrained on Japanese Wikipedia and the Japanese portion of CC 100. As of 2026 04 20, the license of this model has been changed from CC BY SA 4.0 to Apache License 2.0, as Creative Commons licenses are generally not well suited for machine learning models. How to use You can use this model for masked language modeling as follows: You can fine tune this model on downstream tasks. Tokenization The input text should be segmented into words by Juman++ in advance. Juman++ 2.0.0 rc3 was used for pretraining. Each word is tokenized into tokens by sentencepiece. BertJapaneseTokenizer now supports automatic JumanppTokenizer and SentencepieceTokenizer . You can use this model without any data preprocessing. Vocabulary The vocabulary consists of 32000 tokens including words (JumanDIC) and subwords induced by the unigram language model of sentencepiece. Training procedure This model was trained on Japanese Wikipedia (as of 20210920) and the Japanese portion of CC 100. It took a week using eight NVIDIA A100 GPUs. The following hyperparameters were used during pretraining: learning rate: 1e 4 per device tr…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy