BERT base Japanese (unidic lite with whole word masking, CC 100 and jawiki 20230102) This is a BERT model pretrained on texts in the Japanese language. This version of the model processes input texts with word level tokenization based on the Unidic 2.1.2 dictionary (available in unidic lite package), followed by the WordPiece subword tokenization. Additionally, the model is trained with the whole word masking enabled for the masked language modeling (MLM) objective. The codes for the pretraining are available at cl tohoku/bert japanese. Model architecture The model architecture is the same as the original BERT base model; 12 layers, 768 dimensions of hidden states, and 12 attention heads. Training Data The model is trained on the Japanese portion of CC 100 dataset and the Japanese version of Wikipedia. For Wikipedia, we generated a text corpus from the Wikipedia Cirrussearch dump file as of January 2, 2023. The corpus files generated from CC 100 and Wikipedia are 74.3GB and 4.9GB in size and consist of approximately 392M and 34M sentences, respectively. For the purpose of splitting texts into sentences, we used fugashi with mecab ipadic NEologd dictionary (v0.0.7). Tokenization The…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy