BERT base Japanese (character level tokenization with whole word masking, jawiki 20200831) This is a BERT model pretrained on texts in the Japanese language. This version of the model processes input texts with word level tokenization based on the Unidic 2.1.2 dictionary (available in unidic lite package), followed by character level tokenization. Additionally, the model is trained with the whole word masking enabled for the masked language modeling (MLM) objective. The codes for the pretraining are available at cl tohoku/bert japanese. Model architecture The model architecture is the same as the original BERT base model; 12 layers, 768 dimensions of hidden states, and 12 attention heads. Training Data The models are trained on the Japanese version of Wikipedia. The training corpus is generated from the Wikipedia Cirrussearch dump file as of August 31, 2020. The generated corpus files are 4.0GB in total, containing approximately 30M sentences. We used the MeCab morphological parser with mecab ipadic NEologd dictionary to split texts into sentences. Tokenization The texts are first tokenized by MeCab with the Unidic 2.1.2 dictionary and then split into characters. The vocabulary siz…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy