Model Card for Japanese character level DeBERTa V2 large Model description This is a Japanese DeBERTa V2 large model pre trained on Japanese Wikipedia, the Japanese portion of CC 100, and the Japanese portion of OSCAR. This model is trained with character level tokenization and whole word masking. How to use You can use this model for masked language modeling as follows: You can also fine tune this model on downstream tasks. Tokenization There is no need to tokenize texts in advance, and you can give raw texts to the tokenizer. The texts are tokenized into character level tokens by sentencepiece. Training data We used the following corpora for pre training: Japanese Wikipedia (as of 20221020, 3.2GB, 27M sentences, 1.3M documents) Japanese portion of CC 100 (85GB, 619M sentences, 66M documents) Japanese portion of OSCAR (54GB, 326M sentences, 25M documents) Note that we filtered out documents annotated with "header", "footer", or "noisy" tags in OSCAR. Also note that Japanese Wikipedia was duplicated 10 times to make the total size of the corpus comparable to that of CC 100 and OSCAR. As a result, the total size of the training data is 171GB. Training procedure We first segmented te…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy