Model Card for Japanese DeBERTa V2 tiny Model description This is a Japanese DeBERTa V2 tiny model pre trained on Japanese Wikipedia, the Japanese portion of CC 100, and the Japanese portion of OSCAR. How to use You can use this model for masked language modeling as follows: You can also fine tune this model on downstream tasks. Tokenization The input text should be segmented into words by Juman++ in advance. Juman++ 2.0.0 rc3 was used for pre training. Each word is tokenized into subwords by sentencepiece. Training data We used the following corpora for pre training: Japanese Wikipedia (as of 20221020, 3.2GB, 27M sentences, 1.3M documents) Japanese portion of CC 100 (85GB, 619M sentences, 66M documents) Japanese portion of OSCAR (54GB, 326M sentences, 25M documents) Note that we filtered out documents annotated with "header", "footer", or "noisy" tags in OSCAR. Also note that Japanese Wikipedia was duplicated 10 times to make the total size of the corpus comparable to that of CC 100 and OSCAR. As a result, the total size of the training data is 171GB. Training procedure We first segmented texts in the corpora into words using Juman++. Then, we built a sentencepiece model with 3200…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy