Model Card for Japanese DeBERTa V3 base Model description This is a Japanese DeBERTa V3 base model pre trained on LLM jp corpus v1.0. How to use You can use this model for masked language modeling as follows: You can also fine tune this model on downstream tasks. Tokenization The tokenizer of this model is based on huggingface/tokenizers Unigram byte fallback model. The vocabulary entries were converted from llm jp tokenizer v2.2 (100k) . Please refer to README.md of llm jp/llm ja tokenizer for details on the vocabulary construction procedure. Note that, unlike ku nlp/deberta v2 base japanese, pre segmentation by a morphological analyzer (e.g., Juman++) is no longer required for this model. Training data We used the LLM jp corpus v1.0.1 for pre training. The corpus consists of the following corpora: Japanese Wikipedia (1B tokens) mC4 (129B tokens) English Wikipedia (4B tokens) The Pile (126B tokens) Code The Stack (10B tokens) We shuffled the corpora, which has 270B tokens in total, and trained the model for 2 epochs. Thus, the total number of tokens fed to the model was 540B. Training procedure We slightly modified the official implementation of DeBERTa V3 and followed the officia…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy