japanese gpt2 medium This repository provides a medium sized Japanese GPT 2 model. The model was trained using code from Github repository rinnakk/japanese pretrained models by rinna Co., Ltd. How to use the model ~~~~ from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from pretrained("rinna/japanese gpt2 medium", use fast=False) tokenizer.do lower case = True due to some bug of tokenizer config loading model = AutoModelForCausalLM.from pretrained("rinna/japanese gpt2 medium") ~~~~ Model architecture A 24 layer, 1024 hidden size transformer based language model. Training The model was trained on Japanese CC 100 and Japanese Wikipedia to optimize a traditional language modelling objective on 8\\ V100 GPUs for around 30 days. It reaches around 18 perplexity on a chosen validation set from the same data. Tokenization The model uses a sentencepiece based tokenizer, the vocabulary was trained on the Japanese Wikipedia using the official sentencepiece training script. Release date April 7, 2021 (Updated: August 25, 2021) How to cite Licenese The MIT license
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy