Chinese GPT2 Models Model description The set of GPT2 models, except for GPT2 xlarge model, are pre trained by UER py, which is introduced in this paper. The GPT2 xlarge model is pre trained by TencentPretrain introduced in this paper, which inherits UER py to support models with parameters above one billion, and extends it to a multimodal pre training framework. Besides, the other models could also be pre trained by TencentPretrain. The model is used to generate Chinese texts. You can download the set of Chinese GPT2 models either from the UER py Modelzoo page, or via HuggingFace from the links below: Link : : GPT2 distil [ L=6/H=768 ][distil] GPT2 [ L=12/H=768 ][base] GPT2 medium [ L=24/H=1024 ][medium] GPT2 large [ L=36/H=1280 ][large] GPT2 xlarge [ L=48/H=1600 ][xlarge] Note that the 6 layer model is called GPT2 distil model because it follows the configuration of distilgpt2, and the pre training does not involve the supervision of larger models. How to use You can use the model directly with a pipeline for text generation (take the case of GPT2 distil): Training data CLUECorpusSmall is used as training data. Training procedure The GPT2 xlarge model is pre trained by TencentPre…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy